WO2025257130A1

WO2025257130A1 - Recipe converter and method for operating a household appliance

Info

Publication number: WO2025257130A1
Application number: PCT/EP2025/066027
Authority: WO
Inventors: Magdalena Sacha; Annette Blum; Jacek Piotrowski; Alena Jahn; Andreas Wechsler; Daniela BLITSCHKE; Aleksandra Kurzewska; Michael Mück
Original assignee: BSH Hausgeraete GmbH
Current assignee: BSH Hausgeraete GmbH
Priority date: 2024-06-12
Filing date: 2025-06-10
Publication date: 2025-12-18
Anticipated expiration: 2026-12-12
Also published as: EP4664015A1

Abstract

A method (200) for operating a household appliance (155) to carry out a predetermined household task comprises steps of determining functional capabilities of the household appliance (155); providing an LLM (130) that is trained to perform linguistic operations on an input text; reading into the LLM (130) a text-based, especially generic, recipe for carrying out said household task; and causing or triggering or prompting the LLM (130) to generate, based on the text-based recipe, operating instructions for operating the household appliance (155); such that the household appliance (155) can be operated on the basis of said generated instructions to carry out said household task.

Description

RECIPE CONVERTER AND METHOD FOR OPERATING A HOUSEHOLD APPLIANCE

The present invention concerns the generation of a recipe for operating a household appliance to realize a predetermined household task. More specifically, the present invention comprises generating a recipe, which is adapted to a specific household appliance, based on a generic recipe.

A household appliance is configured to be operated to perform or aid in a predetermined household task. The task may be expressed in the shape of a recipe. For instance, a food processor may be operated to prepare a predetermined dish according to a cooking recipe or a washing machine may be operated to clean garments (laundry) according to care instructions.

A modem household appliance may have more options or capabilities than a generic or traditional appliance. For instance, a traditional oven may just have a chamber that can be heated to a predetermined temperature, while a contemporary oven may also have a grill with which a dish may be cooked "au gratin". A recipe, in order to be usable with the largest possible range of household appliances, may be targeted at a very simple appliance, leaving additional functionality of an actual appliance unused.

More and more capable household appliances require more and more knowledge about how additional features can be used in performing a predetermined household task. A user’s manual often remains unread or its contents is not memorized thoroughly. It may also require processing knowledge to adapt a given recipe to a household appliance. A typical user may therefore not be able to operate a household appliance in the best possible manner.

An object underlying the present invention lies in providing an improved technique for operating a household appliance for carrying out a predetermined household task. The invention solves the above-mentioned object through the subject matter of the independent claims. Dependent claims describe preferred embodiments. According to a first aspect of the present invention, a method for operating a household appliance to carry out a predetermined household task comprises steps of determining functional capabilities of the household appliance; providing an LLM that is trained to perform linguistic operations on an input text; and reading into the LLM a text-based, especially generic, recipe for carrying out said household task. Preferably, household specific instructions like settings for certain cooking actions or limitations of the appliance are also read in. The LLM may then be caused or triggered or prompted to generate, based on the text-based recipe, operating instructions for operating the household appliance; such that the household appliance can be operated on the basis of said generated instructions to carry out said household task. While generating the operating instructions for the household appliance, operating instructions may be extracted from the text-based recipe. The generated operating instructions may be much more specific than the generic instructions of the text-based recipe.

The LLM is a large language model which is a deep neural network that is trained to perform linguistic operations. Such training comprises configuration of hidden neural layers in the network and may be called deep learning. The LLM can especially be a generative pre-trained transformer (GPT). The LLM may be realized by a processing unit, especially of a computer. While training of an LLM may be a computationally expensive task, using a pre-trained model may not require large resources so that the LLM may be carried out by a reasonably equipped embedded or home computer. LLMs have become well known and widely available in both free and paid services. Examples for known LLMs comprise ChatGPT, Gemini (formerly known as Bard), Chatsonic or Phind.

The recipe is preferred to be a text description of actions, parameters or hints on how to treat some subject to carry out the household task. The household task may especially comprise cooking or cleaning or a related task. A recipe may be drawn from the Internet or passed from person to person. It may comprise symbols, pictograms or pictures which may be transcribed into text form. In some cas- es, a recipe may be extracted from a book, especially a cookbook, a textbook or an operation manual. The recipe may be attached to a subject to treat, for instance a garment to clean, or it may come as separate instructions. The recipe may comprise a link to an online recipe, copied text or a scanned book extract. Operating instructions may comprise an operation setting for the household appliance, like a temperature, a duration or a magnitude of a physical treatment.

Several approaches for causing the LLM to generate said operating instructions from the recipe are proposed herein, wherein different variants may be combined.

In a first variant, the LLM may be trained to relate recipe steps to appliance capabilities. Training may change an internal structure of the LLM so that an existing LLM may be adapted to the task at hand or a new LLM may be generated for the specific purpose. The training may be done separately from using the trained LLM in a method described herein. Training may require labeled training data, especially a set of generic text-based recipes and expected generated operation instructions.

In some embodiments, the LLM is a GPT and it is trained with a large amount of recipes, which are optimized for a food processor like the Cooklt of Bosch or Thermomix of Vorwerk. These appliance-specific recipes may be in a tabularized form (i.e. in machine syntax). Such a tabularized recipe may be much more technical and less intuitively understandable for a human being like recipes in a textbook. Nevertheless, a tabularized recipe may be directly executable by a food processor and may exploit all capabilities of the food processor.

Generic pre-trained models may exhibit inherent limitations, often manifesting in hallucinations or a lack of expertise in specific domains. Their training demands significant computational resources, relying on extensive energy consumption and specialized GPU infrastructure. It is preferred that the LLM is trained using a fine-tuning technique. In some embodiments, prompting or prompt engineering may be used to optimize the output of the LLM. While many methods of customizing an LLM can be called fine-tuning, it is preferred that fine-tuning comprises changing parameters of the model. That is, the model or a part of it is re-trained using traditional machine learning understanding or algorithms like LoRA (Low-Rank Adaptation; see “LoRA: Low-Rank Adaptation of Large Language Models” written by Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen in 2021 ; arXiv: 2106.09685; https://doi.org/10.48550/arXiv.2106.09685). Such methods may be more efficient and/or consume less hardware resources than the original model training. Note that prompting will generally not change a parameter of the model, while said fine- tuning techniques will.

In deep learning, fine-tuning is an approach to transfer learning in which the weights of a pre-trained model are trained on new data. Fine-tuning can be done on the entire neural network, or on only a subset of its layers, in which case the layers that are not being fine-tuned are "frozen" (not updated during a backpropa- gation step). A model may also be augmented with "adapters" that consist of far fewer parameters than the original model, and fine-tuned in a parameter-efficient way by tuning the weights of the adapters and leaving the rest of the model's weights frozen.

It is especially preferred to use a Parameter-Efficient Fine-Tuning (PEFT) technique for tuning. Such approaches were summarized 2023 by Vladislav Lialin, Vi- jeta Deshpande, Anna Rumshisky in “Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning” (arXiv: 2303.15647; see https://doi.org/10.48550/ arXiv.2303.15647)). Implementations of such techniques may be freely available from existing libraries or packages.

This innovation allows users to adapt a pre-trained model, especially an LLM which may already be trained with vast amounts of data, to their specific needs without requiring a formidable infrastructure. PEFT facilitates fine-tuning on narrower domains, thereby enhancing the LLM's performance in specialized contexts. As we navigate the dynamic landscape of language technology, the pursuit of more efficient, domain-aware, and environmentally sustainable solutions remains crucial for further advancements in this field. Taking this into account, for example, the LoRA algorithm can be used to fine-tune the LLM in an efficient way.

The dataset used for this process may present to the LLM the differences between a traditional generic recipe, which is not optimized to the specific household appliance, which is possessed by the user (x_n), and a recipe, that leverages optimal settings of the specific household appliance (y_n) of the user. A dataset of pairs of a generic recipe and a corresponding optimized converted recipe {(x_n,y_n)|n = 1 , ... , N} may be collected as a training set by a respective machine learning specialist. The converted recipe is a basis for creating a computer/device readable set of instructions for a specific household appliance.

Navigating a landscape rich in domain-specific knowledge like the cooking domain, especially across various types and models of appliances, often demands a sophisticated approach. The use of a prompt chain and the skillful combination of prompts may be advantageous. Leveraging a sequence of prompts allows for the incorporation of specialized knowledge acquired from distinct tools and sources. According to a second variant, which may be combined with all other embodiments, the LLM may be prompted to generate said operating instructions. This approach may be used both with a general LLM or a specifically trained or finetuned LLM. Prompting generally does not change the internal structure of the LLM. By finding or constructing a prompt, the LLM may be caused or triggered to perform just about any expressible operation on the input text. The information that was used to train the LLM may amount to a certain world model that is, in effect, very similar to a certain degree of understanding of a text or its contents. Such a model may be based on an observation of possible uses of a fact, a word or an expression. Synonyms or cause-and-effect chains may be accounted for. An ad- vanced LLM may be prompted to perform logical operations on the text that go far beyond word replacement and target at a novel combination of facts or operations. A sequence of prompts may be generated to arrive at an operating instruction. It may be easier to have the LLM perform the generation in several steps so that the generation is divided into subtasks that may be easier to handle. Also, intermediate results may be checked for correctness. In some cases, a chain of prompts may later be integrated into one prompt. Chain-of-thought Prompting was describe e.g. in “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” written by Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le and Denny Zhou in 2023 (see arXiv: 2201.11903; https://doi.org/10.48550/arXiv.2201 .11903). The sequence may be steered by an LLM-based agent. Preferably, an autonomous agent as described in 2023 by Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, Ee- Peng Lim in “Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models” can be used (see arXiv: 2305.04091 ; https://doi.org/10.48550/arXiv.2305. 04091 ). In the realm of a highly complex and diverse knowledge domain, such as the household appliances and/or cooking domain, autonomous agent systems can represent a compelling solution.

The sequence of prompts may leverage Retrieval-Augmented Generation (RAG). RAG was described in 2021 by Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, Douwe Kiela in “Retrieval- Augmented Generation for Knowledge-Intensive NLP Tasks” (see arXiv: 2005.11401 ; https://doi.org/10.48550/arXiv.2005.11401). RAG is a form of an advanced prompt including retrieval steps. The sequence may be predefined for instance by a developer or designed by an autonomous agent based on knowledge about accessible tools. In this context, a tool may be an element of a sequence, wherein a tool may be just a prompt or a prompt enriched with additional knowledge (RAG). RAG may enable individual tools within the prompt chain to access additional knowledge, contributing to a more informed and contextually aware decisionmaking process. This dynamic interaction between tools and their ability to harness supplementary information can result in a more robust and adaptive problemsolving methodology.

A prompt generally may contain input and some task description, behavior explanation or role definition. In the case of RAG, the prompt may be a combination of input data, a task description and additional internal data.

RAG may offer the advantage of building upon the insights gained in each successive step, creating a non-determ inistic path based on the outcomes of individual stages. Moreover, the application of a technique like RAG may further enhance the process. The utilization of prompt chains and RAG techniques may navigate complex scenarios efficiently.

The proposed system can be designed in two (or more) ways. Generated recipes can be purely natural language texts containing only settings in a human readable way, which can usually not be understood directly by a household appliance. Alternatively or additionally, the system may be responsible for generating sets of commands that can be sent to an appliance (machine syntax). Such output is usually not human-readable or not directly understandable for a human.

The recipe may comprise operating instructions for a generic household appliance of a predetermined type; and the household appliance is of that type. Using the LLM, the recipe’s instruction may be adapted to use functional capabilities of the household appliance that go beyond capabilities of the generic household appliance. For instance, the household appliance may comprise a traditional oven in the shape of a heatable tube section, whereas the specific household appliance in question may comprise a multi-function oven with additional capabilities for air circulation, steaming or microwaving. In some embodiments, the specific household appliance may be of a different, but comparable type to the generic household ap- pliance. This may be the case if the specific appliances is just a microwave oven, which may perform some, but not necessarily all functions of a generic oven, like browning or scalloping.

The functional capabilities may comprise an operating program of the household appliance and the operating instructions may comprise a selection and/or a parametrization of the program. The program may comprise a processing program, especially for food processing, a dishwasher program, or a treating program, especially for garment treatment.

The recipe may comprise operating instructions for a first and a second household appliance; wherein the instructions cause the household appliance to carry out tasks of both the first and second appliances in the original recipe. In this way, a recipe that is targeted at using a plurality of standard household appliances may be implemented by just one household appliance. This may make processing or treating much simpler.

Of course, a step which is intended to be carried out manually by the recipe may be translated into a step that is carried out by the household appliance. It is to be noted that the generation of steps for the appliance may require amending or reordering recipe steps or adjusting time-based relationships between steps and/or instructions.

In some embodiments, the given task may not be carried out with the household appliance alone. A converted recipe may comprise, in addition to the generated instructions, further instructions for carrying out said household task, wherein the further instructions go beyond operation of said household appliance. In some embodiments, the further instructions may not involve the specific household appliance and may be intended to be carried out by hand or by using another household appliance. Thus, a complete recipe may be provided which may be followed to carry out said household task. The recipe may relate to a preparation of food and the household appliance may comprise a kitchen appliance. The appliance may comprise a food processor, especially of an integrated type, which may function as one or several of an oven, a cooktop, a blender, a mixer, a microwave oven or a steamer.

In some embodiments, the recipe may relate to garment treatment and the household appliance comprises a garment treating machine. Such a machine may implement one or more of washing, drying, ironing, steaming, folding or freshing-up garments. The recipe may be drawn from a care label attached to a garment.

In some embodiments, Retrieval-Augmented Generation is used to generate the operating instructions for operating the household appliance. Retrieval-augmented generation (RAG) may be understood as a technique that enables generative artificial intelligence (GenAI) models to retrieve and incorporate new information. It may modify interactions with a large language model (LLM) so that the model responds to user queries with reference to a specified set of documents, using this information to supplement information from its pre-existing training data. This may allow LLMs to use domain-specific and/or updated information. RAG may improve large language models (LLMs) by incorporating information retrieval before generating responses. Unlike traditional LLMs that rely on static training data, RAG may pull relevant text from databases, uploaded documents, or web sources. Especially information about the functional capabilities of the household appliance may be embodied in text, documents or any other form in the database and may be retrieved in the sense of RAG, such that this information about the functional capabilities of the household appliance may be given as a part of the prompt to the LLM. In this way, the LLM may generate a part of a new recipe, which takes into account the functional capabilities of the household appliance.

It is preferred, that the LLM is trained to relate recipe steps to appliance capabilities. The LLM may be prompted with a sequence of prompts to generate said operating instructions. This allows for a chain of thoughts, which may, e.g., mean that the answer to a first question leads to a second question, which is then again answered by the LLM.

In some embodiments, the method comprises a further step of finding a similar recipe, which is similar to the text-based recipe, in a database. Preferably, the database is a vector database. The similarity may be, e.g., determined using a cosine similarity score. If a similar recipe is found, a conversion from a generic textbased recipe to a new appliance-specific recipe may be more efficient or the similar recipe may be even used directly as a conversion result, depending on the similarity score. This may save time and energy.

The method, especially the step of finding a similar recipe, may comprise a step of breaking down the text-based recipe into multiple text components. Conceivable text components are for example a title, ingredients, which are mentioned in the recipe, ingredient results (e.g. intermediate products during the cooking process) and/or recipe steps. For each component a similar component may be found in a database. It is preferred that each of the text components is transformed into a vector. This allows for finding for each component based on its vector a similar component in a vector database. In some embodiments, for each component and the found similar component a component similarity score is calculated. Afterwards, the calculated component similarity scores may be combined into a recipe similarity score for the similar recipe by determining a weighted sum of the component similarity scores.

The similar recipe and/or the similar components may be used as a starting point for generating operating instructions for operating the household appliance if the recipe similarity score and/or the component similarity scores are above a predetermined threshold.

In some embodiments, the method comprises a further step of finding operating instructions, which can be performed by the household appliance, based on ingredients, which are mentioned in the text-based recipe. For example, the household appliance may have a specific process for preparing broccoli. In this case, the preparation process may be already determined based on the ingredient.

In some embodiments, the method comprises the further step of analyzing a step of the recipe to separate it into an operation and an ingredient, which is processed by the operation. The operation may be transformed into an operation vector and/or the ingredient may be transformed into an ingredient vector. In a following step, it may be first tried to find an entry in the vector database, which satisfies the operation vector and the ingredient vector. Thus, it may be tried to find an entry which fits or suits the operation vector and/or the ingredient vector best. If this attempt fails, fallback strategies may be employed. For example, it may be tried to find an entry in the vector database, which satisfies the operation vector and relates to a similar ingredient. Or it may be tried to find an entry in the vector database, which satisfies the ingredient vector and relates to a similar operation. If this does not work out, either, a fallback option may be to look for an entry in the vector database, which relates to a similar operation and a similar ingredient.

In one example, a cooking step within the recipe to be analyzed may be “cutting shallots”. In this case, we would first try to find an entry in the database which directly describes for the household appliance how shallots shall be cut. If there is no exact and/or explicit entry in the database for cutting shallots, we would try to find something similar. For example, we may find an operation for cutting which relates to onions. As it is known that shallots and onions are quite similar, it may be a good option to use this operation. On the other hand, we may not be able to find the operation cutting, but only chopping, wherein the chopping explicitly relates to shallots. If we do not find the operation cutting, but only chopping, and if the chopping relates to onions, but not explicitly shallots, also this operation may be a suitable option.

In some embodiments, the method comprises a further step of determining which steps of the recipe can be performed by the household appliance and which steps of the recipe are to be performed outside of the household appliance or manually. For example, the step of peeling onions may not be performable within the household appliance, but needs to be carried out manually. The mentioned additional step may have the advantage that more realistic recipes may be proposed to the user, which need no amendment by the user, but which can be simply carried out as they are.

In a preferred embodiment, the method comprises the further step of recognizing parallelism of steps in the recipe. For example, water may already be heated in a pot, while the user is still peeling onions manually. Accordingly, in some embodiments, the method may comprise the further step of generating synthetic recipe steps, which are to be performed outside of the household appliance and/or which are to be performed manually. Preferably, the method recommends an optimal order of the steps of the recipe to the user.

Synthetic recipe steps may comprise, for example, a tool change for the household appliance. For example, a dough hook may be exchanged by a beater. Some synthetic recipe steps may relate to adding a predetermined ingredient for processing by the household appliance, to modifying the structure of an ingredient, like chopping or slicing, and/or to preparing a predetermined ingredient, like peeling or washing.

In some embodiments, the method comprises the further steps of generating a Directed Acyclic Graph of the steps of the recipe and calculating a best possible recipe route within the Directed Acyclic Graph. In this way, the user may be supported to reach the result of the recipe especially quickly. Like already mentioned, it is preferred that parallelism of the steps in the recipe is exploited to save time.

According to another aspect of the present invention, a recipe converter for operating a household appliance to carry out a predetermined household task comprises an input interface; an output interface; an interface to an LLM which is trained to perform linguistic operations on an input text; and processing means. The processing means may be adapted to determine functional capabilities of the house- hold appliance; read into the LLM a text-based, especially generic, recipe for carrying out said household task (potentially together with appliance specific data); and cause or trigger or prompt the LLM to generate, based on the text-based recipe, operating instructions for operating the household appliance; such that the household appliance can be operated on the basis of said generated instructions to carry out said household task.

In some embodiments of the recipe converter, the LLM is trained to relate recipe steps to appliance capabilities and the processing means is further adapted to prompt the LLM with a sequence of prompts to generate the operating instructions, wherein the sequence of prompts preferably leverages Retrieval-Augmented Generation.

The processing means may be adapted to carry out, completely or in part, a method disclosed herein. To this end, the recipe converter may comprise means for executing the steps of the above-mentioned embodiments of the method. The processing means may be of electronic nature and comprise a micro-computer or micro-controller, an ASIC or a similar apparatus. The method may be realized as a computer program product with program code means and may be stored on a computer readable medium. Features or advantages of the method may be applicable to a corresponding appliance or system as well as vice versa.

Non-restricting embodiments of the invention will now be discussed in more detail with reference to the enclosed drawings, in which:

Figure 1 shows a system;

Figure 2 shows a flow diagram of a method;

Figure 3 shows exemplary sequences of steps I instructions for carrying out a predetermined household task;

Figure 4 shows options for a transformation of a generic recipe into a converted recipe in machine syntax;

Figure 5 shows a flow diagram of a preferred embodiment of the method; and Figure 6 shows an example of vector addition in latent space.

Figure 1 shows an exemplary system 100. A recipe converter 105 comprises a processing unit 110 with an input interface 115, an output interface 120 and another interface 125 that is connected to a large language model (LLM) 130. The LLM may be provided locally or connected through a communication network like the internet. The LLM may be of a general type and is trained, on the basis of massive amounts of texts, to perform linguistic and/or content based operations on some input text.

The recipe converter 105 is adapted to convert a first recipe 135 for carrying out a predetermined household task into a second recipe 140 for the same task. A recipe 135, 140 may comprise steps or instructions for carrying out said task. The first recipe 135 is preferred to be generic and/or text-based. Optionally, the LLM 130 may be used to convert non-textual information like a logo, a symbol, spoken text or a moving or still picture into text form. The first recipe 135 may have been written with one or several household appliances 145, 150 in mind. Such appliances 145, 150 are preferred to be of general or simple nature. In the exemplary embodiment of Figure 1 , there is an oven 145 and a mixer 150, however, other generic appliances are also possible. However, it is often not known what functions a first appliance 145, 150 is supposed to fulfil.

The second recipe 140 comprises operating instructions for operating a predetermined specific second household appliance 155 which may be more complex, more advanced, more sophisticated or more versatile than a first household appliance 145, 150. In one embodiment, a first household appliance 145, 150 may denote a generic appliance or a type of appliance. The second household appliance 155 may fall into this category and may offer functionality that goes beyond that. The second household appliance may comprise a micro-computer and/or may be programmable. One example of the second household appliance is the Cooklt of Bosch. It is proposed to prompt or trigger the LLM 130 to generate operating instructions for processing the first recipe 135 with the help of the second household device 155. Determining instructions may be done on the basis of a description of capabilities of said second household appliance 155. Several techniques for facilitating this are disclosed herein.

It should be noted that the first recipe 135 does not have to be targeted at a predetermined first appliance 145, 150, but may rely on manual techniques for preparation. In this case, operating instructions may be generated that employ the second appliance 155 to implement a manual step of the first recipe 135. In other cases, there may be more first appliances 145, 150 mentioned in the first recipe 135 than second appliances 155 in the second recipe 140 or vice versa. In yet another case there may be the same number of appliances 145, 150, 155 mentioned in the first and second recipes 135, 140. It is preferred that the second recipe 140 mentions at least one second appliance 155 that is not mentioned in the first recipe 135.

Figure 2 shows an exemplary method for performing such a translation or extraction. In a step 205 a description of the household task at hand may be determined. In a step 210 recipe data of the first recipe 135 may be determined. Both steps may comprise user interaction and information may be drawn from one or several sources. If necessary, data may be converted into text form. This may comprise speech-to-text conversion or image recognition. Possibly, textual data is automatically translated from one language to another, alternatively later processing with an LLM may be done on mixed language data. The text-based first recipe and/or the task description may then be read into the LLM 130. The task description may be used for prompt generation and/or processing inside the LLM 130.

In a step 215, a prompt for the LLM 130 may be generated. The prompt is targeted at causing or triggering the LLM 130 to perform said generation, translation or extraction. Optionally, a sequence of prompts may be generated and a result of the LLM 130 in answer to a prompt may be processed. This may be done in batch fashion, where a series of prompts is determined before passing them on to the LLM 130 or in an interactive mode, where the output from processing a first step forms the basis for determining a second prompt. In this way, prompt generation may for instance comprise error checking, sorting and/or ordering, according to predetermined instructions.

Generating the prompt in step 215 is preferred to be based on knowledge on the second household appliance 155. In some variants the prompt may be generated based also on knowledge of the first household appliance 145, 150. Note that the present technique is not limited to converting a recipe from one device to another, but may work with just any random recipe and the use of a first household appliance 145, 150 in the first recipe 135 is not mandatory.

To this end, corresponding description data may be acquired in a step 220. Said information may also be fed into the LLM 130 for processing according to a prompt. There may be a machine-processable description of the capabilities of a household appliance 145, 150, 155 or information may be drawn from a user manual, a description or handling tips. Such information may be drawn from the internet and may come for instance from the manufacturer, a tools test or comparison article, a collection of appliance recommendations (e.g. on social media) or a cooking website.

Also, knowledge from a previous process, step or instruction may be used for generating the prompt in step 215. Corresponding information may be retrieved from a local data store in a step 225. Such historic data is preferred to be primarily used for prompt generation; the LLM 130 may keep its own history of past steps. However, external step information may be made available to the LLM 130 in some embodiments.

The prompt of step 215 may be applied to the LLM 130 in a step 230. The LLM may be of a general type that is not especially adapted to perform the generation of operating instructions described herein. However, it may be pre-trained or finetuned for such operations on the basis of predetermined training data that provides examples of such a transformation of a first recipe into a second recipe. In a step 235 a result of the processing may be provided.

Figure 3 shows an exemplary flow chart 300 for steps in a second recipe 140 for carrying out a predetermined household task. In a first variant, a sequence of a first step 305, a second step 310 and a third step 315 lead to a result 320. Each step 305-315 may be generated on the basis of an associated prompt using the LLM. In a second variant, steps 305-315 are combined in a step 325. The main difference between the first and the second approach is the possible parallelization. According to the second variant, some or all of the steps 305-315 may be done independently or concurrently and their results may then be combined.

A combination 325 may be made as an encapsulation of the steps 305-315. In the alternative, a prompt may be constructed that encapsulates or otherwise wraps up the steps 305-315. The combination 325 may especially be carried out automatically. Thus, the flow chart of figure 3 may be part of the method shown in figure 2.

Figure 4 shows options for a transformation of a generic recipe into a converted recipe in machine syntax using the described LLM, especially a GPT.

The proposed system can be designed in two (or more) ways. According to a first option, the recipe converter 105 is responsible for generating sets of instructions (machine syntax) that can be sent to a second appliance 155. In an upper portion, figure 4 shows such a generation of a second recipe 140 on the basis of a first recipe 135.

According to a second option, the provided second recipe 140 is a purely natural language text containing only settings in a human readable form. Such a generation is shown in a lower portion of figure 4. In order to use a generated operation instruction for operating the second household appliance 155, an additional converter 405 may be employed to provide a second recipe 140 in an adapted form 410, which comprises machine syntax which can be directly processed by the second appliance 155.

The described embodiments may allow users, for example, to easily and accurately adapt a generic recipe into a guided recipe suitable for a specific household appliance like the Cooklt. The method may use a large language model (LLM), retrieval augmented generation (RAG) and/or conventional algorithms to achieve the predetermined task, which as a result may produce a set of guidelines and appliance settings that follow the recipe structure designed to work with the specific household appliance. During the conversion, the system may check whether additional (synthetic) steps are necessary.

Like already mentioned, the method may use an approach called Retrieval Augmented Generation (RAG). Embodiments of the method may use RAG to find a similar recipe in a vector database, e.g., in the beginning of the conversion and later on to find suitable appliance operations that can be performed inside the appliance on specific ingredients. Each conversion may take a text-based, generic recipe as a starting point and may consist of several technical steps, which may allow the user to follow the recipe on a specific cooking appliance, which instructs the user step by step which operation needs to be performed.

The method may comprise a step of “finding a similar recipe using RAG”. This step may be designed to return a similar recipe from the list of all recipes designed for a specific household appliance. It may use a cosine similarity score. If an identical or almost identical recipe is found, then this recipe may become a result of the conversion. The system can also use the retrieved similar recipe as a basis for the converted recipe if the similarity score is above a predetermined threshold.

A multistep embedding approach may be used. To find the most similar recipe, a similarity score may be calculated separately for the recipe title, its ingredients, ingredient results and processing steps. This approach may involve breaking down each recipe into multiple text components, embedding each of them into a vector space (often called latent space), and then comparing these vectors to assess the overall similarity.

The title may be processed separately to capture its semantic meaning. An embedding model (like those based on transformers or other language models) may convert the title into a vector that represents its concepts and keywords. The list of ingredients may also be embedded on its own. This may ensure that the specific items and quantities (or the way they are described) are represented in a form that can be compared with other recipes. During the calculation of a similarity score for the ingredient results and processing steps, the description of what the ingredients become after processing and the steps involved may be considered. By embedding this text separately, the model may capture information about the method, texture, or final outcome of the dish.

For each of the components of the text-based recipe, a similarity score between the vectors of different recipes may be calculated. For example, one might use cosine similarity to determine how close two title vectors or two ingredient vectors are. After having obtained separate scores for the title, ingredients, and the pro- cessing/result description, these scores may be combined (e.g., via a weighted sum) to produce an overall similarity score, which may be called a recipe similarity score. The weights can be adjusted depending on which aspects (e.g., ingredient list versus cooking method) are deemed more critical for matching similar recipes.

The described multistep embedding approach may have advantages relating to granularity, flexibility and accuracy. By breaking down the recipe into parts, the system may be able to pinpoint similarities and differences more precisely. It may allow for different models or weighting schemes to be applied to different types of text (e.g., titles might be short and keyword-heavy, while processing instructions are usually longer and more detailed). Moreover, the method may be able to capture nuances that a single, holistic embedding might miss, leading to a more robust matching process. Overall, the multistep embedding approach may ensure that the semantic meaning of different facets of a recipe is accurately captured and compared, which can improve the effectiveness of finding the most similar recipe.

The text-based, especially generic, recipe may be converted into a structured format. It may be, e.g., parsed to extract the information from the input text into a structured format that can be used for further processing. For this purpose, LLM calls and especially LLM calls with structured output may be used. A separation of concerns approach may be chosen to improve the focus and thus performance of the LLM calls. Some of the following steps may take place in parallel to improve the latency:

The method may comprise a step of preprocessing the text-based recipe. Preprocessing the text-based recipe, e.g., by extraction and translation, may have the goal to sift through potentially noisy, unstructured input and extract only the essential parts of the text-based recipe. This step may isolate the recipe content from irrelevant data and/or translate the extracted text into English, ensuring consistency in language for further processing. This pre-processing step may be able to ensure that subsequent operations work with a clear and clean version of the recipe, reducing errors later in the pipeline.

The method may comprise a step of understanding the context, in which a step of the recipe is performed. This may comprise steps of identifying dependencies between steps of the recipe (e.g., preparing an ingredient that must be used later) and/or detecting the use of specific containers or tools (like bowls, pans, or other utensils) which might be crucial for the execution of a step of the recipe. Recognizing these relationships and required items may help in scheduling steps correctly and ensuring that the recipe is executable in a practical setting.

Moreover, the method may comprise a step of extracting and structuring high-level details of the recipe. This may comprise steps of parsing the list of ingredients that are needed for the recipe, extracting the recipe title to determine context and give identity to the dish, and/or formatting this information in a structured output so that it is easily accessible for further use. Having a clear, structured overview of ingredients and the title may lead to better organization, searchability, and presentation of the recipe.

In some embodiments, the method comprises a step of parsing each recipe step. This may comprise identifying the boundaries of each step in the text, parsing each step according to a predefined structured format (using types or a schema), and/or ensuring that the output is consistent so that each step's instructions, timings, or methods are clearly defined and machine-readable. This may help to understand and structure each individual recipe step and it may help to create a clear sequence of actions that can be followed or further processed (for example, in cooking apps or automated systems).

The method may further comprise a step of parsing step-specific ingredients. This may comprise analyzing each step of the recipe to extract the used ingredients and/or formatting the output in a structured manner so that each step's ingredient list is separate from the overall ingredients list. While a recipe may have a general list of ingredients, it may be also useful to know which ingredients are used in each specific step. The step of parsing step-specific ingredients may allow for more precise instructions and can be useful for tasks like scaling a recipe or creating shopping lists specific to each stage of the recipe.

In some embodiments, the method may comprise a step of finding appliance settings for each step in parallel. This may comprise one or more of the following steps:

• Identifying if the step falls under a "blacklist" of operations that cannot be done in the appliance. If so, abort. This is done to not waste expensive RAG calls on steps that cannot be performed inside the appliance.

• For each step, performing a multistage query that looks for suitable settings from a master file in the vector database with respect to the ingredients and operations in the step. Using this knowledge, the LLM may decide if and with what settings the step is best performed in the appliance. Attaching the correct appliance settings (level, tool, etc.) to the step and mark if it is performed outside or inside the appliance. This decision can later be overruled again.

In some embodiments, the method may comprise a step of deciding on appliance steps and/or adding synthetic steps. After defining which steps of the recipe could be appliance steps and searching for suitable settings, the method may decide which steps are done in the appliance to account for the fact, that there is only one household appliance and sometimes parallel steps are better done outside the appliance, e.g. boiling pasta. This step may use conventional algorithms to decide for an optimal recipe route, insert synthetically generated steps (e.g. for tool switching and ingredient addition), and define in which order the steps are presented to the user in the final output.

Furthermore, the method may comprise one or more of the following steps:

• Reformulating user instructions to fit a new structure. This step may use the LLM to adjust the converted recipe steps to fit the new recipe structure.

• Checking guardrails. This step may check each step against potential violations of limitations of the appliance, for example, exceeding cooking temperature or processing ingredients that are not suitable for the appliance.

• Translating the recipe to the language of user’s choice by the LLM. This step may utilize a predefined list of appliance specific naming conventions, matching brand’s tool names or operation names.

When Retrieval-Augmented Generation is used, the method may try to find the best fitting information from a set of documents I a source of information and attach this information as context to an LLM call. Here, the information may come from a master file of ingredient operations, a body of knowledge that defines for many ingredients and operations (boiling, chopping, etc.) under what conditions and with which settings they can be done in the appliance. Documents for RAG may be stored in vector forms, called embeddings, and may be retrieved via cosine similarity. In some embodiments, the method may comprise a step of selecting the best route through a recipe by analyzing steps and tool usage, using a weighted scoring system. This step may use a Directed Acyclic Graph (DAG) as a mean to calculate the best possible recipe route that maximizes appliance usage, minimizes cooking time and allows for parallel operations inside and outside of the appliance. The scoring system may be highly adjustable and may allow for tweaking weights to adjust the user recipe route as much as possible. Each of the graph’s branches may represent a dependency route, wherein the next step can only be started after the current step has been completed.

The method may assign weights to the operations to meet the criteria (maximize appliance operations, minimize time, parallelism). While the appliance operations may add value to the final score, tool switches and ingredient switches may subtract values from the final score as each of those operations takes time. Finally, the user route may be selected.

Figure 5 shows a flow diagram of a preferred embodiment of the method. In a step 505 of the illustrated method 500, a user inputs a text-based generic recipe or at least inputs a reference like a URL to the text-based generic recipe. In step 510 the method determines whether the input already is the text-based recipe or whether the input is only a reference to the recipe. If the input is already the textbased recipe, the text is given to step 520. If the input is only a reference, the real text-based recipe is retrieved in step 515, which is then given to step 520. Thus, after these steps, it is assured that the text-based recipe is available as a text.

The text of the recipe may be vectorized and embedded in latent space. In step 520, the method tries to find a similar recipe, which is similar to the text-based recipe, in a database. If the text of the recipe has been vectorized, the database preferably is a vector database. Retrieval-Augmented Generation may be used to find a similar recipe in step 520. In step 525, it is evaluated how similar the original text-based recipe is to the recipe, which was found in the database. The similarity may be determined using a cosine similarity score. If both recipes are highly similar, the method continues with step 530, in which guardrails are checked. Thus, in step 530, each step of the recipe may be checked against potential violations of limitations of the household appliance. In step 535, the recipe may be translated into the language of user’s choice. Preferably the steps 530 and 535 are performed with the help of the LLM. In step 540 the new recipe is given to the user or the household appliance. Of course, the household appliance can be directly operated on the basis of the instructions of the new recipe, i.e. the new recipe is specifically adapted to the functional capabilities of the household appliance.

If it is determined in step 525 that the original text-based recipe and the recipe, which was found in the database, exhibit only a low similarity, the method continues with step 545, in which the text-based recipe is parsed and broken down into multiple text components. This breakdown is preferably performed with the help of the LLM.

Preferably the text components are transformed into vectors. For example, operations may be transformed into operation vectors and ingredients may be transformed into ingredient vectors. Figure 6 shows an example 600 to illustrate how the addition of vectors in a latent space may work. Please note that for the sake of simplicity the latent space is illustrated as three-dimensional. Of course, the latent space may have thousands of dimensions (or even more). In figure 6(a), an ingredient vector 601 is shown, which points to a point in latent space 611 , which represents “pasta”. In figure 6(b), an operation vector 602 is depicted, which points to a point in latent space 612, which represents the operation “boiling”. As illustrated in figure 6(c), an addition of the ingredient vector 601 and the operation vector 602 leads to the point in latent space 613, which represents “boilded pasta”.

Step 550 in figure 5 tries to find for each text component (which was preferably transformed into a vector) a similar component in the database. In this database, preferably operations and ingredients can be found, which are already in a syntax which conforms with the functional capabilities of the intended household appli- ance. Thus, in this step Retrieval-Augmented Generation is used. Using one or more calls to the LLM, a raw recipe is generated, which conforms with the functional capabilities of the intended household appliance.

Of course, while parsing the original text-based recipe, steps of the recipe may be found, which cannot be executed by the intended household appliance (like the Cooklt), but need to be executed, e.g., manually or with a completely different household appliance. For example, it may be necessary to peel onions, which is preferably performed manually. Therefore, in step 555 it is determined whether any step needs to be performed outside of the household appliance and/or manually. If this should be the case, the method branches to step 560, in which at least one synthetic step is generated, before the method continues with step 565. If no step is found in step 555, which needs to be performed outside of the household appliance or manually, step 555 directly jumps to step 565.

After defining which steps of the recipe can be appliance steps and searching for suitable settings, step 565 optimizes the generated raw recipe. Step 565 may determine, which steps are to be performed in the household appliance to account for the fact, that there is only one household appliance and sometimes parallel steps are better done outside the appliance, e.g. boiling pasta. Step 565 may use conventional algorithms to decide on an optimal route to work through the recipe, insert synthetically generated steps (e.g. for tool switching and ingredient addition), and define in which order the steps are presented to the user in the final output. Moreover, step 565 may reformulate user instructions to fit to the new recipe structure. Step 565 preferably uses the LLM to accomplish its task. Afterwards, step 565 branches to step 530, where the guardrails are checked. Once the new recipe, which is optimized for the specific household appliance, is ready, it is preferably stored in the database, such that it can be found by other users as a similar recipe. In this way, over time the database is populated by more and more recipes, which are optimized for the specific household appliance. This saves effort and energy in the long run and may make the conversion of recipes faster. Some embodiments of the described method may have the following advantages:

• The method may allow users to cook whatever they want and convert arbitrary recipes into optimized recipes for a specific household appliance like the Cooklt of Bosch. Restrictions and safety issues may be considered.

• The method may actively determine which tasks should be performed outside of Cooklt (e.g., manual preparation or using an oven or cooktop) to ensure the best results and convenience. The method may intelligently assign steps between Cooklt and external appliances.

• The method may optimize the recipe step sequence with regards to parallel actions based on appliance characteristics, such as preheating the oven at the right point in time, while using Cooklt. It may identify synergies within recipes, such as chopping onions and garlic together in Cooklt, improving convenience and speed.

• The method may allow to expand the Cooklt recipe database with minimal effort, while it may avoid redundant recipe generation by recognizing when a similar Cooklt recipe already exists.

• The method may efficiently balance LLM calls with traditional algorithms, reducing unnecessary queries and optimizing costs. The integration of traditional algorithms may ensure better control over the quality of the output rather than relying solely on LLMs, especially when safety issues are considered.

• Knowledge updates may not require LLM retraining. For example, an Excel sheet combined with Retrieval-Augmented Generation (RAG) may allow for flexible and efficient updates.

• The method may optimize recipes based on different user preferences, such as convenience (fewer tool changes), speed (shortest cooking time) or perfection (optimal result). Some embodiments of the method may allow users to personalize recipes themselves based on these factors.

• Of course, the method may be applied to other appliance categories, beyond Cooklt, like ovens or cooktops. Reference list (as part of the description)

100 system

105 recipe converter

110 processing unit

115 input interface

120 output interface

125 LLM interface

130 LLM

135 first recipe

140 second recipe

145 first household appliance

150 first household appliance

155 second household appliance

200 method

205 task description

210 recipe data

215 prompt

220 household appliance and task specific knowledge

225 knowledge from previous steps

230 pre-trained I fine-tuned LLM

235 result

300 flow chart

305 first step

310 second step

315 third step

320 result

325 combination

405 converter

410 second recipe in machine syntax

500 embodiment of method

505 user input regarding recipe 510 only reference to recipe?

515 retrieve recipe

520 find similar recipe in database

525 determine similarity between text-based recipe and similar recipe

530 check guardrails

535 translation

540 give new recipe to user

545 parse recipe and break down into multiple components

550 try to find for each component a similar component in the database

555 is there any step to be performed outside of appliance or manually?

560 generate synthetic step

565 optimizes the generated raw recipe

600 example of vector addition in latent space

601 ingredient vector

602 operation vector

611 point in latent space representing “pasta”

612 point in latent space representing “boiling”

613 point in latent space representing “boiled pasta”

Claims

1. Method (200, 500) for operating a household appliance (155) to carry out a predetermined household task, the method comprising steps of:

- determining (220) functional capabilities of the household appliance (155);

- providing an LLM (130) that is trained to perform linguistic operations on an input text;

- reading (210) into the LLM (130) a text-based, especially generic, recipe for carrying out said household task; and

- causing (215, 230) or triggering or prompting the LLM (130) to generate, based on the text-based recipe, operating instructions for operating the household appliance (155);

- such that the household appliance (155) can be operated on the basis of said generated instructions to carry out said household task.

2. Method (200, 500) according to claim 1 , wherein Retrieval-Augmented Generation is used to generate the operating instructions for operating the household appliance.

3. Method (200, 500) according to one of the above claims, wherein the LLM (130) is trained (230) to relate recipe steps to appliance (155) capabilities and is prompted (215) with a sequence of prompts to generate said operating instructions.

4. Method (200, 500) according to one of the above claims, wherein the method comprises a further step of

- finding a similar recipe (520), which is similar to the text-based recipe, in a database, especially in a vector database;

- wherein similarity is determined preferably using a cosine similarity score.

5. Method (200, 500) according to claim 4, wherein the step of finding a similar recipe comprises steps of - breaking down the text-based recipe into multiple text components (545), especially a title, ingredients, ingredient results and/or recipe steps;

- transforming each of the text components into a vector;

- finding for each component, especially based on its vector, a similar component in the vector database (550);

- calculating for each component and the found similar component a component similarity score; and

- combining the calculated component similarity scores into a recipe similarity score for the similar recipe by determining a weighted sum of the component similarity scores.

6. Method (200, 500) according to claim 4 or 5, wherein the similar recipe and/or the similar components are used as a starting point for generating operating instructions for operating the household appliance if the recipe similarity score and/or the component similarity scores are above a predetermined threshold.

7. Method (200, 500) according to one of the above claims, wherein the method comprises a further step of finding operating instructions, which can be performed by the household appliance, based on ingredients, which are mentioned in the text-based recipe.

8. Method (200, 500) according to one of the above claims, wherein the method comprises the further steps of a. Analyzing a step of the recipe to separate it into an operation and an ingredient, which is processed by the operation (545); b. transforming the operation into an operation vector; c. transforming the ingredient into an ingredient vector; d. trying to find an entry in the vector database, which satisfies the operation vector and the ingredient vector; e. if step d fails, trying to find an entry in the vector database, which satisfies the operation vector and relates to a similar ingredient; f. if step e fails, trying to find an entry in the vector database, which satisfies the ingredient vector and relates to a similar operation; and g. if step f fails, trying to find an entry in the vector database, which relates to a similar operation and a similar ingredient.

9. Method (200, 500) according to one of the above claims, wherein the method comprises a further step of determining which steps can be performed by the household appliance and which steps are to be performed outside of the household appliance or manually (555), wherein preferably for each step, which needs to be performed outside of the household appliance or manually, a synthetic step is generated (560).

10. Method (200, 500) according to claim 9, wherein a synthetic step comprises at least one of

- a tool change for the household appliance;

- adding a predetermined ingredient for processing by the household appliance;

- modifying the structure of an ingredient, especially chopping or slicing; and

- preparing a predetermined ingredient, especially peeling or washing.

11 . Method (200, 500) according to one of the above claims, wherein the method comprises the further steps of

- Recognizing parallelism of steps in the recipe; and

- Recommending an optimal order of the steps of the recipe to a user.

12. Method (200, 500) according to one of the above claims, wherein the method comprises the further steps of

- Generating a Directed Acyclic Graph of the steps of the recipe and

- Calculating a best possible recipe route within the Directed Acyclic Graph.

13. Method (200, 500) according to one of the above claims, wherein the functional capabilities comprise an operating program of the household appliance (155) and the operating instructions comprise a selection and/or a parametrization of the program.

14. Recipe converter (105) for operating a household appliance (155) to carry out a predetermined household task, the recipe converter comprising:

- an input interface (115);

- an output interface (120);

- an interface (125) to an LLM (130) which is trained to perform linguistic operations on an input text;

- processing means (110) which are adapted to:

- determine functional capabilities of the household appliance (155);

- read into the LLM (130) a text-based, especially generic, recipe for carrying out said household task; and

- cause or trigger or prompt the LLM (130) to generate, based on the textbased recipe, operating instructions for operating the household appliance (155);

15. Recipe converter (105) according to claim 14, wherein the LLM (130) is trained (230) to relate recipe steps to appliance (155) capabilities and the processing means (110) is further adapted to prompt (215) the LLM (130) with a sequence of prompts to generate said operating instructions, wherein the sequence of prompts preferably leverages Retrieval-Augmented Generation.