CN114332685A

CN114332685A - Dish preparation monitored control system

Info

Publication number: CN114332685A
Application number: CN202111515280.6A
Authority: CN
Inventors: 陈志良; 申思涵; 郦悦华
Original assignee: Agricultural Bank of China
Current assignee: Agricultural Bank of China
Priority date: 2021-12-13
Filing date: 2021-12-13
Publication date: 2022-04-12

Abstract

The embodiment of the invention discloses a dish making monitoring system, which comprises an ordering module, a dish getting module, a video action recognition module and a dish process display module, wherein the ordering module is used for ordering dishes; the ordering module is used for sending the dish ordering information corresponding to each dish to the dish getting module and the dish progress display module; the dish getting module is used for generating dish matching information after receiving the dish getting information of the cook; the video action recognition module is used for determining the making state information of the dish corresponding to the dish matching information according to the dish making video; and the dish progress display module is used for determining the dish progress state of each dish according to the dish order information, the dish matching information or the making state information, determining the order return processing result according to the dish progress state when receiving the order return information input by the user, and displaying the dish progress state and/or the order return processing result of each dish. The problem that a user cannot timely solve the order-ordering dish making progress and is difficult to return orders for the order-ordering dishes is solved.

Description

Dish preparation monitored control system

Technical Field

The embodiment of the invention relates to the technical field of service monitoring, in particular to a dish making monitoring system.

Background

In a living scene that a customer goes to a restaurant for dinner, the user is often difficult to know the making progress of dishes in real time after ordering operation, and the dish processing process of a kitchen is completely invisible to the customer, so that the customer often has a question about whether the dishes are done at present, the trust between the customer and the restaurant is low, and the dining experience is poor.

At present, some restaurants meet the requirements of customers on monitoring the dish making process by adopting an open transparent kitchen, but the real-time monitoring on the whole processing process of the dishes ordered by the individual is still difficult due to the kitchen area and arrangement. Even if a real-time video of a dish processing process is provided for a user through a specific camera arranged in a chef working area, the user still has difficulty in determining the processing progress of the dish due to the complex dish cooking process. The specific processing progress still needs a restaurant waiter to supervise and inquire a kitchen room for many times to inform the customer, so that the dining experience of the customer is reduced, and the loss of the customer source is caused.

Disclosure of Invention

The invention provides a dish making monitoring system, so that a user can determine the real-time making progress of ordered dishes, and the credibility and dining experience of customers on a restaurant are improved.

In a first aspect, an embodiment of the present invention provides a dish making monitoring system, including: the system comprises a ordering module, a dish getting module, a video action recognition module and a dish process display module;

the ordering module is used for receiving ordering information input by a user, determining at least one dish according to the ordering information, and sending ordering information of the dish corresponding to each dish to the dish obtaining module and the dish progress display module;

the dish getting module is used for generating dish matching information according to the corresponding relation between the dish getting information and each piece of dish ordering information after receiving the dish getting information of a cook, and sending the dish matching information to the video action recognition module and the dish progress display module;

the video action identification module is used for acquiring a dish making video according to the dish matching information, determining making state information of a dish corresponding to the dish matching information according to the dish making video, and sending the making state information to the dish process display module;

the dish progress display module is used for determining the dish progress state of each dish according to the dish order information, the dish matching information or the making state information, determining a dish return processing result according to the dish progress state when receiving the dish return information input by the user, and displaying the dish progress state and/or the dish return processing result of each dish.

The embodiment of the invention provides a dish making monitoring system, which comprises: the system comprises a ordering module, a dish getting module, a video action recognition module and a dish process display module; the ordering module is used for receiving ordering information input by a user, determining at least one dish according to the ordering information, and sending the ordering information of the dishes corresponding to all the dishes to the dish getting module and the dish progress display module; the dish obtaining module is used for generating dish matching information according to the corresponding relation between the dish obtaining information and each piece of dish ordering information after receiving the dish obtaining information of a cook, and sending the dish matching information to the video action recognition module and the dish progress display module; the video action identification module is used for acquiring a dish making video according to the dish matching information, determining making state information of a dish corresponding to the dish matching information according to the dish making video, and sending the making state information to the dish process display module; and the dish progress display module is used for determining the dish progress state of each dish according to the dish order information, the dish matching information or the making state information, determining the order return processing result according to the dish progress state when receiving the order return information input by the user, and displaying the dish progress state and/or the order return processing result of each dish. By adopting the technical scheme, after the dish ordered by the user is taken by the cook, dish matching information containing the corresponding relation between the cook and the dish is generated, the dish making video of the dish is obtained in real time by the image acquisition device corresponding to the cook making the dish according to the dish matching information, the making state information of the dish is determined according to the dish making video, so that the real-time making state of the dish is fed back to the ordering user, and the order returning operation can be executed on the dish meeting the order returning condition according to the making state information of the dish when the order returning information input by the user is received. The problem of the user can't solve the single dish preparation progress of ordering in real time, be difficult to order the single dish of order is returned is solved, the accuracy of the dish preparation progress that the customer obtained has been promoted, the flexibility that the user ordered is improved, the tolerance of customer to the dish of long preparation cycle has been promoted, the customer has been improved to the trust level and the dining experience of dining room.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.

Fig. 1 is a schematic structural diagram of a dish making monitoring system according to a first embodiment of the present invention;

fig. 2 is a schematic structural diagram of a dish making monitoring system according to a second embodiment of the present invention;

FIG. 3 is a diagram illustrating an architecture of a CNN-Transformer model according to a second embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings. It should be understood that the embodiments described are only a few embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.

When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the application, as detailed in the appended claims.

In the description of the present application, it is to be understood that the terms "first," "second," "third," and the like are used solely to distinguish one from another and are not necessarily used to describe a particular order or sequence, nor are they to be construed as indicating or implying relative importance. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate. Further, in the description of the present application, "a plurality" means two or more unless otherwise specified. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Example one

Fig. 1 is a schematic structural diagram of a dish making monitoring system according to an embodiment of the present invention, where the dish making monitoring system includes: the menu ordering system comprises an ordering module 10, a dish getting module 11, a video action recognition module 12 and a dish progress display module 13.

The ordering module 10 is configured to receive ordering information input by a user, determine at least one dish according to the ordering information, and send ordering information of the dishes corresponding to the dishes to the dish retrieving module 11 and the dish progress displaying module 13;

the dish getting module 11 is configured to generate dish matching information according to a corresponding relationship between the dish getting information and each piece of dish ordering information after receiving the dish getting information of the chef, and send the dish matching information to the video action recognition module 12 and the dish progress display module 13;

the video action recognition module 12 is configured to obtain a dish making video according to the dish matching information, determine making state information of a dish corresponding to the dish matching information according to the dish making video, and send the making state information to the dish progress display module 13;

and the dish progress display module 13 is configured to determine a dish progress state of each dish according to the dish order information, the dish matching information or the making state information, determine a return processing result according to the dish progress state when receiving the return information input by the user, and display the dish progress state and/or the return processing result of each dish.

In this embodiment, the ordering module 10 may be specifically understood as a module for assisting a user in ordering dishes, the ordering module 10 may be implemented by software and/or hardware, for example, the ordering module 10 may be a tablet computer including an electronic menu, or an applet for assisting the user in ordering dishes after scanning a two-dimensional code for a customer, and the presentation form of the ordering module 10 is not specifically limited in the embodiment of the present invention. One or more ordering modules 10 may be included in a food preparation monitoring system, and ordering information entered by the same ordering module 10 may be understood as ordering information made by the same customer.

In the present embodiment, the dish retrieving module 11 can be specifically understood as a module for assisting a cook in retrieving a single dish ordered by a user and for matching the single dish ordered by the user with a cook, so as to clarify the matching relationship between the dish and the cook. For example, the dish retrieving module 11 may be implemented by software and/or hardware, such as a tablet computer disposed in a kitchen dish retrieving area, and optionally, the number of the dish retrieving modules 11 may be one or more, that is, multiple cooks may share the same dish retrieving module 11, or each cook may be configured with an independent dish retrieving module 11, which is not limited in this embodiment of the present invention.

In this embodiment, the video motion recognition module 12 may be specifically understood as a module that performs motion recognition on a production video acquired in a dish production process, and specifies a stage of dish production according to a cook cooking motion in the video. The dish progress display module 13 may be specifically understood as a module that displays the dish making progress ordered by the user to the user and assists the user in returning the order of the ordered dish. Optionally, the dish progress displaying module 13 may be implemented by the same software and/or hardware as the ordering module 10, for example, a tablet computer disposed at the ordering user, or an applet used for assisting the ordering user after scanning the two-dimensional code, and the like.

In the present embodiment, the ordering information may be understood as information received by the ordering module 10, which is formed by the user clicking the corresponding dish or inputting the number of the dish, and is used to represent the dish ordered by the user. The menu ordering information can be understood as information for representing the corresponding relation between the menu ordered by the user and the menu ordering user, the menu ordering table number or the menu ordering device, namely, the menu ordering information can be used for acquiring the type of the menu to be made and executing the menu ordering module 10 of the menu ordering, and further, the making state of the menu ordered menu item can be clearly displayed at a position and a display object. The dish retrieval information is understood to be information that is entered by the cook and that characterizes the cook's desire to retrieve the dish. The dish matching information may be understood as information for characterizing a correspondence between a cook making the dish and the dish. The dish making video can be understood as a video in the dish making process collected by a camera arranged in a working area of a chef. The preparation status information may be understood as information determined according to the degree of completion of the dish during the preparation process. The dish progress information may be understood as information for characterizing a process of making a dish ordered by a user. The order return information may be understood as information input by the user wishing to cancel one or more ordered dishes. Optionally, the dish progress state at least includes one of ordered dish, picked dish, cooked dish and filled dish, and the user can know the real-time making progress of the ordered dish through the displayed dish progress state.

Specifically, the ordering module 10 disposed on the dining table receives ordering information input by the user, the ordering information may include one or more dishes ordered by the user, and the ordering information identifies dish information used for representing identities of different dishes, wherein the dish information may be preset number information or legend information including dish display diagrams. And generating menu ordering information according to the incidence relation between the menu information of each menu and the menu ordering module 10, and sending the menu ordering information to the menu getting module 11 and the menu process display module 13. The dish getting module 11 can determine the number and type of dishes to be made and the user who places the order after receiving the dish order information sent by the ordering module 10, after the kitchen chef determines the dishes that the kitchen chef can make, the dish getting module 11 inputs the dish getting information, the dish getting information comprises the information of the dishes that the kitchen chef wants to get the dishes and the information for representing the identity of the kitchen chef, the dish getting module 11 compares the dish information in the dish getting information with the received dish order information, generates dish matching information comprising the information of the kitchen chef, the information of the dishes that the kitchen chef gets and the user information of the dishes that are ordered according to the corresponding relation, and sends the dish matching information to the video action recognition module 12 and the dish progress display module 13. After receiving the dish matching information, the video action recognition module 12 determines the making position of the dish according to the cook information in the dish matching information, obtains the dish making video of the dish from the video acquisition device that takes video shooting of the making position, further recognizes the cooking action of the cook in the dish making video, determines the making state of the dish, further generates making state information of the corresponding dish by combining with the user information of the dish that orders in the dish matching information, and sends the making state information to the dish progress display module 13. The dish progress display module 13 determines the dishes ordered by the user and the dish progress states of the dishes according to the received dish ordering information, the dish matching information and the making state information, displays the dish progress states of the dishes to the user in real time, determines the order-returning processing result according to the dish progress states of the dishes needing to be returned by the user in the order-returning information when receiving the order-returning information input by the user for the ordered dishes, and displays the order-returning processing result corresponding to the order-returned dish and the dish progress states of the other dishes to the corresponding user at the same time.

Example two

Fig. 2 is a schematic structural diagram of a dish making monitoring system according to a second embodiment of the present invention. According to the technical scheme of the embodiment, on the basis of the technical scheme, the dish obtaining module 11, the video action recognition module 12 and the dish progress display module 13 are further refined, meanwhile, a dish making monitoring system is further refined, and the dish making monitoring system further comprises a video storage module 14 and an increment training module 15.

To a dish preparation monitored control system that this application provided, wherein, dish receive module 11 includes:

a cook information receiving unit 111 configured to receive dish pickup information of a cook, where the dish pickup information includes cook information and target dish information picked up by the cook;

a matching information generating unit 112, configured to determine a target dish according to a comparison result between the target dish information and each item ordering information, and generate dish matching information of the target dish according to an association relationship between the chef information and the target dish;

and the matching information sending unit 113 is configured to send the dish matching information to the video action recognition module 12 and the dish progress display module 13.

In the present embodiment, the cook information receiving unit 111 can be specifically understood as a unit disposed at the back of the restaurant for displaying the single dishes ordered by the user for the query of the cook, and receiving the input or selection information of the cook. The target dish information may be understood as dish information that the cook receives the dish to be cooked, which may include a dish number or other information characterizing the identity of the dish.

Specifically, the cook can check the menu information sent by the user to the menu getting module 11 in the cook information receiving unit 111, and further clarify the menu to be made, after selecting the menu desired to be made, the cook first inputs the cook information representing the identity of the cook to the cook information receiving unit 111, and meanwhile, the cook information receiving unit 111 determines the information of the menu selected by the cook as the target menu information according to the clicking or entering operation of the cook, and combines the cook information and the target menu information to serve as the menu getting information of the cook.

In the present embodiment, the matching information generating unit 112 may be specifically understood as a unit that matches the chef information with the menu order information of the user menu order.

Specifically, the matching information generating unit 112 determines target dish information to be picked up by a chef from the received dish picking-up information, matches the target dish information with the received one or more pieces of dish ordering information, determines a dish that is successfully matched as a target dish, further forms an association relationship between the chef information in the received dish picking-up information and the target dish, and determines the association relationship as dish matching information of the target dish.

In this embodiment, the matching information sending unit 113 may be specifically understood as a unit that sends information to other modules in the dish making monitoring system. The matching information transmitting unit 113 may transmit the dish matching information generated by the matching information generating unit 112 to the video action recognition module 12 and the dish progress presentation module 13. Optionally, the matching information sending unit 113 may perform information transmission through WiFi, mobile communication, and/or bluetooth transmission, and the information transmission method is not limited in the embodiment of the present invention.

To a dish preparation monitored control system that this application provided, wherein, video action identification module 12 includes:

the video acquisition unit 121 is configured to determine a target camera according to the cook information and acquire a dish making video acquired by the target camera;

the action recognition unit 122 is used for inputting the dish making video into a pre-trained action recognition model and determining making state information of a target dish according to an output result of the action recognition model;

a status information sending unit 123, configured to send the making status information to the dish progress display module 13;

the motion recognition model is a model established by a convolutional neural network-conversion algorithm.

In this embodiment, the video acquiring unit 121 may be specifically understood as a unit that is connected to each camera provided in the kitchen for shooting the cook food preparation area in a communication manner, and can acquire the video of the shooting area of each camera according to the requirement. The target camera may be understood as a camera of the target dish making area.

Specifically, each cook in the restaurant often has a fixed cooking position, that is, there is a certain corresponding relationship between the cook and the dish making area, so the video acquisition unit 121 can determine the dish making area of the cook according to the information of the cook in the received dish matching information, and then determine the camera used for shooting the dish making area and use it as the target camera, the number of the target cameras can be one or more, and the video acquired by the target camera is acquired and used as the dish making video of the target dish.

In this embodiment, the motion recognition unit 122 may be specifically understood as a unit integrated with a pre-trained motion recognition model for performing motion recognition on the input video and generating a production state label. The motion recognition model is a model established by using a convolutional neural network-transform (CNN-Transformer) algorithm. Further, a Convolutional Neural Network (CNN) is a feedforward type Neural Network composed of an input layer, a hidden layer and an output layer, a transform model is a Neural Network model using a Self-Attention mechanism (Self-Attention) to replace a Long Short Term memory artificial Neural Network (LSTM) structure, a CNN-transform model is understood as a model that extracts spatial information of a video using a Convolutional Neural Network to obtain a video feature map, converts the feature map into feature vectors through operations such as pooling, and inputs the feature vectors to the transform model for motion recognition using the Self-Attention mechanism, and fig. 3 is an architectural example of a CNN-transform model provided in the second embodiment of the present invention.

In this embodiment, the motion recognition model may be specifically understood as a neural network model for performing motion capture and recognition on the video input therein, recognizing the motion of the dish making process in the video according to the pre-training, and outputting the corresponding motion recognition tag.

Specifically, the action recognition unit 122 preprocesses the received dish making video, inputs the processed dish making video into a pre-trained action recognition model, recognizes the dish making progress action in the dish making video by the action recognition model, uses the output action recognition tag as the output result of the action recognition model, determines the state of the cook cooking dish according to the output result, and generates making state information including the cooking state of the target dish and the incidence relation between the target dish and the cooking state.

Further, the action recognition unit 122 is specifically configured to:

carrying out specific target identification on the dish making video, and determining a candidate video according to a specific target identification result;

inputting the candidate video into a pre-trained action recognition model, and determining the making state of the target dish according to the output result;

and intercepting a target image for indicating the production state from the candidate video, and generating production state information of the target dish by combining the production state and the target image.

Specifically, the action recognition unit 122 may perform specific target recognition on the received dish making video through a target detection network to remove useless parts in the dish making video, determine the dish making video with the specific target recognized as a candidate video, input the candidate video into a pre-trained action recognition model, determine the making state of the target dish at the dish making video acquisition time according to an output result of the candidate video, further intercept a frame from the candidate video as a target image according to the determined making state, the target image may be used to indicate the making state of the target dish, and combine the making state, the target image and the relationship between the target image and the target dish to generate making state information of the target dish. Optionally, the target detection network for identifying the specific target in the embodiment of the present invention may be an R-CNN-Two-Stage target detection algorithm. Alternatively, the preparation state of the target dishes may be pre-treatment, cooking, dishing and the like.

In the embodiment of the invention, the action of the dish making process is identified by adopting the action identification model trained by the CNN-Transformer algorithm, and the action identification is carried out by using an attention mechanism, so that the calculation complexity in the action identification process is reduced, the accuracy of the action identification is improved on the basis of reducing the training time, and the performance is improved by realizing parallelism.

In this embodiment, the status information sending unit 123 may be specifically understood as a unit that sends information to other modules in the dish making monitoring system. The status information transmitting unit 123 may transmit the production status information generated by the motion recognition unit 122 to the dish progress presentation module 13. The optional status information sending unit 123 may perform information transmission through WiFi, mobile communication and/or bluetooth transmission, and the information transmission method is not limited in the embodiment of the present invention.

To a dish preparation monitored control system that this application provided, wherein, dish process display module 13 includes:

the process state determining unit 131 is configured to determine a dish process state of each dish according to the order information of each dish, the matching information of each dish, or the making state information of each dish; wherein the dish progress state at least comprises one of ordering, getting up, cooking and holding;

the order-returning processing unit 132 is used for receiving order-returning information input by a user, determining the dish progress state of the order-returning dish corresponding to the order-returning information, and determining an order-returning processing result of the order-returning dish according to the dish progress state and a preset order-returning standard;

the progress status display unit 133 is configured to display a progress status and/or a return processing result of each dish.

In this embodiment, the process status determination unit 131 can be specifically understood as a unit for determining the process of making the order menu ordered by the user.

Specifically, the progress state determining unit 131 may receive information sent by other modules in the dish making monitoring system, for example, determine a dish ordered by the user according to the menu ordering information sent by the ordering module 10, adjust the dish progress state of the corresponding dish according to the menu ordering information to be ordered, determine a dish picked up by the cook according to the dish matching information sent by the dish picking module 11, further update the progress state of the corresponding dish from ordered to picked up, similarly determine the making progress of the cook on the dish according to the making state information sent by the video action identifying module 12, and further update the dish progress state.

Further, the process state determining unit 131 is specifically configured to:

if receiving menu ordering information of the dishes, determining the dish progress state of the dishes as the menu ordering;

if the dish matching information of the dish is received, updating the dish process state of the dish to be received;

and if the making state information of the dish is received, determining the making state of the dish according to the making state information, and updating the dish progress state of the dish to be cooked or filled according to the making state.

Further, the method for updating the dish progress state of the dish to be cooked or filled according to the making state comprises the following steps:

if the cooking state is the cooking state, updating the dish progress state of the dish to be cooked;

and if the making state is the dish holding state, updating the dish progress state of the dishes to the dish holding state.

Specifically, for a dish that can be subjected to process state determination, when the process state determination unit 131 receives the corresponding dish ordering information, the dish may be regarded as a dish ordered by the user, and at this time, the process state of the dish is determined as having been ordered; when the process state determining unit 131 receives the dish matching information corresponding to the dish, it may be considered that the dish is already received by the cook in the back kitchen, that is, the cooking operation is to be started, and at this time, the dish process state of the dish is determined to be received; when the process state determining unit 131 receives the preparation state information corresponding to the dish, the dish may be considered to be cooked by the cook, the preparation state information may include the preparation state of the dish, the preparation state may include that the dish is cooked or the dish is completely cooked and placed, and at this time, the process state of the dish may be updated to be cooked or placed according to the determined preparation state.

In the present embodiment, the order-returning processing unit 132 can be specifically understood as a unit for returning the order of the ordered dishes by the user. The order-returning information may be understood as information formed by clicking a corresponding dish or inputting a number of the dish on the ordered dish interface by the user to represent the dish that the user desires to return the order. The preset order-returning standard can be understood as a preset condition for judging whether the dish can be returned or not according to the dish making process. For example, the dish is still in the raw material stage before being put into the pot, and even if the dish is cancelled, no great loss of interest is generated in the restaurant, so the preset menu returning standard may be that the dish progress state is not cooked or is filled, and the preset menu returning standard may be adaptively adjusted according to actual situations, which is not limited in the embodiment of the present invention.

Further, the order rejection processing unit 132 is specifically configured to:

receiving order returning information input by a user, and determining the dish progress state of the order returning dish corresponding to the order returning information;

if the dish process state of the dish returning the order is cooked or filled, determining that the result of the order returning processing of the dish returning the order is the failure of the order returning; otherwise, determining that the order returning processing result of the order returning dish is that the order returning is successful, and sending dish canceling information to the dish getting module 11 so that the cook corresponding to the order returning dish stops cooking.

Specifically, after receiving the order-returning information input by the user, the order-returning processing unit 132 determines one or more order-returning dishes that the user wants to return from the order-returning information, and the progress state determining unit 131 obtains the dish progress state of each order-returning dish, and if the dish progress state of the order-returning dish is cooked or filled, it may be determined that the raw material of the dish is used, and the dish may be served within a short time during the manufacturing process, and does not satisfy the preset order-returning condition, and at this time, it is determined that the order-returning processing result of the order-returning dish is an order-returning failure; otherwise, the dish is still made by the cook, the raw materials can still be reserved for the next use, the dish is difficult to finish serving in a short time, a preset order returning condition is met, the order returning result of the order returning dish is determined to be the order returning success, and dish canceling information is sent to the dish getting module 11, so that after the dish canceling information is received by the dish getting module 11, the information that the dish is canceled to cook can be prompted to the corresponding cook according to the contained dish information and the cook information, the cook corresponding to the order returning dish stops cooking, and the waste of the raw materials is avoided.

In the present embodiment, the progress status display unit 133 can be specifically understood as a unit for displaying the status of the order making progress and/or the result of the order returning process to the user in real time.

Further, the process state display unit 133 is specifically configured to:

if the dish is not the returned dish, displaying the dish progress state of the dish;

if the dish is the returned dish and the returned dish processing result is the returned dish failure, displaying the returned dish processing result and displaying the dish progress state of the dish;

if the dish is the returned dish and the returned dish is successfully returned, the returned dish is displayed.

Specifically, if the dish displayed by the progress status display unit 133 is not the dish returned, the progress status of the dish is displayed in real time. For example, when the dish progress state of the dish is the order, the finished product schematic diagram and the word of the order can be displayed; when the dish progress state of the dish is the dish receiving state, displaying a finished product schematic diagram and a received character of the dish; and when the dish progress state of the dish is cooked or filled, displaying a target image in the dish making state information, and displaying the making state of the dish in a character form. If the dish displayed by the progress status display unit 133 is a menu returning dish and the menu returning processing result of the menu returning dish is a menu returning failure, the menu returning processing result needs to be displayed to the user to prompt the user that the dish fails to return, and the progress status of the dish is continuously displayed, so that the user can determine the real-time making progress of the dish. If the dish displayed by the progress status display unit 133 is the order returned dish and the result of the order returning process of the order returned dish is the order returning success, the result of the order returning process needs to be displayed to the user to prompt the user that the order returning of the dish is successful, and the progress status of the dish is not displayed to the user.

Further, the dish making monitoring system provided by the embodiment of the present invention further includes: a video storage module 14 and an incremental training module 15;

the video storage module 14 is used for storing the candidate video determined by the video action recognition module 12 and the reference dish making video uploaded by the chef;

and the increment training module 15 is used for carrying out increment training on the motion recognition model according to the candidate video and the reference dish making video.

In the present embodiment, the video storage module 14 can be specifically understood as a storage module for storing videos collected in the process of monitoring and producing dishes and reference videos uploaded by a cook, so as to train the motion recognition model. The reference dish making video can be understood as a video uploaded by a chef and including a new dish making method or other dish making methods, and can be used as a reference standard. The incremental training module 15 may be understood as a module for retraining the motion recognition model for motion recognition from each video stored in the video storage module 14.

Specifically, after the video storage module 14 receives the original video uploaded by the chef, sparse sampling and resolution adaptation can be performed on the original video, data are preprocessed for identification and classification, the processed video is compressed to reduce the occupied space of the video, and the compressed food is used as reference dish making video for storage. Meanwhile, the video storage module 14 may obtain the candidate video determined by the video motion recognition module 12 in real time when the dish making monitoring system is in operation, and store the candidate video. The incremental training module 15 may perform incremental training on the motion recognition model by periodically using the candidate videos and the reference dish making videos stored in the video storage module 14, so as to enhance the recognition accuracy of the motion recognition model.

In the embodiment of the invention, the sparse sampling can extract the key frames from the continuous video frames by setting a proper sampling interval to obtain effective video information, simultaneously save the time required by the subsequent processing of the video, reduce the load of a server, and reduce the redundant information brought by a high-definition image by carrying out resolution adaptation on the video subjected to the sparse sampling. By carrying out incremental training on the motion recognition model at regular time, the recognition precision of the motion recognition model can be improved, and meanwhile, recognition results corresponding to different manufacturing actions of a chef for new dishes can be introduced, so that the motion recognition model can carry out more kinds of motion recognition, and the accuracy of showing the progress state of the dishes for a user is improved.

For example, the incremental training method of the motion recognition model can comprise the following steps:

A) integrating the back chord video and the reference dish making video in the video storage module 14 into video segments with the same resolution and the same frame number and duration;

B) according to the process of dish making, dividing the actions identified in the video into three labels, namely food material pretreatment, dish cooking and dish holding;

C) and inputting the video clips and the labeled video clips into the prime motion recognition model for training until the model converges.

Optionally, the actions corresponding to the food material preprocessing tag can comprise a vegetable cutting action, a peeling action, a vegetable washing action and the like; the dish cooking actions can comprise a pan-lowering action, a stir-frying action, a ladle-turning action, a pot cover closing action and the like; the dish containing action can comprise dish discharging and containing action, dish swinging action, dish end transferring action and the like.

It should be clear that, except for the training mode of the motion recognition model which is basically the same as the above-mentioned training mode, only the existing motion recognition model needs to be replaced by the untrained CNN-Transformer model.

The embodiment of the invention provides a dish making monitoring system which comprises a ordering module, a dish getting module, a video action recognition module, a dish process display module, a video storage module and an increment training module. The method comprises the steps of identifying actions of a dish making video collected by a cook making area matched with dishes through a pre-trained action identification model, determining making state information of the dishes according to an identification result, further determining the dish process state of the dishes, so that the cooking process of the dishes can be displayed to a user in real time, determining whether the user can perform order returning operation or not according to the dish process state when the user needs to return orders, performing increment training on the action identification model at regular time through an increment training module, improving the accuracy of action identification of the action identification model, improving the accuracy of acquiring the dish making progress by the user and the flexibility of ordering by the user, avoiding the waste of raw materials and manpower caused by the order returning of the user for making the dishes, and improving the trust and dining experience of the user on a restaurant.

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A system for monitoring the preparation of a dish, comprising: the system comprises a ordering module, a dish getting module, a video action recognition module and a dish process display module;

2. The system of claim 1, wherein the dish retrieval module comprises:

the system comprises a cook information receiving unit, a display unit and a display unit, wherein the cook information receiving unit is used for receiving dish getting information of a cook, and the dish getting information comprises cook information and target dish information taken by the cook;

the matching information generating unit is used for determining a target dish according to the comparison result of the target dish information and each piece of dish ordering information and generating dish matching information of the target dish according to the incidence relation between the chef information and the target dish;

and the matching information sending unit is used for sending the dish matching information to the video action recognition module and the dish process display module.

3. The system of claim 2, wherein the video motion recognition module comprises:

the video acquisition unit is used for determining a target camera according to the cook information and acquiring a dish making video acquired by the target camera;

the action recognition unit is used for inputting the dish making video into a pre-trained action recognition model and determining making state information of the target dish according to an output result of the action recognition model;

the state information sending unit is used for sending the making state information to the dish progress display module;

4. The system according to claim 3, wherein the action recognition unit is specifically configured to:

carrying out specific target recognition on the dish making video, and determining a candidate video according to the specific target recognition result;

inputting the candidate video into the pre-trained action recognition model, and determining the making state of the target dish according to an output result;

and intercepting a target image used for indicating the production state from the candidate video, and generating production state information of the target dish by combining the production state and the target image.

5. The system of claim 1, wherein the dish progress display module comprises:

the progress state determining unit is used for determining the progress state of the dishes of each dish according to the order information of each dish, the matching information of each dish or the making state information of each dish; wherein the dish progress state at least comprises one of ordering, getting up, cooking and serving;

the order-returning processing unit is used for receiving order-returning information input by the user, determining the dish progress state of an order-returning dish corresponding to the order-returning information, and determining an order-returning processing result of the order-returning dish according to the dish progress state and a preset order-returning standard;

and the progress state display unit is used for displaying the progress state of each dish and/or the order returning processing result.

6. The system of claim 5, wherein the process state determining unit is specifically configured to:

if receiving the menu ordering information of the menu, determining the menu process state of the menu as the menu ordering;

7. The system of claim 6, wherein the updating the dish progress status of the dish to cooked or filled according to the production status comprises:

if the cooking state is the cooking state, updating the dish process state of the dish to be cooked;

and if the making state is the dish holding state, updating the dish progress state of the dish to be the dish holding state.

8. The system of claim 5, wherein the de-ordering processing unit is specifically configured to:

receiving the order-returning information input by the user and determining the dish progress state of the order-returning dishes corresponding to the order-returning information;

if the dish process state of the dish returning is cooked or filled, determining that the dish returning processing result of the dish returning is that the dish returning fails; otherwise, determining that the order returning processing result of the order returning dish is that the order returning is successful, and sending dish canceling information to the dish getting module so that the cook corresponding to the order returning dish stops cooking.

9. The system of claim 8, wherein the process state presentation unit is specifically configured to:

if the dish is a returned dish and the returned dish processing result is a returned dish failure, displaying the returned dish processing result and displaying the dish progress state of the dish;

and if the dish is a menu returning dish and the menu returning processing result is the menu returning success, displaying the menu returning processing result.

10. The system of claim 4, further comprising: the system comprises a video storage module and an increment training module;

the video storage module is used for storing the candidate video determined by the video action recognition module and the reference dish making video uploaded by the chef;

and the increment training module is used for carrying out increment training on the action recognition model according to the candidate video and the reference dish making video.