WO2022025282A1 - Learning control system - Google Patents

Abstract

Provided is a learning control system that, on the basis of information that can be reliably detected from a work site such as a cooking environment, can respond to changes in the environment as well, and is highly versatile.　A learning control system according to an embodiment of the present invention is characterized by comprising one or more of: (a) a demand prediction module that predicts sales; (b) an image recognition module that recognizes an object; (c) an audio recognition module that recognizes the state of the object using audio; (d) a recommendation module that recommends a product to a customer; and (e) a fault detection module that detects a fault state of an device, the learning control system further comprising (f) a learning module that performs machine learning on the basis of information from the one or more modules, and the modules being controlled by a system trained by the learning module.

Description

Learning control system

The present invention relates to a learning control system.

In recent years, due to the background of the shortage of employees in restaurants, there has been a request to automate the service process in restaurants, and technologies such as Patent Documents 1 to 3 have been proposed conventionally.

Patent Document 1 describes a learning device applied to cooking by a robot, which is provided with a relationship information learning unit that learns the relationship between biological information, emotional information, and environmental information by machine learning.

In Patent Document 2, the detection information detected by the sensor that monitors the cooking state is acquired, and the appropriate information indicating the information indicated by the sensor when the cooking state is appropriate is acquired from the storage means. A cooking system is described in which instruction information indicating an instruction to a cook is generated using the acquired detection information and appropriate information, and the instruction information is output to an output means.

In Patent Document 3, an appropriate cooking time is given to the cook by learning using a set of cooking name, cooking process, ingredient name to be cooked, quantity, cutting method, cut size and cooking time as a teacher data table. A cooking support system that can be instructed is described.

Japanese Unexamined Patent Publication No. 2020-017104 Japanese Unexamined Patent Publication No. 2011-058782 Japanese Patent No. 6692960

In the learning device described in Patent Document 1, three pieces of information, biological information, emotional information, and environmental information, are indispensable for machine learning of a cooking robot, but the information acquisition means for acquiring these information has a complicated configuration. In addition, it is difficult to obtain accurate information. Moreover, it is a learning device for cooking robots and cannot be applied to human cooking.

In the method described in Patent Document 2, the judgment that the cooking condition is appropriate and the content of the instruction do not take into consideration the cooking environment, but are premised on specific conditions uniformly determined. It is difficult to respond to changes in the environment at stores.

The method described in Patent Document 3 is not universally usable for various stores because it is learning under limited conditions under a specific environment.

In view of the above problems, an object of the present invention is to provide a highly versatile learning control system that can respond to environmental changes based on information that can be reliably detected from the field such as a cooking environment, and a cook. It is to eliminate the variation in cooking quality due to the above, and to share the menu and make the cooking quality uniform.

The above object of the present invention can be achieved by the following configuration. That is, the learning control system according to the first aspect of the present invention is
(A) Demand forecast module for forecasting sales,
(B) Image recognition module that recognizes objects,
(C) A voice recognition module that recognizes the state of an object by voice,
(D) A recommended module that recommends products to customers, or a recommended module
(E) A machine learning module of at least one abnormality detection module for detecting an abnormal situation of a device is provided, and further, the abnormality detection module is provided.
(F) An AI platform including a learning module that performs machine learning other than the above (a) to (e) and / or uses the information and results of at least one of the above (a) to (e). Have and
It is characterized by performing learning control using a model trained by the AI platform.

According to the embodiment of the present invention, it is possible to provide a learning control system that can respond to changes in the environment based on information that can be reliably detected from the site such as the cooking environment. As a result, there is a difference in the quality of the dish according to the difference in the skill of the cook, but by using such a learning control system, it is possible to eliminate the variation in the quality of the dish by the cook. In addition, by sharing information on cooking methods and recipes at all stores, menus can be shared and cooking quality can be made uniform.

It is a block diagram of a learning control system. It is explanatory drawing of image recognition of steak. It is explanatory drawing of the cooking process of steak. It is explanatory drawing of the image recognition result for each color component of steak. It is explanatory drawing of time series image analysis about the grilling color of the surface of a steak. It is explanatory drawing about image recognition which analyzes the balance between lean meat of steak and fat, and estimates the position to cut into equal parts by volume. It is explanatory drawing of the state of deep learning using the image of a thermography camera. It is explanatory drawing of lettuce labeling. It is explanatory drawing of the training result of the image recognition of lettuce. It is explanatory drawing of the output image of lettuce image recognition. It is explanatory drawing of the line which cuts lettuce. It is explanatory drawing of lettuce labeling. It is explanatory drawing of the training result of the image recognition of lettuce. It is explanatory drawing of the correct answer rate of lettuce image recognition. It is explanatory drawing of the output video of lettuce image recognition. It is explanatory drawing of the method of determining the cutting line which cuts the core of cabbage or lettuce. FIG. 17 is an explanatory diagram of the recognition result of sushi material.

Hereinafter, the learning control system according to the embodiment of the present invention will be described with reference to the drawings. However, the embodiments shown below exemplify a learning control system for embodying the technical idea of the present invention, and do not specify the present invention as these, but other embodiments included in the claims. It is equally applicable to the form.

[First Embodiment]
The learning control system according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 17.

FIG. 1 is a block diagram of the learning control system of the present embodiment. The AI platform 10 is connected to the cooking robot 21, the business automation AI robot 22, and the AI restaurant 23 via the information network 20, and communicates with each other.

The cooking robot 21 automates a series of cooking processes in the kitchen, and is applicable to stores and retail stores. Business automation The AI robot 22 automates simple tasks associated with cooking, and is expected to be used in factories outside stores, for example. An AI restaurant is a restaurant that is automatically managed by AI trained by machine learning, and the level of automation can be freely selected, but if automation is advanced, it is possible to achieve a fully automated restaurant. It is possible. Actually, it is often implemented as a labor-saving restaurant that is fully automated during business hours and manually performs some adjustments or work such as maintenance.

The AI platform 10 uses various analysis data including AI training for cooking robot 21, business automation AI robot 22 and AI restaurant 23, collecting big data and training AI, and using various analysis data including AI output. It can provide information for the management and marketing of stores, factories and restaurants. For example, in the AI platform 10, deep learning is performed based on big data, and a trained model (trained model) is provided to the cooking robot 21, the business automation AI robot 22, the AI restaurant 23, and the like, thereby providing the AI restaurant 23. Image recognition, automatic cooking, cooking assistance, etc. using a trained model are possible in stores such as.

For example, using the image data of customers collected at each store or restaurant, various data including these image data are analyzed on the AI platform in order to use it for the congestion situation in the store, store operation, marketing, etc.

The data analyzed by the AI platform includes, for example, store information including store name, number of store seats, number of visitors, time distribution of number of visitors, product name, price, cost, sales quantity for each product, provision for each product. Examples include product information including time, expiration date for each ingredient, purchase information, etc., image information and audio information related to the product and food, and their cooking. Further, in AI customer analysis, customer attributes including customer age, gender, etc., visitor time distribution, staying time, etc. are image-recognized using image data of the store, and are automatically analyzed and visualized. However, in customer analysis, we do not perform analysis that identifies individuals. The vast amount of data containing this information is also used for training various AIs on the AI platform.

The AI platform 10 includes an AI supply / demand forecasting unit 11, an AI automatic food loss forecasting unit 12, an AI customer analysis unit 13, an AI product analysis unit 14, an AI staff work analysis unit 15, an AI quality control unit 16, and the like. There is. The AI platform 10 is composed of, for example, the modules (a) to (f) described later, and for example, the modules (a) to (f) are used for the analysis in each of the analysis units 11 to 16.

In the AI supply / demand forecasting unit 11, AI can predict the number of visitors, the number of sales, etc. by machine learning from information such as past results, weather, events, promotions, other store information, etc., and can visualize the results.

The AI automatic food loss forecasting unit 12 predicts and calculates the required raw materials and the number of staff based on the demand forecast, reduces the shift creation work and purchasing work of the store manager, avoids overemployment of human resources, and reduces food loss.

In the AI customer analysis unit 13, the camera with AI function automatically recognizes the customer's attributes, analyzes and visualizes the customer's gender, age group, visitor time zone, staying time, etc., and these analysis information is used for marketing. ..

The AI product analysis unit 14 analyzes and visualizes product trends based on customer attributes, regions, etc. in product purchases, and these analysis information is used for product development.

The AI staff work analysis unit 15 can analyze and visualize the work time by associating the work of the store staff with the order menu, and can improve the store turnover rate by using the analysis information.

The AI quality control department 16 analyzes the cooking process to pursue deliciousness, to make the taste uniform regardless of the cook / store, and to optimize the cooking procedure. In addition, HACCP (Hazard Analysis and Critical Control Point: process control to ensure the safety of food, etc.) related information in the store kitchen is automatically managed to reduce the work of ensuring safety during cooking. Although not particularly limited, deep learning is used, for example, for the analysis of the cooking process in the AI quality control unit 16, and training based on image information, audio information, etc. regarding products and foods and their cooking is performed. ..

Further, the learning control system of the present embodiment includes the following steps, and machine learning by the AI platform is possible.
(1) Store reservation (automatic restaurant management, remote reservation, seat information)
(2) Order / payment (customized order and remote payment; recommended menu proposal, nutritional management, food allergies, seasoning preferences, ingredients / toppings)
(3) Ingredients arrangement (central kitchen, preparation, cutting)
(4) Supply of ingredients (preservation according to ingredients)
(5) Cooking (recipe, heating, mixing)
(6) Filling (automation of filling)
(7) Providing and serving (automatic transportation)
(8) Lower set / washing (handling of tableware by image recognition)
(9) Preparation (purchasing based on demand forecast, material replenishment, management / analysis)

As a specific example of the AI platform, a configuration example including the following (a) to (f) can be constructed.
(B) Demand forecast (manufacture freshly cooked products to meet demand)
(B) Adjusting the temperature and heat (c) Baking, baking time (for example, image recognition or thermography may be used to determine when to turn the steak over).
(D) Boiled cooking and frying Victory adjustment and time adjustment (for example, the boiled state and fried state are judged by recognition of images and sounds).
(E) Ingredient recognition (for example, by recognizing from images and sounds, the type and state of ingredients are recognized and an appropriate cooking method is determined.)
(F) Recognition of cooking condition (for example, by recognizing images and sounds, the degree of heating, heating time, etc. are managed).

Further, as an example of the configuration example of the AI platform, those provided with the following learning modules (a) to (f) can be mentioned, and in each module, for example, training by deep learning using big data is performed. ..
(A) Demand forecast module: In order to forecast sales, for example, unsupervised learning is adopted, although it is not particularly limited.
(B) Image recognition module: In order to recognize an object, for example, supervised learning is performed, although it is not particularly limited. Performs segmentation and pixel recognition. Segmentation is effective when the shape is not fixed, not just squares and circles.
(C) Voice recognition module: For example, it recognizes a cooking situation. For example, as a deep learning network, for example, LSTM (Long short-term memory) or the like is used, and features such as periodicity and peaks are analyzed.
(D) Recommended module: A favorite is recommended based on past data, and unsupervised learning is adopted, for example, in order to find a person's characteristics and show a tendency, although not particularly limited. Has a recommendation function.
(E) Abnormality detection module: Anomaly detection module for tempura oil is alerted at ± 10 degrees to notify the abnormal situation of equipment. On the factory line, an alert is issued even if one out of 10,000 defects occurs. Strict management is required for safety management. Although not particularly limited, for example, unsupervised learning is performed. Alert only when the threshold is removed.
(F) Learning module (three types of learning): Machine learning other than the above (a) to (e), or using the information and results of the above (a) to (e), or these information Perform machine learning that combines the results. There are three types of machine learning: supervised learning, unsupervised learning, and reinforcement learning. Supervised learning includes, for example, regression, unsupervised learning includes, for example, LSTMs, and reinforcement learning includes segmentation (eg, recognition of sushi material). Reinforcement learning is also suitable for robot learning. Reinforcement learning is learning based on limited data, and typical methods include TD learning and Q-learning. It is possible to adopt deep learning in any module.

Hereinafter, a specific configuration example of the AI platform 10 will be described. For example, in the case of steak, in AI quality control 16, the state, size, volume, and thickness of fats and oils and lean meat are recognized, and the degree of heating, baking time, and timing of turning over are optimized by supervised learning.

Also, for example, in the case of fried food, the fried condition can be optimized by supervised learning from the sound of oil and the like. Furthermore, not only the cooking time (fried time, baking time, boiling time, etc.) but also the maintenance time can be notified. In addition, it is possible to cook by taking into account the timing of provision according to the demand according to the demand forecast. As the sensor, learning and recognition are performed based on an image sensor, a voice sensor, a document, and the like. Further, it is also possible to provide a temperature sensor, a weight sensor and the like. For example, the remaining quantity and quality of the soup bar can be estimated from the image, temperature, weight and the like.

As an application example of the AI platform, in the case of a franchise chain, it is possible to use the relationship between the headquarters and the stores, that is, the model in which the data is aggregated and learned at the headquarters at each store.

In the present embodiment, the products provided to the customer are not particularly limited, but various foods and drinks such as, for example, eat-in menu, take-out menu, food, soft drink, alcoholic drink, hot drink, and cold drink. Includes menu. The stores are not particularly limited, but for example, restaurants, mobile stores, delivery stores, temporary stores, eat-in corners, food courts, accommodation facilities, schools, hospitals, cafeterias, supermarkets, department stores, mass retailers. , Stores, and convenience stores, etc., are included.

For deep learning, for example, CNN is used, and the signal is not particularly limited, but for example, as preprocessing, the teaching data is inflated by labeling and 2D conversion, and signals such as RGB series signals, HSV series signals, and infrared image signals are used. By performing the processing, it is possible to improve the learning efficiency and the accuracy. Hereinafter, (Example 1) to (Example 9) will be described as specific examples of machine learning.

(Example 1)
In this embodiment, the process of optimizing the cooking of steak by the AI quality control unit 16 will be described. The ratio of lean meat and fat content for steak, its calories, etc. are judged from the image, and the condition of the meat and the degree of baking are adjusted from the difference in color by image recognition for the image data input from the camera. By such image recognition, it is possible to eliminate the difference in taste depending on the store and the variation in the baking condition depending on the cook, and to standardize. Optimal cooking of ingredients based on past cooking data and at any store. In addition, the state of raw meat is judged and optimal cooking is performed.

(1) Data set generation The steak cooking process is photographed with two cameras. As a camera, for example, GoPro can be used. The acquired image is labeled with the type of operation (inside-out flip, input put_in, no operation none), start point, and end point, and the labeled moving image is converted into multiple still images. Although not particularly limited, the image size can be 256 × 455 pixels. Next, the mean and standard deviation of the converted image pixels are calculated. Next, divide the image into a training list and a validation list. Although not particularly limited, for example, the ratio of the training list is 2/3 and the ratio of the verification list is 1/3.

(2) PyTorch and PyTorchVision were used as the training framework, and 3D ResNet-18 (18layers) capable of action recognition was used as the network (hereinafter, also referred to as "core network"). The loss function is Cross_Entropy_Loss

Is used.

As a pre-processing for the image
・ Random scaling ratio (0.8 ~ 1.3),
・ Randomly cut out a part of the image at five places (center, upper left, upper right, lower left, lower right).
・ Randomly rotate,
・ Randomly horizontal inversion or
・ Normalize with mean and standard deviation,
The number of images was increased by performing processing such as. Also, in order to generate time-series images, 16 consecutive images were cut out from the similarly labeled images at random.

(3) Verification The number of images of the verification data was increased by using a method of cutting out from the center, a method of normalizing with the average value and the standard deviation, and the like. In addition, as a method for generating time-series images, 16 continuous images were cut out in order on the time axis, and if the number was less than 16, the last image was copied and added up to 16.

(4) Results The learning results will be described with reference to FIGS. 2 to 4. FIG. 2 shows the determination results of the meat feeding time and the meat turning over time, FIG. 2A correctly determines the feeding time (Put_in, 0.927), and FIG. 2B correctly determines the turning over time (flip, 0.907). The accuracy of the judgment was 97.78%.

FIG. 3 is the result of analyzing the number of operations, and the timing of putting in and the timing of turning over are appropriately determined. Further, FIG. 4 shows the results of analyzing the pixels of the meat image, and the grayscale, red, green, and blue image data were analyzed. FIG. 4A shows a color image and FIG. 4B shows a grayscale image. Further, FIG. 4C shows the analysis result of the image by the gray scale, FIG. 4D shows the analysis result of the image by the red component, FIG. 4E shows the analysis result of the image by the green component, and FIG. 4F shows the analysis result of the image by the blue component. The state of meat can be image-recognized by AI trained based on each image and the results of each of these analyzes.

From the number of movements in FIG. 3 and the image of the meat in FIG. 4, the time when the meat is put in and the time when the meat is turned over are learned, and the optimum time when the meat is put in and the time when the meat is turned over are output. As a result, by machine learning, it is possible to discriminate the degree of baking and appropriately determine the cooking time such as the appropriate charging time and the turning time, and the cooking time for each degree of baking (rare, medium, well-done, etc.). It is also possible to improve the learning accuracy by using the data taken by the customer.

Further, the state of the meat can be discriminated by the image recognition of FIG. 4, and the baking condition can be adjusted based on the discriminating result. For example, the condition of meat can be determined based on the balance between lean and fat, the distribution of fat, the color of lean, the color of fat, the volume of meat, the thickness of meat, the shape of meat, the distribution of fat, etc. Since it is possible to calculate the calories of the meat, adjust the baking condition, grasp the part to start baking, the part to be baked well, etc., it is possible to provide the steak with the optimum baking condition according to the condition of the meat.

When a steak is cooked by a cook, it is possible to provide the cook with information such as the degree of heating, the baking time, and the time to turn the meat over. Further, when cooking by the cooking robot, the cooking robot can cook the steak in the optimum baking condition based on the information from AI. This makes it possible to provide uniform and high-quality steaks anytime, anywhere, without depending on the store or the cook.

(5) Determining the degree of grilling based on the brightness FIG. 5 is an explanatory diagram of time-series image analysis of the grilled color of the surface of the steak. The steak meat (beef) portion is extracted from the cooked image by object recognition by the same deep learning as described above, and the grilled color of the beef surface is analyzed by time-series image. FIG. 5A is an image in which beef has begun to be grilled, and FIG. 5D is an image in which beef is grilled in the order of FIGS. 5A, 5B, 5C, and 5D, and FIG. 5D is a baked image. FIG. 5E is a graph showing the change in brightness of beef as it is cooked and until it is cooked. The vertical axis of FIG. 5E is the brightness in the color expression by the HSV model, and the horizontal axis is the cooking time. A straight line drawn parallel to the horizontal axis in the graph (for example, a straight line with Brightness = 196) is a judgment line used as one of the criteria when analyzing the baking. Based on the above deep learning, by quantitatively observing the time transition of the grilled color, we analyze the difference between the cooking process of the apprentice and the apprentice, and give the apprentice cook the same cooking process as the apprentice. It becomes possible to teach. In FIG. 5E, the graph in which Brightness gradually decreases while repeating minute changes in increase / decrease shows a state in which beef for steak is cooked by a certain cook. When the meat is turned over, a sharp decrease in Brightness is observed, and a change in Brightness on the surface of the meat is observed. By analyzing the characteristics of the cooking process of a proficient person by deep learning about the state of this change in Brightness, it is possible to teach the cook the optimum meat baking time and the timing of turning over the meat.

(6) Analysis of balance between lean meat and fat FIG. 6 is an explanatory diagram of image recognition for analyzing the balance between lean meat and fat of steak and estimating the position of cutting into equal parts by volume. In FIG. 6, the gray color is the lean beef portion for steak, the black color is the lipid portion, and the vertical line is the cut line indicating the cutting position where the volume is equally divided. In addition, according to the image recognition of this example, based on the same deep learning as described above, in addition to the analysis of the balance between lean meat and fat and the estimation of the position to be cut into equal parts by volume, the weight and volume ( It is possible to recognize Meat Volume, Area, Fat Rate, and Calorie. Since the display of the cut line that divides the volume into equal parts differs depending on the shooting direction of the camera, practically, the optical axis of the camera is installed so that it is within the range of inclination of about 20 degrees with respect to the perpendicular line from the observation target. It is desirable to do.

(7) Analysis of Infrared Thermography Camera Image FIG. 7 is an explanatory diagram showing a state in which the same deep learning as described above is performed using the image of the thermography camera. When a thermography camera is used as a camera, it is possible to capture and analyze not only the color change of the surface but also the temperature change, so that it is easier to more appropriately teach a cooking method equivalent to that of a skilled person. In FIG. 7A, the average temperature is 53.85 ° C., and the temperature variation value (standard deviation STD) is 2.56 ° C. FIG. 7B has an average temperature of 52.49 ° C and an STD of 20.65 ° C. When the STD is large, it is an indicator that the unevenness of burning is large. For example, when the heat is stronger than the surroundings of the frying pan, the center of the frying pan outputs the teaching content such as adjusting the cooking target on the frying pan, here the position where the beef for steak is placed, as the output of the deep learning system. Then, according to this teaching content, cooking can be performed so that the STD is within a predetermined allowable value.

In this embodiment, beef has been described as an example of meat for steak, but the application of this example is not limited to meat for steak, and other meat, fish, vegetables, etc., for example. It can be applied to cooking and ingredients as well.

(Example 2)
In this embodiment, a system for giving advice on the boiling time of noodles, the temperature of hot water, and the degree of heating will be described by analyzing the cooking process in the AI quality control unit 16. In this embodiment, boiling noodles is exemplified, but the present embodiment is not limited to this, and when a heating medium other than hot water is used, a cooking utensil using various heating media such as a fryer using oil, for example. Applicable to. If necessary, analysis by AI demand forecasting unit 11, AI customer analysis unit 12, and the like is also performed.
(1) For store reservations, automatic restaurant management, remote reservations, and seat guidance will be performed.
(2) For orders and payments, customized orders and remote payments are possible, and recommended menu proposals, nutritional management, food allergies, seasoning preferences, ingredients and toppings are possible. The demand forecast module predicts sales according to the date, weather, events, etc., and the recommended module recommends favorite menus that correspond to customers based on past customer data.
(3) Regarding the arrangement of ingredients, the central kitchen will be prepared with cuts and the like.
(4) Regarding the supply of foodstuffs, store them according to the foodstuffs.
(5) For cooking (recipe, heating, mixing), the image recognition module recognizes the object, the voice recognition module recognizes the cooking status, etc., and the abnormality detection module notifies the abnormal situation of the equipment, etc. Learn based on limited data with the reinforcement learning module.
(6) As for the filling, the filling will be automated.
(7) Fully automatic transportation will be provided and served. The congestion status of stores can be grasped in advance by the demand forecast module.
(8) For tableware and washing, tableware will be handled by image recognition.
(9) Regarding preparations, we will purchase, replenish materials, manage and analyze by demand forecast. The demand forecast module forecasts sales based on the date, weather, events, etc. The image recognition module recognizes objects such as the type, shape, and size of foodstuffs.

(Example 3)
In this embodiment, an example of a salad bar will be described, but the present embodiment is not limited to the salad bar, and can be applied to serving dishes such as a soup bar, a buffet style, and a buffet style. By using the AI platform, it is possible to appropriately determine the remaining amount of food and the state of food, and perform replenishment, serving, serving, and the like. If necessary, analysis by AI demand forecasting unit 11, AI food loss forecasting unit 12, AI customer analysis unit 13, AI product analysis unit 14, AI quality control unit 16, and the like is also performed.
(1) For store reservations, automatic restaurant management, remote reservations, and seat guidance will be performed. Forecast sales with the demand forecast module. The image recognition module recognizes objects such as the situation in the store, the number of customers, and the condition, and grasps the availability of seats in the store. With the voice recognition module, not only the situation of the audience seats but also the cooking situation can be recognized.
(2) Customization orders and remote payments are possible for orders and payments. Recommended menu suggestions, nutritional management, food allergies, seasoning preferences, ingredients and toppings are also possible. The demand forecast module also forecasts sales.
(3) Regarding the arrangement of ingredients, the central kitchen will be prepared with cuts and the like. The image recognition module recognizes objects such as the type and quantity of foodstuffs, and the voice recognition module also recognizes store conditions, cooking conditions, etc., enabling appropriate food arrangement according to the situation.
(4) Regarding the supply of foodstuffs, store them according to the foodstuffs. At this time, the image recognition module recognizes objects such as the type and amount of foodstuffs, and also uses the voice recognition module to recognize the store status and cooking status.
(5) For cooking (recipe, heating, mixing), the image recognition module recognizes the object, the voice recognition module recognizes the cooking status, etc., and the abnormality detection module detects abnormal situations such as equipment. Notify and learn based on limited data in the reinforcement learning module.
(6) Regarding provision and serving, the image recognition module recognizes objects such as store status during automatic transportation, and the voice recognition module recognizes not only store status but also cooking status.
(7) Regarding the lower set and washing, when the tableware and the like are automatically handled, the image recognition module recognizes the object such as the tableware, and the voice recognition module recognizes the store situation and the cooking situation.
(8) Regarding preparations, purchase, material replenishment, management and analysis will be carried out based on demand forecasts. The demand forecast module predicts sales based on the date, weather, events, etc., and the image recognition module recognizes objects such as the situation of the audience seats.

(Example 4)
In this embodiment, an example of an automatic restaurant will be described. In the automatic restaurant, analysis is performed by AI demand forecasting unit 11, AI food loss forecasting unit 12, AI customer analysis unit 13, AI product analysis unit 14, AI staff work analysis unit 15, AI quality control unit 16, etc., as necessary. It will be done.
(1) For store reservations, automatic restaurant management, remote reservations, and seat guidance will be performed. The demand forecast module predicts sales, the image recognition module grasps the number and status of customers in the store by object recognition, the number of vacant seats is automatically determined, and the voice recognition module uses only the seat status. It also recognizes the cooking situation.
(2) For orders and payments, customize orders and remote payments, recommend menu proposals, nutritional management, food allergies, seasoning preferences, ingredients and toppings. Sales can be grasped in advance with the demand forecast module. In addition, guests will be greeted at the entrance and guidance will be provided automatically to the seats.
(3) Regarding the arrangement of ingredients, the central kitchen will be prepared with cuts and the like. The image recognition module recognizes objects such as foodstuffs, and the voice recognition module recognizes cooking conditions.
(4) Regarding the supply of foodstuffs, store them according to the foodstuffs. At this time, the image recognition module recognizes objects such as the type and amount of foodstuffs, and also uses the voice recognition module to recognize the store status and cooking status.
(5) For cooking (recipe, heating, mixing), the image recognition module recognizes the object, the voice recognition module recognizes the cooking status, etc., and the abnormality detection module notifies the abnormal situation of the equipment, etc. Then, learning is performed based on limited data in the reinforcement learning module.
(6) As for the filling, the filling will be completely automated. The image recognition module recognizes objects such as dishes and dishes, the voice recognition module recognizes cooking conditions, etc., the abnormality detection module notifies abnormal situations of equipment, etc., and the reinforcement learning module is limited. You can learn based on the data.
(7) Regarding provision and serving, the image recognition module recognizes objects such as store status during automatic transportation, and the voice recognition module recognizes not only store status but also cooking status. Furthermore, it is possible to detect that the audience seats are not cold and add water to the empty glass.
(8) Regarding the lower set and washing, when the tableware etc. are automatically handled, the image recognition module recognizes the object such as the tableware, and the voice recognition module recognizes the store situation, cooking situation, etc. .. The prepared dishes are automatically washed by the automatic washing robot. Furthermore, it is determined whether the customer is out of the seat, temporarily left to go to the bathroom, or returned. For returning guests, guide them to the exit and see them off.
(9) Regarding preparations, we will purchase, replenish materials, manage and analyze by demand forecast. The demand forecast module predicts sales based on the date, weather, events, etc., and the image recognition module recognizes objects such as the situation of the audience seats.

The automatic restaurant according to this embodiment will be described in more detail. The automatic restaurant of the present embodiment is equipped with an automatic food serving system, accepts food orders according to orders from customers' mobile terminals, etc., and automatically cooks food according to the recipe and cooking method according to the order. And you can serve food to your customers. In addition, the automated restaurant of the present embodiment automatically grasps the vacant seat status in the store, allows the customer to confirm the vacant seat remotely, allows the customer to reserve a seat remotely, and allows the customer to reserve a seat remotely. Orders can be placed, food is served to customers automatically, payment of charges is possible from the customer's terminal, serving is automated, and collection of served containers after use is automated. That, the cleaning of the serving container after use is automated, the replenishment of the serving container after cleaning is automated, the cleaning of the cooking container after use is automated, the customer's seat in the store The guidance to customers is automated, the serving of food to customers is automated, the prediction of orders from customers is automated, and the ordering for purchasing ingredients is automated. It is possible that the delivered ingredients are automatically replenished in the feeder.

An example of using an image pickup device will be described as an example of a means for automatically grasping the vacancy status in the store. An image pickup device such as a camera is installed in the store to take pictures of the audience seats in the store. By recognizing the captured image, it is possible to automatically and in real time grasp the vacancy status in the store. As a result, vacant seats in the store can be automatically determined and the vacant seat information can be reflected in the reservation system. By using this reservation system, customers can check the vacant seats of the store in real time at any time and reserve the desired seats, and can also reserve the seats and order the food at the same time. .. In addition, since the food serving time can be adjusted according to the reservation time, it is possible to serve the ordered food to the customer at an appropriate timing after entering the store. As the image pickup means, an appropriate means such as a black-and-white camera, a color camera, an infrared camera, and a video camera can be used. Further, the vacant seat can be determined not only by image recognition by moving images but also by image recognition of still images at predetermined intervals. Furthermore, by analyzing the food provided to the customer by image recognition, it is possible to grasp the pace at which the customer recommends the meal. Therefore, the automatic food serving system 1 can serve food at the pace of the customer's meal, and can recommend the order of additional food (including dessert and drink) via the mobile terminal. can.

It is also possible to automate the guidance to the seats of customers who have entered the store. When a customer has reserved a seat, by indicating the reservation number, customer code, etc. issued at the time of reservation, for example, the barcode, two-dimensional bar code, etc. registered in the customer's mobile terminal can be used at the store entrance. The system 1 recognizes the customer by holding it over the reading device in the above, or by authenticating the ID code by communication such as short-range wireless communication from the customer's mobile terminal. Next, since the customer's reserved seat is displayed on the customer's mobile terminal or the display device of the store, the customer can recognize the seat reserved by the customer. In addition, the automatic serving machine 21 can be used to guide the customer to the reserved seat.

If the customer has not made a reservation, the customer notifies the system 1 by wireless communication at the entrance of the store that the customer has visited the store by a mobile terminal, or by communication such as short-range wireless communication from the customer's mobile terminal. By communicating with the system 1, the system 1 grasps the customer's ID, confirms the past visit history, etc., and proposes an appropriate vacant seat to the customer by the customer's mobile terminal or the display device of the store. When the customer approves the seat, the customer's seat is determined, and the customer is guided to the reserved seat by using the mobile terminal, the display device in the store, and the automatic serving machine 21. Guides customers to their seats. Food orders can be placed using the customer's mobile terminal, the store's dedicated terminal, or the like. When there are no vacant seats, the system 1 presents the expected waiting time, the number of people waiting, etc. to the customer by displaying the waiting time on the customer's mobile terminal or displaying it on the display device of the store. , Ask for an answer as to whether the customer is waiting for a seat. If the customer chooses to wait for a vacant seat, the customer will be notified of the waiting status at any time, and information such as recommended menus and store introductions will be provided. According to the above, when it is the customer's turn, the seat will be guided to the vacant seat as described above.

Regarding accounting, in the case of an order from a mobile terminal, payment can be made electronically from the mobile terminal. Electronic payment is possible from the customer's mobile terminal even when placing an order using a store's dedicated terminal or voice recognition device, but if the store is equipped with an automatic payment device, the customer can use the automatic payment device. , Cash card, credit card, cash, electronic money, prepaid card and other appropriate payment methods can be used for payment. If a gate is provided at the exit of the store so that the customer can open the gate by presenting a predetermined ID, for example, by presenting the customer's ID by short-range wireless communication using a mobile terminal, the fee is unpaid. Customers can be identified and the opening and closing of gates can be controlled, and voice or display can be used to encourage unpaid customers to pay. If a system that pays the fee in advance is adopted, it is possible to prevent the customer from leaving the store without paying the fee.

Next, a means for automating ordering for purchasing ingredients, inventory management, and a means for automatically replenishing the delivered ingredients to the ingredients supply device 13 or the noodle supply device 10 will be described. In the system 1 of this embodiment, in addition to the order data of all customers, the order data of other stores, the past order history data, the weather data, the temperature data, the humidity data, the calendar information, the event information, the crowd forecast information, We grasp various information such as congestion status in the store obtained by recognizing the image by the image pickup means, predict the number of visitors to the store and the order of each menu from this information, and stock the ingredients or noodles in advance. And can automate the ordering for the purchase of ingredients. In order to predict the number of visitors to the store and the order of each menu, order data of each customer, past order history data, information from related systems such as other stores, information of information research organizations, and information on the Internet Etc. are also available, and it is possible to analyze such a large amount of information by machine learning using artificial intelligence, for example.

The inventory of foodstuffs, etc. corresponding to the predicted order is managed according to the inventory management of the system 1 according to the contents of the order. The delivered inventory is managed in a predetermined stock place, and is appropriately supplied to the food material supply device 13, the noodle supply device 10, and the like. The replenishment of foodstuffs or noodles to the foodstuff supply device 13 and the noodle supply device 10 and the like can be automated by using, for example, an automatic serving machine 21.

(Example 5)
In this embodiment, the types of sensors are illustrated. The state of foodstuffs and cookers can be determined by sensors that analyze sugar content, taste, weight, salt content, hardness (pressure sensor), temperature, humidity, time, and the like. Also in image recognition, for example, by determining color change, deliciousness can be detected. In addition, for example, the quality can be judged by the elapsed time after replenishing the soup. These sensors can be used for analysis in AI demand forecasting unit 11, AI food loss forecasting unit 12, AI customer analysis unit 13, AI product analysis unit 14, AI staff work analysis unit 15, AI quality control unit 16, etc., as necessary. Used.

(Example 6)
By recognizing the movement of a person (cooker) as an image, it is possible to properly advise the cook on the state of cooking. If necessary, the AI staff work analysis unit 15 and the AI quality control unit 16 and the like perform analysis.

(Example 7)
Image recognition of foods displayed at supermarkets to grasp the sales situation. In addition to the remaining amount of products displayed in the store, the condition of food is also determined. If necessary, analysis is performed by AI demand forecasting unit 11, AI food loss forecasting unit 12, AI customer analysis unit 13, AI product analysis unit 14, AI quality control unit 16, and the like.

(Example 8)
In this embodiment, an example will be described in which lettuce or cabbage is recognized by deep learning by analysis of the cooking process in the AI quality control unit 16, and only the core portion is determined and separated. Deep learning was used for image recognition of cabbage or lettuce. In particular, ResNet18 was used as the core network, CNN was used as the basic feature extractor, and PyTorch and PyTorchVision were used as the framework. As the image for training, we used ball lettuce purchased in Japan.

(1) Data set generation Labeling was performed on the images (73 sheets) of ball lettuce. Specifically, the photos of ball lettuce are manually labeled and formatted for the custom data set. Specifically, as shown in FIG. 8, the positions of the lettuce cores are identified by two points. 8A-8D show four images as an example of leveling ball lettuce. The two points that identify the lettuce core are a pair of diagonal vertices of a rectangle circumscribing the lettuce core. In addition, the image was inflated by 2D conversion as a pretreatment. As a 2D conversion, for example
・ Mult = 1.0
・ Do_flip = True
・ Max_rotate = 360
・ Pad_mode ＝'reflection'
including. The batch size was 16.

(2) Learning Data set division (test validation) for setting hyperparameters, learning data (73 images) for using multiple images for model construction, and hyperparameter tuning (model evaluation) It was divided into the learning data (31 sheets) used for. ResNet18 was used as the core network, and the output was the four vertices of the bbox based on ResNet18. CNN was used as the basic feature extractor, and PyTorch and PyTorchVision were used as the framework. FIG. 10 is an explanatory diagram of an output image of lettuce image recognition. The four points around the lettuce core correspond to the four vertices of the bbox rectangle, and the central point is the point of contact of the two diagonals of the bbox.

(3) Verification The discrimination rate of the lettuce core was evaluated visually, and the hyperparameters were manually evaluated and tuned. 31 photographs were used as learning data by tuning hyperparameters.

(4) Results Figure 9 shows the training results. The solid line is the training loss coefficient, and the broken line is the evaluation loss coefficient. When both the training loss coefficient and the evaluation loss coefficient decrease and become parallel to the X-axis, it indicates that the learning has converged. If either the output loss coefficient or the verification loss coefficient rises or falls erratically, it indicates overlearning. The correct answer rate of the output was 95.6% when the IoU threshold was 0.75 and the maximum number of bboxes was 100.

FIG. 11 is an explanatory diagram of a line for cutting lettuce. Using the information of the four vertices of the bbox obtained from the trained CNN, the line to cut the lettuce core is determined as shown in FIG. First, draw an ellipse through the four vertices of bbox. Next, draw a rhombus with two pairs of tangents to the ellipse. Then cut the lettuce in half along one of the diagonals of the diamond. Remove the core from the half lettuce by cutting twice along the two sides of the rhombus for each half lettuce cut in half.

It was confirmed that it is possible to recognize the cabbage as an image and identify the core of the cabbage by the CNN trained by the photograph of the ball lettuce. It was found that if the cross section of the core is an image facing the front direction, the core is discriminated with high probability regardless of the type of cabbage or lettuce. By the CNN trained with ball lettuce, what can be identified is almost all commercially available cabbage such as cold ball, spring cabbage, purple cabbage, green ball, type of ball lettuce to be headed, and shooting conditions. However, it can be used for many types of cabbage or lettuce such as non-heading scratched chisha, standing chisha, and leaf chisha. However, for non-headed lettuce, it is a condition that it can be discriminated that the cross section of the core faces the front.

(5) Another Example of Example 8 (Example of recognizing the shape of the lettuce core)
In the above lettuce recognition, the four vertices of the bbox are output, but in another embodiment, an example of recognizing the shape of the lettuce core will be described. ResNet50 + FPN was used as the core network, and the mask was set to conv and the box was set to FC as the head. CNN was used as the basic feature extractor, and Detectron2 was used as the framework.

Similar to Example 8, data set division (test validation) is performed to set hyperparameters, and learning data (73 images) for using a plurality of images for model construction and hyperparameter tuning (situation). It was divided into learning data (31 sheets) used for model evaluation). FIG. 12 is an explanatory diagram of lettuce labeling. As shown in FIG. 12, the photographs of ball lettuce are manually labeled and formatted. FIG. 12A is a learning photograph before labeling, and FIG. 12B is a learning photograph after labeling. In FIG. 12B, the core portion is marked with an irregular shape. The output is a polygon that combines multiple line segments and is indeterminate.

FIG. 13 is an explanatory diagram of the training result of lettuce image recognition. The solid line is the training loss coefficient, and the broken line is the evaluation loss coefficient. When both the output loss coefficient and the verification loss coefficient decrease and become parallel to the X-axis, it indicates that the learning has converged. If either the training loss coefficient or the evaluation loss coefficient rises or falls in an unstable manner, it indicates overlearning.

FIG. 14 is an explanatory diagram of the correct answer rate of lettuce image recognition. When the IoU (Intersection over Union) threshold was 0.5 or 0.75, the correct answer rate was 0.956, and when the IoU threshold was 1.0 (All), the correct answer rate was 0.796. Here, IoU threshold is the ratio of the area of the area of the output mask that overlaps the area of the mask in the teaching photograph. FIG. 15 is an explanatory diagram of an output image of lettuce image recognition. Four output images are exemplified in FIGS. 15A to 15D. Each image is masked with its core surrounded by an irregular shape, and a rectangular box is displayed so as to circumscribe this mask.

FIG. 16 is an explanatory diagram of a method for determining a cutting line for cutting a core of cabbage or lettuce. First, with reference to the predicted center of the box, extend the diagonal length by 5% at both ends. Next, draw an enlarged box and move on to the cutting process. The cutting process first halves the cabbage or lettuce along the diagonal of one of the box's rectangles. Cut every half twice along each side of the rectangle and cut the core. In another example as well, it is possible to identify the core of cabbage or lettuce other than the ball lettuce by the CNN trained by the photograph of the ball lettuce as in the above-mentioned Example 8, and the core of the non-headed lettuce. Although it is a condition that it is possible to discriminate the image whose cross section faces the front, it is possible to discriminate the core of most commercially available cabbage or lettuce.

In this embodiment, an example of discriminating the core of cabbage or lettuce has been described, but the method for discriminating the separated portion of the present embodiment is not limited to cabbage or lettuce, and other than vegetables, meat, fish and the like. It can also be applied to the separation of ingredients and dishes.

(Example 9)
In this embodiment, a process of recognizing an image of sushi ingredients by analyzing the cooking process in the AI quality control unit 16 and arranging them neatly in a container or a container will be described. The shape of sushi ingredients is indefinite. In addition, the material is recognized by image recognition from the hue. Recognize the type and number of sushi. For example, image recognition of shrimp and eggs is performed by supervised learning.

Deep learning was used for image recognition of sushi. In particular, ResNet50 was used as the core network, CNN was used as the basic feature extractor, and Detectron2 was used as the framework.

(1) Data set generation Labeling was performed on sushi images (80 sheets). Specifically, the photos of sushi are manually labeled and converted to COCO format for the custom data set. In addition, the image is subjected to 2D conversion as preprocessing, for example, processing such as randomly scaling ratio (0.5 to 1.5), randomly cutting out a part of the image at multiple points, randomly rotating, and further lighting. The number of images was increased (increased from 80 to about 300) by normalizing with the mean value and standard deviation in consideration of the above conditions.

(2) Learning Data set division (test validation, for example, Train-Test-Validation division) is performed to set hyperparameters, and learning data for using multiple images for model construction and hyperparameter tuning. It was divided into training data used for use (model evaluation) and test data used for final evaluation of the model. For example, the inflated sushi images were divided into 100 for training and 100 for testing. In addition, the image data has been optimized for the COCO format to improve convenience. The iteration is 3000 times. Detectron2 was used as the framework.
For setting various options such as hyperparameters,
-Batch size 2 ⁿ and n are integers-No shuffle-Network input size All input images-Output mask, bbox and label.

(3) Verification The sushi material discrimination rate was evaluated visually, and the hyperparameters were manually evaluated and tuned. The hyperparameter settings are as follows.
・ To avoid overlearning, the number of times was 500 or less.
・ Cross-validation was performed so that the influence of parameters could be seen on learning.
・ Number of classes 20 (20 types of sushi)

(4) Results Figure 17 is an explanatory diagram of the recognition results of sushi ingredients. The correct answer rate for determining the type of sushi material was 0.75 with an IoU threshold and 96.02% when the maximum number of bboxes was 100. In FIG. 17, each sushi material is surrounded by a square marker, and the state in which the image is recognized is shown. Sushi is often similar in shape and color, but the AI trained as described above accurately recognizes the sushi material.

Sushi image-recognized in this way is served on a container or container in consideration of color and shape. For this assortment, AI trained with good assortment information, information on avoidable assortment examples, etc. is used to judge the degree of suitability of assortment, and appropriate assortment information is obtained from AI. It can also be output.

(5) Another Example of Example 9 It is necessary to distinguish between lean tuna, medium fatty tuna, and large fatty tuna in sushi ingredients, but for cooks, a lot of knowledge and skill are required to determine the physical condition of tuna. Cooking skills are required. Therefore, here we explain how to distinguish red tuna, medium fatty tuna, and large fatty tuna by deep learning. ResNet18 was used as the core network, CNN was used as the basic feature extractor, and PyTorch was used as the framework.

At the time of data set generation, sushi images (80 sheets) were labeled in the same manner as in (1) above. Specifically, the sushi photos are manually labeled and converted to VOC format for the custom dataset. In addition, the image is subjected to 2D conversion processing as the preprocessing in the same manner as in (1) above, and further normalized by the mean value and standard deviation in consideration of the lighting conditions, and the number of images is increased (80). 2D conversion to increase from 1 sheet to about 300 sheets). However, if the lighting conditions are the same, normalization is not necessary.

In training, data set division is performed to set hyperparameters, and multiple images are used for training data for use in model construction, training data for tuning hyperparameters, and final evaluation of the model. Separated from test data. For example, the inflated sushi images were divided into 100 for training, 100 for verification, and 100 for testing. As various options such as hyperparameters,
・ Batch size 16
・ With shuffle ・ Network input size 32n * 32n (n = integer)
・ Framework PyTorch
・ Basic feature extractor CNN
-It was used as an output label.

As a result of verification, the accuracy of label discrimination is lean: 99.39%
Medium Toro: 98.23%
Daitoro: 95.18%
Overall average: 97.61%
was gotten. With the AI trained in this way, similar types of sushi ingredients are accurately image-recognized. Here, the distinction between lean tuna, medium fatty tuna, and large fatty tuna was recognized, but in the same manner, it is also possible to discriminate the type of tuna, the type of bluefin tuna, southern bluefin tuna, bigeye tuna, and the like. Furthermore, it is possible to discriminate not only tuna but also types of yellowtail, such as yellowtail, yellowtail, and yellowtail. By applying Example 9 or another example, it can be used to discriminate various fish types.

In this embodiment, an example of discriminating sushi ingredients has been described, but the method for discriminating ingredients in this embodiment is not limited to sushi ingredients, but can be applied to other ingredients and dishes such as fish, meat, and vegetables. Is also applicable.

[Embodiment 2]
In the second embodiment, the provision of an AI assistant solution and a store optimization platform for assisting management by a manager (store manager or the like) in a store by using a learning control system provided with an AI platform will be described.

First, we will explain the recognition of the hall situation using AI's image recognition technology. Inventory notification assistant, lower set notification assistant, visitor notification and demand forecasting assistant, menu recommendation assistant, etc. are used to recognize the hall situation. In these assistants, the AI demand forecasting unit 11, AI food loss forecasting unit 12, AI customer analysis unit 13, AI product analysis unit 14, AI staff work analysis unit 15, and AI quality of the AI platform 10 described in the first embodiment. A trained model is used in the management unit 16 and the like.

The staff did not frequently confirm the lack of inventory of self-service salad bars, etc., leading to complaints. Therefore, it is possible to optimize the inventory by recognizing the inventory status of the salad bar and notifying the store manager or the staff before the inventory status becomes insufficient. This is expected to improve turnover and customer satisfaction.

The staff was chased by the provision of meals to the customer, the tableware set was postponed, and the customer may be kept waiting at the entrance even if the seat was vacant. Therefore, the lower set notification assistant can notify the seats that require the tableware lower set, thereby maximizing the store turnover rate and improving sales. As a result, the turnover rate and sales can be improved.

In some cases, it led to complaints without noticing the customer's visit, or the customer who entered the store returned as it was, leading to loss. Therefore, the visitor notification and demand forecast assistant can reduce opportunity loss and food loss by notifying the customer's waiting status and demand from the past, and can improve customer satisfaction and reduce opportunity loss and food loss. ..

The clerk may not be aware that the customer does not have enough drinks. Therefore, the menu recommendation assistant can reduce the opportunity loss, maximize the sales, and at the same time improve the customer satisfaction by notifying the store manager or the staff when the customer's glass is empty or low.

A cooking reproduction assistant, etc. is used to recognize the cooking situation in the kitchen. The deliciousness of the menu offered differed depending on the cook, which sometimes led to complaints. Therefore, with the cooking reproduction assistant, AI can learn the cooking method of a senior cook and teach the cooking method so that even an apprentice cook can reproduce it. For example, by adopting the example of cooking the steak of the first embodiment of the first embodiment, it is possible to improve customer satisfaction and maintain and improve the brand power by making the taste uniform.

The AI supply / demand forecasting unit 11, the AI automatic food loss forecasting unit 12, the AI customer analysis unit 13, the AI product analysis unit 14, the AI staff work analysis unit 15, and the AI quality control unit provided in the AI platform 10 described in the first embodiment. With functions such as 16, it is possible to provide the AI assistant solution and store optimization platform as described above. For example, the cooking reproduction assistant teaches the cooks of each store the appropriate cooking method according to the store situation by using the trained model trained by the cooking method of the senior cook in the AI quality control unit 16. be able to. This makes it possible to provide uniform dishes at each store, which are similar to those cooked by senior cooks.

Although the embodiments of the present invention have been described above, these embodiments and examples exemplify a learning control system for embodying the technical idea of the present invention, and specify the present invention to these. However, it can be equally applied to those of other embodiments and examples, and parts of these embodiments and examples may be omitted, added, or changed, and each embodiment and each embodiment may be omitted. It is also possible to combine examples.

10 ... AI platform 11 ... Demand forecasting department 12 ... AI automatic food loss forecasting department 13 ... AI customer analysis department 14 ... AI product analysis department 15 ... AI staff work analysis department 16 ... AI quality control department 20 ... Information network 21 ... Cooking robot 22 … Business automation AI robot 23… AI restaurant

Claims (10)

1. (A) Demand forecast module for forecasting sales,
   (B) Image recognition module that recognizes objects,
   (C) A voice recognition module that recognizes the state of an object by voice,
   (D) A recommended module that recommends products to customers, or a recommended module
   (E) A machine learning module of at least one abnormality detection module for detecting an abnormal situation of a device is provided, and further, the abnormality detection module is provided.
   (F) An AI platform including a learning module that performs machine learning other than the above (a) to (e) and / or uses the information and results of at least one of the above (a) to (e). Have and
   A learning control system characterized by performing learning control using a model trained by the AI platform.
2. Machine learning in the AI platform includes at least one of supervised learning, unsupervised learning or reinforcement learning, and / or deep learning.
   Being able to perform at least one of store management, store reservation, order or settlement, food arrangement, food supply, preparation, cooking, serving, serving or serving, serving or cleaning, preparation, or purchasing. The learning control system according to claim 1.
3. The learning control system according to claim 2, wherein CNN is used as a neural network for the deep learning, and the teaching data is inflated by labeling and 2D conversion as preprocessing.
4. The learning control system according to claim 2 or 3, wherein the RGB series signal, the HSV series signal, and the infrared image signal are used in the preprocessing of the deep learning.
5. By a neural network trained by deep learning based on the image of the cooking process of the food, from the input image of the food, the volume, weight, proportion of fat, calorie, surface area, temperature difference, heating temperature, cooking of the food. The learning control system according to any one of claims 1 to 4, wherein the timing, the timing of turning over the cooking, the timing of finishing, or at least one of the food cutting lines is output.
6. The learning control according to any one of claims 1 to 4, wherein the type of food is recognized as an image by a neural network trained by deep learning based on images of a plurality of types of food. system.
7. The learning control system according to any one of claims 1 to 4, wherein the food separation line is image-recognized by a neural network trained by deep learning based on the food image. ..
8. Claims 1-7, comprising at least one of an inventory notification assistant, a lower set notification assistant, a visitor notification or demand forecasting assistant, a menu recommendation assistant, or a cooking reproduction assistant, which is used for store management. The learning control system according to any one of the items.
9. The item according to any one of claims 1 to 8, wherein the product provided to the customer includes at least one of an eat-in menu, a take-out menu, a dish, a soft drink, an alcoholic drink, a hot drink, or a cold drink. Described learning control system.
10. Stores include at least one of restaurants, mobile stores, delivery stores, temporary stores, eat-in corners, food courts, accommodation facilities, schools, hospitals, cafeterias, supermarkets, department stores, mass retailers, shops, or convenience stores. The learning control system according to any one of claims 1 to 9, wherein the learning control system is characterized.

Images