CN110826574A - Food material maturity determination method and device, kitchen electrical equipment and server - Google Patents

Abstract

The application relates to the technical field of intelligent equipment, and discloses a method and a device for determining food material maturity, kitchen electrical equipment and a server. The method comprises the following steps: acquiring a current frame image and a starting frame image of a cooked food material in electric equipment, wherein the starting frame image is an image of the food material when cooking is started; performing fusion processing on the current frame image and the initial frame image to obtain a current fusion processing image; and inputting the current fusion processing image into a configured maturity machine learning training algorithm model, and determining the maturity corresponding to the food material. Therefore, the maturity of the food materials is determined through feature extraction and model prediction of the fusion processing images after the double-image fusion, and the reliability of determining the maturity of the food materials is improved.

Description

Method, device, kitchen electrical equipment and server for determining maturity of food materials

technical field

The present application relates to the technical field of smart devices, for example, to a method, a device, a kitchen appliance, and a server for determining the maturity of an ingredient.

Background technique

With the advancement of science and technology and the development of artificial intelligence, intelligent algorithms are increasingly applied to smart home appliances, such as refrigerators, air conditioners, ovens, etc. Among them, ovens, microwave ovens, air fryers and other kitchen appliances can be Information such as type and weight of ingredients, intelligently determine cooking time, cooking power, etc.

At present, some grilling kitchen appliances can also judge the maturity of the ingredients. After detecting the temperature inside the ingredients through a temperature sensor, the maturity of the ingredients can be judged according to the temperature. However, during the roasting process, the temperature of the ingredients is unstable, making the maturity judgment unreliable, and detecting the temperature through the contact sensor may damage the appearance of the ingredients and may also contaminate the ingredients.

SUMMARY OF THE INVENTION

In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be an extensive review, nor to identify key/critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

Embodiments of the present disclosure provide a method, a device, a kitchen appliance, and a server for determining the maturity of an ingredient, so as to solve the technical problem of low reliability in determining the maturity of an ingredient.

In some embodiments, the method includes:

acquiring a current frame image and an initial frame image of the ingredient to be cooked in the electrical device, wherein the initial frame image is an image of the ingredient when cooking is started;

Perform fusion processing on the current frame image and the initial frame image to obtain a current fusion processed image;

The current fusion processing image is input into the configured maturity machine learning training algorithm model to determine the corresponding maturity of the ingredients.

In some embodiments, the apparatus includes:

an information acquisition module, configured to acquire a current frame image and a start frame image of the ingredients to be cooked in the kitchen appliance, wherein the start frame image is an image of the ingredients when cooking is started;

a feature extraction module, configured to perform fusion processing on the current frame image and the initial frame image to obtain a current fusion processed image;

The prediction and determination module is configured to input the current fusion processed image into the configured maturity machine learning training algorithm model to determine the corresponding maturity of the food material.

In some embodiments, the kitchen electrical equipment includes: the above-mentioned device for determining the maturity of the food material.

In some embodiments, the server includes: the above-mentioned device for determining the maturity of the food material.

The method, device, kitchen appliance, and server for determining the maturity of ingredients provided by the embodiments of the present disclosure can achieve the following technical effects:

Obtain the current frame image and the starting frame image during the cooking process of the ingredients, fuse the two images to obtain the corresponding current fusion image, and input them into the configured maturity machine learning training algorithm model to determine the maturity of the ingredients. In this way, through the feature extraction and model prediction of the fused image after the fusion of the two images, the maturity of the ingredients is determined according to the degree of difference between the two images in the fusion image, which improves the reliability of determining the maturity of the ingredients. Reliability can be determined by contacting the ingredients, reducing the chance of damaging the appearance of the ingredients and contaminating the ingredients.

The foregoing general description and the following description are exemplary and explanatory only and are not intended to limit the application.

Description of drawings

One or more embodiments are exemplified by the accompanying drawings, which are not intended to limit the embodiments, and elements with the same reference numerals in the drawings are shown as similar elements, The drawings do not constitute a limitation of scale, and in which:

1 is a schematic flowchart of a method for determining the maturity of an ingredient provided by an embodiment of the present disclosure;

1-1 is a schematic diagram of a fusion processing image provided by an embodiment of the present disclosure;

2 is a schematic flowchart of a method for determining the maturity of an ingredient provided by an embodiment of the present disclosure;

3 is a schematic structural diagram of a food material maturity determination system provided by an embodiment of the present disclosure;

4 is a schematic flowchart of a method for determining the maturity of an ingredient provided by an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of a device for determining the maturity of an ingredient provided by an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of a device for determining the maturity of an ingredient provided by an embodiment of the present disclosure;

FIG. 7 is a schematic structural diagram of a device for determining the maturity of an ingredient provided by an embodiment of the present disclosure;

FIG. 8 is a schematic structural diagram of an apparatus for determining the maturity of an ingredient provided by an embodiment of the present disclosure.

Detailed ways

In order to understand the features and technical contents of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings, which are for reference only and are not intended to limit the embodiments of the present disclosure. In the following technical description, for the convenience of explanation, numerous details are provided to provide a thorough understanding of the disclosed embodiments. However, one or more embodiments may be practiced without these details. In other instances, well-known structures and devices may be shown simplified in order to simplify the drawings.

In the embodiment of the present disclosure, the current frame image and the starting frame image during the cooking process of the ingredient are acquired, the two images are fused to obtain the corresponding current fusion processed image, and the image is input into the configured maturity machine learning training algorithm model, Determine the maturity of the ingredients. In this way, through the feature extraction and model prediction of the fused image after the fusion of the two images, the maturity of the ingredients is determined according to the degree of difference between the two images in the fusion image, which improves the reliability of determining the maturity of the ingredients. When configuring the maturity machine learning training algorithm model, the dual-image fusion process can effectively avoid the problem of non-convergence of the maturity regression network caused by differences in the shape, color, and texture of different ingredients. In addition, the reliability can be determined without touching the ingredients, reducing the probability of damaging the appearance of the ingredients and contaminating the ingredients.

FIG. 1 is a schematic flowchart of a method for determining the maturity of an ingredient provided by an embodiment of the present disclosure. As shown in Figure 1, the process of determining the maturity of ingredients includes:

Step 101: Acquire a current frame image and an initial frame image of the ingredients to be cooked in the kitchen appliance.

In the embodiment of the present disclosure, the kitchen electrical equipment may include: an oven, a microwave oven, or an air fryer, and other electrical appliances that can be closed and can bake food. Generally, an image capture device can be configured in the kitchen appliance, for example, a camera is configured on the top, and the image or video information in the kitchen appliance can be captured through the image capture device.

The image acquisition device can be started regularly to acquire images in the kitchen appliances, or record video information at a set time. Alternatively, the image acquisition device can be turned on or off under the control of the trigger signal, so as to acquire corresponding image or video information. The trigger signal may be sent by the kitchen appliance, or the server may be sent by the kitchen appliance. For example, when it is determined that the kitchen appliance starts cooking, the kitchen appliance or the server sends a start instruction to start the configured image capture device to collect video information; while it is determined that the kitchen appliance is opened, or the set time is reached Under the circumstance, the kitchen appliance or the server sends a shutdown command to turn off the image capture device and stop the video information capture.

Therefore, in some embodiments, when the method is applied to a kitchen appliance, acquiring the current frame image and the start frame image of the ingredients to be cooked in the kitchen appliance may include: when cooking is started, controlling the camera to start cooking For video recording, obtain and save the starting frame image corresponding to the ingredient, and obtain the current frame image of the ingredient through the camera.

In some embodiments, the server that communicates with the kitchen appliance acquires the current frame image and the starting frame image of the ingredients in the kitchen appliance, that is, when the method is applied to the server, receives the ingredients sent by the kitchen appliance. The current frame image, and the starting frame image of the saved ingredients.

Step 102: Perform fusion processing on the current frame image and the initial frame image to obtain the current fusion processed image.

In this embodiment, the current frame image and the starting frame image need to be double-image fusion processing, which will be merged into one frame image. The current frame image and the starting frame image are fused to obtain the current fusion processed image. Alternatively, the current frame image and the starting frame image are fused by splicing the upper and lower images of the two images to obtain the current fusion processed image.

FIG. 1-1 is a schematic diagram of a fusion-processed image provided by an embodiment of the present disclosure. In this embodiment, the kitchen appliance may be an oven, and the ingredients may be cakes. The current frame image of the cake in the process of baking the cake in the oven is obtained, and then, the current frame image and the starting frame image corresponding to the cake when the cake is put into the oven are obtained by splicing the upper and lower images of the two images, as shown in Figure 1-1. the current fused image shown. Of course, the present disclosure is not limited to this, and other ways of fusing two frames of images into one frame of image can also be applied to this.

Step 103: Input the current fusion processing image into the configured maturity machine learning training algorithm model to determine the maturity corresponding to the ingredients.

Convolutional Neural Networks (CNN) can automatically learn the features of images at various levels through convolution and pooling operations, which is in line with people's common sense in understanding images. When people perceive images, they are layered and abstract. First, they understand color and brightness, then local details such as edges, corners, and lines, and then more complex information and structures such as textures and geometric shapes. The concept of the whole object. Each convolutional layer contains multiple convolution kernels, and these convolution kernels are used to scan the entire image sequentially from left to right and from top to bottom to obtain output data called feature map, that is, to obtain image feature information. In the convolutional neural network, the previous convolutional layer captures the local and detailed information of the image, and has a small receptive field, that is, each pixel of the output image only uses a small range of the input image; The layers are enlarged to capture the more complex and abstract information of the image. After the operation of multiple convolutional layers, the abstract representation of the image at different scales is finally obtained.

In a convolutional neural network, the convolutional layer corresponding to each convolutional kernel is actually a system, a system for judging a certain feature in the image. When all the single convolutional layers, that is, a single feature judgment system, When the task of feature judgment can be effectively completed, the complex system CNN composed of these many convolutional layers can complete the complex tasks given to the convolutional neural network that humans need.

Therefore, in the embodiment of the present disclosure, a mature machine learning training algorithm model can be configured based on the convolutional neural network CNN and through a large amount of sample data.

In some embodiments, the configuration process of the maturity machine learning training algorithm model includes: extracting feature image information of a plurality of sample images based on a convolutional neural network CNN, wherein the sample images are images of food ingredients whose maturity has been calibrated and corresponding It is generated after the fusion processing of the starting image of the food; through the regression network, each feature image information is supervised and trained to generate a maturity machine learning training algorithm model.

The image of the food has been calibrated for maturity, and each image of the food corresponds to the starting image of the food. In this way, it is also necessary to perform the fusion processing of the two images, which can be arranged by staggering the pixel rows of the two images, or, the top and bottom of the two images. In the splicing method, the image of the food material whose maturity has been calibrated is fused with the corresponding initial image of the food material to obtain the fusion processed image, that is, the sample image. Then, based on the convolutional neural network CNN, the feature image information of the image can be extracted, and the feature image information can be supervised and trained through the regression network to generate the machine learning training algorithm model of the maturity.

After the double image is fused, based on the convolutional neural network CNN, when extracting the feature image information of the fusion image, the convolution operation takes into account the positional relationship between the pixels and the pixels in the double image. The result of the food maturity can be integrated with the information between the food image and the starting image to measure the degree of difference between the two images. In this way, the extracted feature information uses the degree of difference between the images. If the food image is different from the starting image The greater the image difference, the higher its maturity value, and vice versa.

In addition, the feature image information of the fusion-processed image after the double-image fusion processing can effectively avoid the maturity regression network caused by "differences in the shape, color and texture of different ingredients" when performing supervised training through the regression network. . Among them, the difference between the same food of different maturity is enlarged; at the same time, the difference between different food of the same maturity is reduced, so that the characteristics of judging the maturity of food are more obvious, which makes the network easier to converge.

In some embodiments, in order to make the convergence of the regression network for supervised training better, the loss function corresponding to formula (1) can be used:

Among them, ε is the convergence coefficient, and Δx is the difference between the predicted value and the calibration value.

In this embodiment, the regression network is used to predict the maturity of the ingredients. Therefore, the predicted value may be the predicted maturity, and the calibration value may be the calibration maturity. At present, there is no convergence coefficient ε in the loss function adopted by the regression network. In this way, when the absolute value of Δx is greater than 1, it can converge faster, and a linear derivation can make the network converge quickly; and when the absolute value of Δx is less than 1, the original smooth _L1 is insensitive to outliers and outliers, and the gradient changes. Relatively smaller, it is not easy to run and fly during training. But the result is that the convergence is too slow, especially the training of large networks and large samples. Therefore, in this embodiment, ε can be introduced for smooth _L1 Plus (Δx), ε is a dynamic coefficient, the absolute value of Δx is less than 1 stage, and ε can be adjusted, so that there is a mean absolute value error (MAE, Also known as L1 loss), the gradient is not too small, and there is also the effect of mean square error (MSE, also known as L2 loss) to prevent network divergence.

In the process of configuring the maturity machine learning training algorithm model, the sample images are combined with the images of the ingredients that have been calibrated for maturity, that is, each food image has a corresponding calibration maturity, and maturity calibration is a difficult problem, generally very experienced. Only cooks can accurately judge the doneness of a certain food among all kinds of food, which is the premise of supervised network training of deep learning. In some embodiments, the maturity calibration process includes: when the cooking of the ingredients is started, controlling the camera to start the recording of the cooking video, and when the maturity of the ingredients reaches the set maturity, controlling the camera to stop the recording of the cooking video ; determine the total number of frames of the recorded cooking video; determine the maturity corresponding to the first image according to the frame serial number corresponding to the first image, the total number of frames, and the set maturity.

The process of determining that the maturity of the ingredients reaches the set maturity may be determined by the cook observing the ingredients, of course, it is not limited to this, but is determined through image comparison and the like.

For example, when the ingredients to be calibrated are placed in the oven, video recording is started, and the critical state of maturity of food baking is observed manually. When it is determined to arrive, the video recording is stopped. Therefore, the total number of frames of the video can be obtained, and the calculation formula of the doneness of the nth frame can be doneness=n/nall.

After the maturity machine learning training algorithm model is configured, the current fusion processing image after fusion processing in step 102 can be input. In this way, based on the convolutional neural network CNN, the current image feature information of the current fusion processing image is extracted, and through the regression network, The current image feature information is supervised and trained to predict the maturity of the ingredients, that is, the model is used to predict, and the prediction result is obtained, that is, the corresponding maturity of the ingredients is determined.

It can be seen that in the embodiment of the present disclosure, the current frame image and the starting frame image during the cooking process of the ingredients are obtained, the double images are fused, and the processed fusion image is input into the configured maturity machine learning training algorithm model , to determine the maturity of the ingredients. In this way, the model prediction is performed by merging the features of the two images, and the maturity of the ingredients is determined according to the degree of difference between the two images in the fusion image, which improves the reliability of determining the maturity of the ingredients. When learning and training the algorithm model, the dual-image fusion processing can effectively avoid the problem of non-convergence of the maturity regression network caused by the differences in the shape, color and texture of different ingredients. In addition, the reliability can be determined without touching the ingredients, reducing the probability of damaging the appearance of the ingredients and contaminating the ingredients.

In the embodiments of the present disclosure, the kitchen appliance may locally determine the maturity of the ingredients, and may also send the current frame image and the starting frame image to the server, so that the server determines the maturity of the ingredients. Therefore, in some embodiments, in this method In the case of kitchen appliances, after determining the corresponding maturity of the ingredients, the maturity information can also be reminded. It may include at least one of the following ways: displaying maturity information on the display screen of the kitchen appliance, playing the maturity information through a voice broadcasting device; sending the maturity information to a terminal for display and reminder. In this way, the kitchen electrical equipment locally determines the maturity of the ingredients, which improves the speed of determining the maturity and reduces the occupation of network resources.

In some embodiments, when the method is applied to the server, after determining the maturity level corresponding to the ingredients, the maturity level is sent to the kitchen electrical device for information reminder of the maturity level. The specific reminder method of the kitchen appliance can be as above. In this embodiment, the server determines the maturity of the ingredients, which reduces the occupation of resources of the kitchen electrical equipment, saves memory, and improves the multi-control function of the kitchen electrical equipment.

The operation procedures are collected into specific embodiments below, and the process of determining the maturity of the food material provided by the embodiments of the present invention is exemplified.

In an embodiment of the present disclosure, the kitchen appliance may be an oven, and a camera is disposed on the top of the oven. In addition, kitchen appliances have been equipped with a mature machine learning training algorithm model through machine learning.

FIG. 2 is a schematic flowchart of a method for determining the maturity of an ingredient provided by an embodiment of the present disclosure. As shown in Figure 2, the process of determining the maturity of ingredients includes:

Step 201: Acquire a current frame image of the ingredients to be cooked in the oven.

Step 202: Determine whether the current frame image is the starting frame image? If yes, go to step 203; otherwise, go to step 204.

Step 203 : save the current frame image as the initial frame image, and return to step 201 .

Step 204 : fuse the current frame image and the starting frame image by arranging the pixel rows alternately in two images to obtain the current fused processed image.

Step 205: Input the current fusion processing image into the configured maturity machine learning training algorithm model to determine the maturity corresponding to the ingredients.

Step 206: Display maturity information on the display screen of the oven.

It can be seen that in this embodiment, the oven can obtain the current frame image and the starting frame image during the baking process of the ingredients, perform fusion processing on the two images to obtain the corresponding current fusion processing image, and input them into the configured maturity machine learning training. In the algorithm model, the maturity of the ingredients is determined. In this way, through the feature extraction and model prediction of the fused image after the fusion of the two images, the maturity of the ingredients is determined according to the degree of difference between the two images in the fusion image, which improves the reliability of determining the maturity of the ingredients. Reliability can be determined by contacting the ingredients, reducing the chance of damaging the appearance of the ingredients and contaminating the ingredients. In addition, the oven determines the maturity of the ingredients locally, which improves the speed of determining the maturity and reduces the occupation of network resources.

In an embodiment of the present disclosure, the determination of the maturity of the food material may be performed by the server.

FIG. 3 is a schematic structural diagram of a system for determining the maturity of an ingredient provided by an embodiment of the present disclosure. As shown in FIG. 3 , the system includes: a kitchen appliance 100 , an image acquisition device 200 configured on the kitchen appliance, and a server 300 that communicates with the kitchen appliance 100 .

Wherein, the kitchen appliance 100 can acquire the current frame image and the starting frame image of the ingredients to be cooked in the kitchen appliance through the image acquisition device 200 , and can send them to the server 300 . The server 300 is configured with a maturity machine learning training algorithm model through machine learning, so that the server 300 can characterize the current frame image and the current fusion processed image after fusion of the initial frame image based on the maturity machine learning training algorithm model. After extraction and prediction, the maturity of the ingredients can be obtained, and the maturity can be sent to the kitchen appliance 100 for prompt processing.

FIG. 4 is a schematic flowchart of a method for determining the maturity of an ingredient provided by an embodiment of the present disclosure. The food material maturity determination system can be shown in Figure 3. As shown in Figure 4, the process of food maturity determination includes:

Step 401 : the kitchen appliance acquires a current frame image of the food to be cooked in the kitchen appliance through the image acquisition device.

Step 402: The kitchen appliance determines whether the current frame image is the starting frame image? If yes, go to step 403; otherwise, go to step 404.

Step 403 : the kitchen appliance saves the current frame image as the initial frame image, and returns to step 401 .

Step 404: The kitchen appliance sends the current frame image and the starting frame image to the server.

Step 405 : The server fuses the current frame image and the starting frame image by splicing two images up and down to obtain the current fusion processed image.

Step 406: The server inputs the current fusion processing image into the configured maturity machine learning training algorithm model to determine the maturity corresponding to the ingredients.

Step 407: The server sends the maturity level to the kitchen appliance.

Step 408: The kitchen appliance displays the maturity information on the display screen, and performs a voice broadcast when the maturity is greater than the set value.

It can be seen that, in this embodiment, the server can obtain the current frame image and the starting frame image of the ingredients during the cooking process through the kitchen appliance, perform fusion processing on the two images to obtain the corresponding current fusion processing image, and input them into the configured maturity level. In the machine learning training algorithm model, the maturity of the ingredients is determined. In this way, through the feature extraction and model prediction of the fused image after the fusion of the two images, the maturity of the ingredients is determined according to the degree of difference between the two images in the fusion image, which improves the reliability of determining the maturity of the ingredients. Reliability can be determined by contacting the ingredients, reducing the chance of damaging the appearance of the ingredients and contaminating the ingredients. In addition, the server determines the maturity of the ingredients, which reduces the resource occupation of the kitchen electrical equipment and improves the multi-control function of the kitchen electrical equipment.

According to the above process of determining the maturity of the food, a device for determining the maturity of the food can be constructed.

FIG. 5 is a schematic structural diagram of an apparatus for determining the maturity of an ingredient provided by an embodiment of the present disclosure. As shown in FIG. 5 , the apparatus for determining the maturity of the food material includes: an information acquisition module 510 , an image fusion module 520 and a prediction determination module 530 .

The information acquisition module 510 is configured to acquire the current frame image and the starting frame image of the ingredients to be cooked in the kitchen appliance, wherein the starting frame image is the image of the ingredients when cooking is started.

The image fusion module 520 is configured to perform fusion processing on the current frame image and the initial frame image to obtain the current fusion processed image.

The prediction and determination module 530 is configured to input the current fusion processing image into the configured maturity machine learning training algorithm model to determine the corresponding maturity of the ingredients.

In some embodiments, the method further includes: a model configuration module configured to extract feature image information of a plurality of sample images based on a convolutional neural network CNN, wherein the sample images are images of ingredients whose maturity has been calibrated and corresponding ingredients. obtained after the initial image fusion processing; and, through the regression network, supervised training is performed on each feature image information to generate a mature machine learning training algorithm model; the loss function in the regression network includes:

Among them, ε is the convergence coefficient, and Δx is the difference between the predicted value and the calibration value.

In some embodiments, it further includes: a calibration module, configured to control the camera to start the recording of the cooking video when the cooking of the ingredients is started, and to control the camera to stop the cooking video when the maturity of the ingredients reaches the set maturity. determine the total number of frames of the recorded cooking video; determine the maturity corresponding to the first image according to the frame serial number corresponding to the first image, the total number of frames, and the set maturity.

The following is an example to illustrate the device for determining the maturity of the food material provided by the embodiments of the present disclosure.

FIG. 6 is a schematic structural diagram of an apparatus for determining the maturity of an ingredient provided by an embodiment of the present disclosure. As shown in FIG. 6 , the device for determining the maturity of food materials can be applied to kitchen electrical equipment, including: an information acquisition module 510 , an image fusion module 520 , a prediction and determination module 530 , and a model configuration module 540 , a calibration module 550 and a processing module 560 .

The calibration module 550 can calibrate the maturity of the sample ingredients, that is, when the cooking of the ingredients is started, the camera is controlled to start recording the cooking video, and when the maturity of the ingredients reaches the set maturity, the camera is controlled to stop cooking Video recording; determine the total number of frames of the recorded cooking video; determine the maturity corresponding to the first image according to the frame number corresponding to the first image, the total number of frames, and the set maturity.

In this way, the model configuration module 540 can extract feature image information of a plurality of sample images based on the convolutional neural network CNN, wherein the sample images are obtained after fusion processing of the image of the food material whose maturity has been calibrated and the corresponding initial image of the food material; and , through the regression network, supervised training is performed on each feature image information to generate a mature machine learning training algorithm model; the loss function in the regression network includes:

Among them, ε is the convergence coefficient, and Δx is the difference between the predicted value and the calibration value.

When cooking starts, the camera starts recording the cooking video, the information acquisition module 510 can acquire and save the starting frame image corresponding to the ingredient, and through the camera, the information acquisition module 510 can acquire the current frame image of the ingredient.

In this way, the image fusion module 520 can perform fusion processing on the current frame image and the initial frame image to obtain the current fusion processed image.

The prediction and determination module 530 can input the current fusion processing image into the maturity machine learning training algorithm model configured by the model configuration module 540 to determine the corresponding maturity of the ingredients. The processing module 560 can display maturity information on the display interface.

It can be seen that in this embodiment, the device for determining the maturity of the ingredients in the kitchen electrical equipment can obtain the current frame image and the starting frame image during the cooking process of the ingredients, perform fusion processing on the two images to obtain the corresponding current fusion processing image, and input Go to the configured maturity machine learning training algorithm model to determine the maturity of the ingredients. In this way, through the feature extraction and model prediction of the fused image after the fusion of the two images, the maturity of the ingredients is determined according to the degree of difference between the two images in the fusion image, which improves the reliability of determining the maturity of the ingredients. Reliability can be determined by contacting the ingredients, reducing the chance of damaging the appearance of the ingredients and contaminating the ingredients. In addition, the kitchen electrical equipment determines the maturity of the ingredients locally, which improves the speed of determining the maturity and reduces the occupation of network resources.

FIG. 7 is a schematic structural diagram of an apparatus for determining the maturity of an ingredient provided by an embodiment of the present disclosure. As shown in FIG. 7 , the food material maturity determination device can be applied to the server, including: an information acquisition module 510 , an image fusion module 520 , a prediction determination module 530 , a model configuration module 540 , a calibration module 550 and a transmission module 570 .

The calibration module 550 can calibrate the maturity of the sample ingredients, that is, when the cooking of the ingredients is started, the camera is controlled to start recording the cooking video, and when the maturity of the ingredients reaches the set maturity, the camera is controlled to stop cooking Video recording; determine the total number of frames of the recorded cooking video; determine the maturity corresponding to the first image according to the frame number corresponding to the first image, the total number of frames, and the set maturity.

In this way, the model configuration module 540 can extract feature image information of a plurality of sample images based on the convolutional neural network CNN, wherein the sample images are obtained after fusion processing of the image of the food material whose maturity has been calibrated and the corresponding initial image of the food material; and , through the regression network, supervised training is performed on each feature image information to generate a mature machine learning training algorithm model; the loss function in the regression network includes:

Among them, ε is the convergence coefficient, and Δx is the difference between the predicted value and the calibration value.

When cooking is started, the camera starts recording the cooking video, and the kitchen appliance can acquire the starting frame image and the current frame image corresponding to the ingredient. In this way, the information acquisition module 510 can receive the current frame image of the ingredient sent by the kitchen appliance, and save it. The starting frame image of the ingredients.

Similarly, the current frame image and the starting frame image can be fused through the image fusion module 520 to obtain the current fusion processed image.

The prediction and determination module 530 can input the current fusion processing image into the maturity machine learning training algorithm model configured by the model configuration module 540 to determine the corresponding maturity of the ingredients. The sending module 570 can send the maturity to the kitchen electrical equipment to remind the maturity of the information.

It can be seen that in this embodiment, the device for determining the maturity of the ingredients in the server can obtain the current frame image and the starting frame image during the cooking process of the ingredients through the kitchen electrical equipment, and perform fusion processing on the two images to obtain the corresponding current fusion processing image, and Input into the configured maturity machine learning training algorithm model to determine the maturity of the ingredients. In this way, through the feature extraction and model prediction of the fused image after the fusion of the two images, the maturity of the ingredients is determined according to the degree of difference between the two images in the fusion image, which improves the reliability of determining the maturity of the ingredients. Reliability can be determined by contacting the ingredients, reducing the chance of damaging the appearance of the ingredients and contaminating the ingredients. In addition, the server determines the maturity of the ingredients, which reduces the resource occupation of the kitchen electrical equipment and improves the multi-control function of the kitchen electrical equipment.

An embodiment of the present disclosure provides a device for determining the maturity of an ingredient, the structure of which is shown in FIG. 8 and includes:

A processor (processor) 1000 and a memory (memory) 1001 may also include a communication interface (Communication Interface) 1002 and a bus 1003 . The processor 1000 , the communication interface 1002 , and the memory 1001 can communicate with each other through the bus 1003 . The communication interface 102 may be used for information transfer. The processor 1000 may invoke the logic instructions in the memory 1001 to execute the method for determining the maturity of the food material in the above-mentioned embodiment.

In addition, the above-mentioned logic instructions in the memory 1001 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

As a computer-readable storage medium, the memory 1001 can be used to store software programs and computer-executable programs, such as program instructions/modules corresponding to the methods in the embodiments of the present disclosure. The processor 1000 executes the function application and data processing by running the program instructions/modules stored in the memory 1001, that is, the method for determining the maturity of the food material in the above method embodiments.

The memory 1001 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal device, and the like. In addition, the memory 101 may include high-speed random access memory, and may also include non-volatile memory.

An embodiment of the present disclosure provides a kitchen appliance, including the above-mentioned device for determining the maturity of an ingredient.

An embodiment of the present disclosure provides a server, including the above-mentioned device for determining the maturity of an ingredient.

An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, where the computer-executable instructions are configured to execute the foregoing method for determining the maturity of an ingredient.

Embodiments of the present disclosure provide a computer program product, where the computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions that, when executed by a computer, cause all The computer executes the above-mentioned method for determining the maturity of the food material.

The above-mentioned computer-readable storage medium may be a transient computer-readable storage medium, and may also be a non-transitory computer-readable storage medium.

The technical solutions of the embodiments of the present disclosure may be embodied in the form of software products, and the computer software products are stored in a storage medium and include one or more instructions to enable a computer device (which may be a personal computer, a server, or a network equipment, etc.) to execute all or part of the steps of the methods described in the embodiments of the present disclosure. The aforementioned storage medium may be a non-transitory storage medium, including: U disk, removable hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk, etc. A medium that can store program codes, and can also be a transient storage medium.

The foregoing description and drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process, and other changes. The examples are only representative of possible variations. Unless expressly required, individual components and functions are optional and the order of operations may vary. Portions and features of some embodiments may be included in or substituted for those of other embodiments. The scope of the disclosed embodiments includes the full scope of the claims, along with all available equivalents of the claims. When used in this application, although the terms "first," "second," etc. may be used in this application to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without changing the meaning of the description, a first element could be termed a second element, and similarly, a second element could be termed a first element, so long as all occurrences of "the first element" were consistently renamed and all occurrences of "the first element" were named consistently The "second element" can be renamed consistently. The first element and the second element are both elements, but may not be the same element. Also, the terms used in this application are used to describe the embodiments only and not to limit the claims. As used in the description of the embodiments and the claims, the singular forms "a" (a), "an" (an) and "the" (the) are intended to include the plural forms as well, unless the context clearly dictates otherwise. . Similarly, the term "and/or" as used in this application is meant to include any and all possible combinations of one or more of the associated listings. Additionally, as used in this application, the term "comprise" and its variations "comprises" and/or including and/or the like refer to stated features, integers, steps, operations, elements, and/or The presence of a component does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groupings of these. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in the process, method, or device that includes the element. Herein, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method sections disclosed in the embodiments, reference may be made to the descriptions of the method sections for relevant parts.

Those skilled in the art can realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Skilled artisans may use different methods for implementing the described functionality for each particular application, but such implementations should not be considered beyond the scope of the disclosed embodiments. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described systems, devices and units can refer to the corresponding processes in the foregoing method embodiments, and details are not repeated here.

In the embodiments disclosed herein, the disclosed methods and products (including but not limited to apparatuses, devices, etc.) may be implemented in other ways. For example, the apparatus embodiments described above are only illustrative. For example, the division of the units may only be a logical function division. In actual implementation, there may be other division methods, for example, multiple units or components may be combined Either it can be integrated into another system, or some features can be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. This embodiment may be implemented by selecting some or all of the units according to actual needs. In addition, each functional unit in the embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more functions for implementing the specified logical function(s) executable instructions. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, operations or steps corresponding to different blocks may also occur in different sequences than those disclosed in the description, and sometimes there is no specific relationship between different operations or steps. order. For example, two consecutive operations or steps may, in fact, be performed substantially concurrently, or they may sometimes be performed in the reverse order, depending upon the functionality involved. Each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in special purpose hardware-based systems that perform the specified functions or actions, or special purpose hardware implemented in combination with computer instructions.

Claims (10)

1. A method for determining food material maturity is characterized by comprising the following steps:

acquiring a current frame image and a starting frame image of a cooked food material in kitchen electrical equipment, wherein the starting frame image is an image of the food material when cooking is started;

performing fusion processing on the current frame image and the initial frame image to obtain a current fusion processing image;

and inputting the current fusion processing image into a configured maturity machine learning training algorithm model, and determining the maturity corresponding to the food material.

2. The method of claim 1, wherein before the obtaining the current frame image and the starting frame image of the food material cooked in the kitchen electrical appliance, further comprising:

extracting characteristic image information of a plurality of sample images based on a Convolutional Neural Network (CNN), wherein the sample images are obtained by fusing food material images with the ripeness calibrated and corresponding food material initial images;

and performing supervision training on each feature image information through a regression network to generate the maturity machine learning training algorithm model.

3. The method of claim 2, wherein the loss function in the regression network comprises:

wherein epsilon is a convergence coefficient, and deltax is a difference value between a predicted value and a calibrated value.

4. The method of claim 2, wherein the maturity calibration process comprises:

under the condition that the cooking of food materials is started, controlling a camera to start recording of cooking videos, and under the condition that the maturity of the food materials reaches the set maturity, controlling the camera to stop recording of the cooking videos;

determining the total frame number of the recorded cooking videos;

and determining the maturity corresponding to the first image according to the frame number corresponding to the first image, the total frame number and the set maturity.

5. The method of claim 1, wherein the obtaining the current frame image and the starting frame image of the cooked food material in the kitchen appliance comprises:

when the method is applied to kitchen electrical equipment, when cooking is started, a camera is controlled to start recording of a cooking video, a starting frame image corresponding to the food material is obtained and stored, and a current frame image of the food material is obtained through the camera;

in the case that the method is applied to a server, receiving a current frame image of the food material sent by the kitchen electrical equipment and a stored starting frame image of the food material.

6. The method of claim 1 or 5, wherein after determining the maturity corresponding to the food material, further comprising:

when the method is applied to kitchen electrical equipment, information reminding of maturity is carried out;

and under the condition that the method is applied to a server, sending the maturity to the kitchen electric equipment for information reminding of the maturity.

7. An apparatus for determining the ripeness of a food material, comprising:

the cooking system comprises an information acquisition module, a processing module and a display module, wherein the information acquisition module is configured to acquire a current frame image and a starting frame image of a cooked food material in the kitchen electrical equipment, and the starting frame image is an image of the food material when cooking is started;

the image fusion module is configured to perform fusion processing on the current frame image and the initial frame image to obtain a current fusion processing image;

and the prediction determining module is configured to input the current fusion processing image into a configured maturity machine learning training algorithm model, and determine the maturity corresponding to the food material.

8. The apparatus of claim 7, further comprising:

the model configuration module is configured to extract feature image information of a plurality of sample images based on a convolutional neural network CNN, wherein the sample images are obtained by performing fusion processing on the food material images with the calibrated maturity and corresponding food material initial images; performing supervision training on each feature image information through a regression network to generate the maturity machine learning training algorithm model; the loss function in the regression network includes:

wherein epsilon is a convergence coefficient, and deltax is a difference value between a predicted value and a calibration value;

the calibration module is configured to control the camera to start recording of the cooking video when the cooking of the food materials is started, and control the camera to stop recording of the cooking video when the maturity of the food materials reaches the set maturity; determining the total frame number of the recorded cooking videos; and determining the maturity corresponding to the first image according to the frame number corresponding to the first image, the total frame number and the set maturity.

9. A kitchen appliance, characterized in that it comprises a device according to any one of claims 7 to 8.

10. A server, characterized in that it comprises a device according to any one of claims 7 to 8.

Images