WO2018040105A1

WO2018040105A1 - System and method for food recognition, food model training method, refrigerator and server

Info

Publication number: WO2018040105A1
Application number: PCT/CN2016/098120
Authority: WO
Inventors: 杨世清; 石周; 唐红强
Original assignee: 合肥华凌股份有限公司; 合肥美的电冰箱有限公司; 美的集团股份有限公司
Priority date: 2016-09-05
Filing date: 2016-09-05
Publication date: 2018-03-08

Abstract

Disclosed is a food recognition system. The food recognition system comprises: a refrigerator and a server, wherein the refrigerator comprises an image collection apparatus used for collecting images of food in the refrigerator. The server acquires the images of food in the refrigerator and recognizes the images according to a food model so as to acquire food information. The food model is a neural network acquired by training by means of a deep learning algorithm. The food recognition system has a high recognition ability and model generalization ability. Also disclosed are a food recognition method, a food model training method, a refrigerator and a server.

Description

Food identification system and method, food model training method, refrigerator and server

Technical field

The invention belongs to the technical field of electrical appliances manufacturing, and particularly relates to a food material identification system, a food material identification method, a food material model training method, a refrigerator and a server.

Background technique

With the development of integrated circuits, artificial intelligence, and Internet technologies, the traditional white goods industry has taken on a new position. For example, refrigerators not only carry the function of preserving food, but also become part of the home network, providing more for family members. Intelligent service. Among them, food identification as a module for front-end information collection provides a basis for the establishment of a subsequent food library.

The traditional image recognition technology has low recognition rate and low real-time performance in complex scenes, and can not be well applied to the identification of a large number of ingredients in the refrigerator. In the traditional recognition algorithm, the computational complexity is not easy to use in embedded systems.

Summary of the invention

The present invention aims to solve at least one of the technical problems in the related art to some extent.

To this end, an object of the present invention is to provide a food material identification system which has a high recognition rate and a strong model generalization ability.

The invention also provides a food material identification method, an food material model training method, and a refrigerator and a server.

In order to solve the above problems, a food material identification system according to an aspect of the present invention includes: a refrigerator, the refrigerator includes an image collection device, the image collection device is configured to collect an image of the foodstuff in the refrigerator; and the server obtains the refrigerator. An image of the inner food material, and the image is identified according to a food material model, the food material model being a neural network obtained by training through a deep learning algorithm.

The food material identification system of the embodiment of the invention combines local photographing and remote recognition, and uses the computing power of the server to apply the deep learning algorithm to image recognition, and the recognition rate is improved, which can be better applied to the foodstuffs with a larger amount in the refrigerator. Identification.

Further, the image collection device includes: a camera module for collecting an image of the foodstuff in the refrigerator; a communication module, configured to send an image of the foodstuff in the refrigerator to the server; and a control module, respectively Controlling the camera module and the communication module.

The image capturing device further includes: a lighting module, configured to illuminate an environment in which the foodstuff in the refrigerator is located, and is more convenient for the camera module to collect the image of the foodstuff.

Specifically, the control module, after receiving the door closing signal sent by the controller of the refrigerator, controls the camera module to collect an image of the foodstuff in the refrigerator, and controls the communication module to send the image to the office Said server.

Further, the food material model is obtained by training: taking an image of the foodstuff in the refrigerator that is pre-acquired and calibrated as a training image, and determining a model parameter according to the depth learning algorithm based on the training image to obtain the food material model.

Specifically, the food material model includes a convolutional neural network, a recurrent neural network, and a circulating neural network.

A food material identification method according to another aspect of the present invention comprises the steps of: obtaining an image of the foodstuff in the refrigerator; and identifying the image according to the food material model to obtain the food material information, wherein the food material model is obtained by training through a deep learning algorithm Neural network.

According to the food material identification method of the embodiment of the present invention, by using the deep learning method to identify the food material information, the recognition rate is improved, and the more complicated food material identification in the refrigerator can be better dealt with.

Wherein, the food material model is obtained by training: taking an image of the foodstuff in the refrigerator collected and calibrated in advance as a training image, and determining a model parameter according to the depth learning algorithm based on the training image to obtain the food material model.

Specifically, the food material model includes one of a convolutional neural network, a recurrent neural network, and a circulating neural network.

In order to solve the above problems, a food material model training method according to another embodiment of the present invention includes the steps of: obtaining an image of a food material in a refrigerator that is pre-acquired and calibrated as a training image; and processing the training image according to a depth learning algorithm. Determine model parameters to obtain a food model.

The food material model training method according to the embodiment of the present invention uses the deep learning algorithm to take the food image in the refrigerator as input data, and obtains the food material model which can be applied to the identification of the food in the refrigerator to improve the recognition rate.

Wherein the convolutional neural network comprises a convolutional layer, a pooling layer, an excitation layer and a fully connected layer, the input feature of the first layer is the training image, and the output features of each layer are used as input features of the next layer, according to The depth learning algorithm processing the training image further comprises: convolution layer performing feature compression on the input feature by convolution operation; the pooling layer performing pooling processing on the input feature; and the excitation layer obtaining the output feature by using the input feature through the excitation function, The output features are normalized; the input features of the fully connected layer and the output features are all reconnected between all nodes. Among them, the input data is pooled (downsampled), which can reduce the redundancy of parameters. Normalizing the data after inputting the excitation function can improve the effectiveness of backpropagation and improve the generalization ability of the model.

Generally, the convolutional neural network includes a plurality of convolutional layers, a pooling layer, and an excitation layer, and a fully connected layer of the tail of the convolutional neural network, wherein different convolutional cores of the convolutional layer acquire different Output characteristics. The input parameters are convoluted multiple times through different convolution kernels, and the parameters are more global.

In addition, the food material model training method further comprises: adjusting the model parameters of the food material model according to the actual food material information, thereby ensuring the generalization ability and the learning ability of the model.

In order to solve the above problems, a refrigerator according to still another aspect of the present invention includes: an image collecting device, the image capturing The apparatus is for taking an image of the foodstuff in the refrigerator and transmitting the image to a server, so that the server identifies the image according to the foodstuff model to obtain the foodstuff information, wherein the foodstuff model is trained by a deep learning algorithm And get the neural network.

The refrigerator of the embodiment of the invention sends the image of the collected food material to the server through the image collecting device, and provides a data foundation for the server to use the deep learning algorithm to identify the food material in the refrigerator.

The image collecting device further includes: a lighting module, configured to illuminate an environment in which the foodstuff in the refrigerator is located, and more conveniently collect the image of the foodstuff.

In order to solve the above problems, a server according to still another aspect of the present invention, the server obtains an image of the foodstuff in the refrigerator, and identifies the image according to the food material model to obtain the food material information, wherein the food material model passes Neural network obtained by deep learning algorithm training.

The server of the embodiment of the present invention applies the deep learning algorithm to image recognition based on its powerful computing capability, and has a simple method and improved recognition rate, and can be better applied to the identification of a large number of food materials in the refrigerator.

DRAWINGS

1 is a block diagram of a food material identification system in accordance with an embodiment of the present invention;

2 is a block diagram of a food material identification system in accordance with one embodiment of the present invention;

Figure 3 is a block diagram of a refrigerator in accordance with another embodiment of the present invention;

4 is a flow chart of a method for identifying a foodstuff according to an embodiment of the present invention;

FIG. 5 is a flowchart of a food material model training method according to an embodiment of the present invention; FIG.

6 is a process flow diagram of a convolutional neural network based algorithm in accordance with an embodiment of the present invention;

7 is a flow chart of feature extraction of a training image in accordance with another embodiment of the present invention;

Figure 8 is a flow chart of a food material model training process in accordance with yet another embodiment of the present invention;

Figure 9 is a block diagram of a refrigerator in accordance with one embodiment of the present invention;

Figure 10 is a block diagram of a refrigerator in accordance with another embodiment of the present invention.

detailed description

The embodiments of the present invention are described in detail below, and the examples of the embodiments are illustrated in the drawings, wherein the same or similar reference numerals are used to refer to the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the drawings are intended to be illustrative of the invention and are not to be construed as limiting.

The food material identification system, the food material identification method, the food material model training method, and the refrigerator and the server according to an embodiment of the present invention are described below with reference to the accompanying drawings.

1 is a block diagram of a food material identification system according to an embodiment of the present invention. As shown in FIG. 1, the food material identification system 1000 includes a refrigerator 100 and a server 200.

The refrigerator 100 includes an image capture device 10 for collecting images of foodstuffs in the refrigerator. The refrigerator 100 and the server 200 can perform data interaction. The server 200 obtains an image of the foodstuff in the refrigerator, and identifies the obtained image according to the food material model to obtain the food material information, such as the category of the foodstuff and its coordinates, wherein the food material model is a neural network obtained by training the deep learning algorithm, and the deep learning is performed. The algorithm introduces the identification of the food in the refrigerator, and by increasing the number of neural network layers, the mapping relationship between output and input can be better, and the recognition accuracy is improved.

The food material identification system 1000 of the embodiment of the present invention combines local photographing and remote recognition, and utilizes the computing power of the server 200 to apply the deep learning algorithm to image recognition. The method is simple, the recognition rate is improved, and the method can be better applied to the refrigerator 100. Identification of a large number of ingredients.

Further, as shown in FIG. 2, the image capture device 10 includes a camera module 11, a communication module 12, and a control module 13. The camera module 11 is configured to collect an image of the foodstuff in the refrigerator 100; the communication module 12 is configured to send an image of the foodstuff in the refrigerator 100 to the server 200; and the control module 13 is configured to respectively control the camera module 11 and the communication module 12.

As shown in FIG. 2, the image capture device 10 further includes a lighting module 14 for illuminating the environment in which the foodstuffs in the refrigerator 100 are located, so that the camera module 11 can capture the image of the foodstuff.

The control module 13 controls the camera module 11 to collect an image of the foodstuff in the refrigerator after receiving the door closing signal from the controller of the refrigerator 100, and controls the communication module 12 to send the image to the server 200.

Specifically, as shown in FIG. 3, the refrigerator in the embodiment of the present invention can be provided with an image capturing device 10, including a control module 13, such as a CPU (Central Processing Unit), and a refrigerator thereof. Related peripherals, the control circuit 01 of the refrigerator 100 is controlled by a controller 02 such as an MCU (Microcontroller Unit), the CPU and the MCU can perform data interaction, and the CPU peripheral circuit can mount related peripherals, in the present invention. In an embodiment, the related peripherals include a lighting module 14, a camera module 11 and a communication module 12 such as a WIFI module. The CPU controls the camera module 11 to cooperate with the illumination module 14 to complete the shooting of the local food image. In the process of identifying the ingredients, when the MCU detects the door closing signal, it means that the ingredients in the refrigerator may change. At this time, the control module 13 controlling the camera module 11 and the illumination module 14 to jointly capture an image, uploading the captured image to the server 200 through the WIFI module, and the server 200 performs identification of the food material information, after which the server 200 can feed the identification information to the local or provide Give relevant personnel the recognition results.

In an embodiment of the present invention, the food product model deployed on the server 200 can be obtained by training: taking an image of the food in the refrigerator that is pre-acquired and calibrated as a training image, and determining a model parameter according to the depth learning algorithm based on the training image. To get a model of the ingredients. Among them, the food material model may include one of a convolutional neural network, a recurrent neural network, and a cyclic neural network, and the model parameters may be determined by a BP algorithm.

Specifically, the food model training can be completed on the off-line side, taking the image of the food in the refrigerator or in the simulated refrigerator environment and manually marking the position of various ingredients, obtaining the input vector through the operation of the deep learning algorithm, and then pre-calibrating As a result, the whole connection is performed, and the parameters of the model are determined by training a large number of training images, which is equivalent to determining the coefficient in the functional relationship. When the parameter training is completed, the food material model is obtained, and the food material model is deployed on the server 200. The training process for the food model is described in detail below.

A food material identification method according to another embodiment of the present invention will now be described with reference to the accompanying drawings.

4 is a flow chart of a method for identifying a food material according to an embodiment of the present invention. As shown in FIG. 4, the food material identification method includes the following steps:

S1, obtaining an image of the food in the refrigerator. For example, an image of the food in the refrigerator is collected by an image acquisition device and sent to a server.

S2, the image is identified according to the food material model to obtain the food material information, wherein the food material model is a neural network obtained by training through a deep learning algorithm.

Specifically, the food material model is obtained by training in the following manner: an image of the foodstuff in the refrigerator that is pre-acquired and calibrated is used as a training image, and the model parameter is determined based on the training image according to the depth learning algorithm to obtain the food material model. Among them, the food material model may include one of a convolutional neural network, a recurrent neural network, and a cyclic neural network, and the model parameters may be determined by a BP algorithm.

The food material identification method of the embodiment of the invention identifies the food material information by using the deep learning method, and the recognition rate is improved, and the method is simple, and the complex material identification in the refrigerator can be better dealt with.

The food material model training method of the embodiment of the present invention will be described below. FIG. 5 is a flowchart of a food material model training method according to an embodiment of the present invention. As shown in FIG. 5, the food material model training method includes the following steps:

S10, obtaining an image of the foodstuff in the refrigerator collected and calibrated in advance as a training image, for example, collecting an image of the foodstuff in the refrigerator by using an image collecting device of the refrigerator or photographing an image in a simulated refrigerator environment as a training image, and calibrating the position of the foodstuff, and uploading to the image The food model training end, the food model training can be carried out at the offline end.

S20. The training image is processed according to a depth learning algorithm to determine a model parameter to obtain a food material model.

In an embodiment of the invention, the food material model may include a convolutional neural network, a recurrent neural network, and a circulating neural network. One of them. Those skilled in the art understand that the training process of the model parameters of the neural network can be implemented by the BP algorithm, which has been described in many cases and will not be described here. There are different processing flows for different neural networks, and their differences in different data sets are mainly determined by the structure of the neural network. The input parameters are different for the same neural network structure, and the model parameters are also different. In the embodiment of the present invention, the food material model uses the image of the food material in the refrigerator as input data, determines the model parameters according to the deep learning method, and then deploys the food material model on the identification server, and applies the food material identification in the refrigerator to improve the recognition rate and facilitate Identification of complex situations.

According to the food material model training method of the embodiment of the present invention, the food material model in the refrigerator can be used as the input data by using the deep learning algorithm, and the food material model can be applied to the identification of the foodstuff in the refrigerator, and the recognition rate is improved.

Taking the convolutional neural network as an example, the training process of the food model, namely the food neural network, is described. Based on the deep learning method of convolutional neural network, the deep network can better fit the relationship between the input food image and the output recognition result, and has a high recognition rate.

In an embodiment of the invention, the convolutional neural network comprises a convolutional layer, a pooling layer, an excitation layer and a fully connected layer, the input features of the first layer are training images, and the output features of each layer are used as input features of the next layer. The processing of the training image according to the deep learning algorithm further comprises: the convolution layer performs feature compression on the input feature by convolution operation; the pooling layer performs pooling on the input feature, and the input data is pooled (downsampled), The redundancy of the parameters is reduced; the excitation layer obtains the output features through the excitation function, and normalizes the output features, and the commonly used excitation functions include Relu, Maxout, sigmoid, and the like. Wherein, the input data after the excitation function is normalized, which can improve the effectiveness of backpropagation and improve the generalization ability of the model; the input features of the fully connected layer and the output features have the right to reconnect between all nodes, ie, The output data of one layer and the output data of the next layer are fully connected. The full connection layer is usually at the end of the network, and the data of the last layer is the calibration result of the training image.

In a nutshell, the processing flow of the convolutional neural network is as shown in FIG. 6. For the input training image, the feature extraction is first performed to obtain the feature image, that is, the convolution operation is performed, and the feature image is downsampled, that is, pooled, and then, The feature vector obtained after the normalization process is input to the neural network as fully connected layer data. The parameters of the food material model are determined by a large amount of image training, thereby obtaining a food material model. Of course, the food model can adjust and design the network structure according to the actual situation and needs.

Specifically, when the food material model adopts the convolutional neural network structure, after obtaining the image of the food material in the refrigerator, that is, the training image, the training image is convoluted to obtain a feature image, and the feature image is pooled. Through convolution and pooling operations, the redundant information contained in the input vector can be reduced, the dimension is reduced, and the amount of computation is greatly reduced.

The various steps of the training of the food material model using the convolutional neural network structure are further explained below.

As shown in FIG. 7, the feature extraction process of the convolutional neural network includes: S710, obtaining a training image; S720, using a convolution kernel to perform convolution operation on the training image, S730, adding offset; S740, obtaining features image.

Among them, the training image is processed by the selected convolution kernel, firstly based on the principle of local perception field. General It is believed that people's perception of the outside world is from local to global, and the spatial connection of images is also a close connection of local pixels, while the correlation of pixels far away is weak. Therefore, it is not necessary for each neuron to perceive the global image. It only needs to perceive the local part. Then, the local information is combined at a higher level to obtain global information.

Secondly, it can be considered that the statistical features of the various parts of the image are the same, which also means that the features learned in this part can also be used in another part, so the same learning characteristics can be used for all positions on the training image. . The same convolution kernel can be used for processing to obtain a feature image.

Through convolution operations, dimensionality reduction of the image to be processed can be achieved. Using a feature to describe an image has a loss of information. Different convolution kernels can be used to convolve the training image separately to obtain a series of feature images.

After obtaining the feature image by the convolution operation, the next step can be used to classify these features. In theory, all the extracted features can be used to train the classifier, but the feature image expresses only a certain feature of the image, and there is still a large redundancy. With the spatial correlation of static pictures, redundancy can be further reduced by downsampling. This kind of aggregation operation is called pooling. According to the pooling method, it is also called average pooling or maximum pooling.

In order to prevent the distribution of training data and test data from being different, the problem of generalization ability is reduced, and the excitation function causes the data distribution to change, and the data is normalized after the excitation layer, that is, the data of the excitation function is normalized. deal with. At the same time, if the distribution of training data is different, the normalization process can also increase the training speed. Normalization can simply rescale the input parameters of each layer to a mean of 0 and a distribution of variance. Through normalization, generalization ability can be improved and training speed can be improved.

The food model can be completed at the offline end, as shown in Figure 8. The food model training process includes: S810, the training image is manually marked with the position of various ingredients; S820, convolution operation; S830, pool processing; S840, normalized The input operation obtains an input vector; in S850, the normalized processed data is fully connected with the calibration result of the training image.

Referring to FIG. 6, a layer convolution operation is employed, and in practical applications, the convolutional neural network generally includes a plurality of convolution layers, a pooling layer, and an excitation layer, and a convolutional neural network tail. Fully connected layer. That is to say, the training image is subjected to multiple convolution operations, pooling processes, and normalization, wherein different convolution kernels of the convolutional layer can be understood as acquiring different output features. Features obtained by a layer of convolution operation are often local, and the higher the number of layers, the more comprehensive the features obtained. Therefore, the use of multi-layer convolution can make the features more global, and the food model can be made as needed. Specific design.

In addition, the model parameters of the food material model can be adjusted based on the actual food material information. Specifically, the image of the food material uploaded to the server may be determined by manual calibration, and the actual food material information may be compared with the recognition result. For the image identifying the error or the food not included in the previous food product model, the above may be adopted. The training process of the food model is retrained to adjust the parameters of the food model.

Compared with the related art, it is unable to cope with the change of the food type, the replacement of the new package, and the like, and the fixed food model cannot be coped with the increase of the identification type. The food material training method of the present application can be based on the food material information for the food material model. The model parameters are adjusted to suit the new ingredients. In addition, compared with the related art, there is no effective monitoring for the recognition result, and the food with low recognition rate cannot be optimized. The method of the present application uses the supervised learning method to train the food material model, and detects the recognition result, and Model parameter adjustment can improve the generalization ability and learning ability of the model.

A refrigerator according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

9 is a block diagram of a refrigerator including an image capture device 10, as shown in FIG. 9, in accordance with an embodiment of the present invention. The image capture device 10 is configured to adopt an image of the foodstuff in the refrigerator, and transmit the image to a server, so that the server recognizes the image according to the food material model to obtain the food material information, wherein the food material model is a nerve obtained by training through a deep learning algorithm. The internet.

The refrigerator 100 of the embodiment of the present invention sends an image of the collected food material to the server through the image capturing device 10, and provides a data foundation for the server to use the deep learning algorithm to identify the foodstuff in the refrigerator.

Further, as shown in FIG. 10, the image capture device 10 includes a camera module 11, a communication module 12, and a control module 13. The camera module 11 is configured to collect an image of the foodstuff in the refrigerator; the communication module 12 is configured to send an image of the foodstuff in the refrigerator to the server; and the control module 13 is configured to respectively control the camera module 11 and the communication module 12.

Referring to FIG. 10, the image capture device 10 further includes a lighting module 14 for illuminating the environment in which the foodstuffs in the refrigerator 100 are located, so that the camera module 11 can be photographed to obtain an image of the foodstuff.

The server of the embodiment of the present invention will be described below. A food product model can be deployed in the server 100. Specifically, when the server 200 identifies the foodstuff in the refrigerator, the server 100 obtains an image of the foodstuff in the refrigerator, and identifies the image according to the foodstuff model to obtain the foodstuff information, such as the category of the foodstuff and its coordinates, wherein the foodstuff model is A neural network obtained by training in deep learning algorithms. The deep learning algorithm is introduced into the identification of the food in the refrigerator, and the mapping relationship between the output and the input can be better by increasing the number of neural network layers, thereby improving the recognition accuracy.

The server 200 of the embodiment of the present invention applies the deep learning algorithm to image recognition based on its powerful computing capability, and has a simple method and improved recognition rate, and can be better applied to the identification of a large number of food materials in the refrigerator.

The food material model is obtained by training in the following manner: an image of the food in the refrigerator that is pre-acquired and calibrated is used as a training image, and the model parameter is determined according to the depth learning algorithm based on the training image to obtain a food material model. In an embodiment of the present invention, the food material model may include one of a convolutional neural network, a recurrent neural network, and a cyclic neural network, and the model parameters may be determined by a BP algorithm.

Specifically, the food model training can be completed at the offline end, the image of the food in the refrigerator is photographed or the image of the food in the simulated refrigerator environment is photographed, and the position of various ingredients is manually marked, and the input vector is obtained by the operation of the deep learning algorithm, and further Fully connected to the pre-calibrated results, trained through a large number of images to determine model parameters, equivalent to The coefficient in the function relationship is obtained, and the parameter training is completed to obtain the food material model, and the food material model is deployed on the server 200, so that when the image recognition is performed, the server 200 can input the obtained food material image in the refrigerator as an input, according to the food material model. Realize the identification of the location and category of the ingredients, and provide identification results, which can be fed back to the refrigerator or provided to the appropriate personnel.

In summary, the embodiments of the present invention provide a food material identification system, an food material identification method, a food material model training method, a refrigerator, and a server, and apply a deep learning algorithm to the food material identification, using a combination of local camera and online recognition. The computing power of the server performs image recognition. Among them, in the training stage of the food material model, a large number of food material images are collected, and the food ingredients in the image are marked, for example, 4-point mark, and the spatial coordinate information and the pixel information are processed for the food material to establish a deep neural network model for various food materials. In the training process of the food material model, the feature mapping method is used to extract various characteristic parameters of the image, that is, the features of the image are preserved, and the redundancy is removed, and the operation is greatly reduced. In the identification phase, after uploading the local image, the food product model is matched on the server side to generate the food and its coordinates. In the inspection stage, by manually calibrating the food and position of the image to be measured, comparing the recognition results of the server, the recognition rate result is obtained, and the model parameters are adjusted through retraining to improve the actual performance.

It should be noted that, in the description of the specification, any process or method description in the flowcharts or otherwise described herein may be understood to include one or more steps for implementing a particular logic function or process. Modules, segments or portions of code of executable instructions, and the scope of preferred embodiments of the invention includes additional implementations, which may not be in the order shown or discussed, including in a substantially simultaneous manner depending on the functionality involved. The functions are performed in the reverse order, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

The logic and/or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer readable medium, Used in conjunction with, or in conjunction with, an instruction execution system, apparatus, or device (eg, a computer-based system, a system including a processor, or other system that can fetch instructions and execute instructions from an instruction execution system, apparatus, or device) Or use with equipment. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport a program for use in an instruction execution system, apparatus, or device, or in conjunction with the instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections (electronic devices) having one or more wires, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer readable medium may even be a paper or other suitable medium on which the program can be printed, as it may be optically scanned, for example by paper or other medium, followed by editing, interpretation or, if appropriate, other suitable The method is processed to obtain the program electronically and then stored in computer memory.

It should be understood that portions of the invention may be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system to realise. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques well known in the art: having logic gates for implementing logic functions on data signals. Discrete logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

One of ordinary skill in the art can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, one or a combination of the steps of the method embodiments is included.

In the description of the present specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means a specific feature described in connection with the embodiment or example. A structure, material or feature is included in at least one embodiment or example of the invention. In the present specification, the schematic representation of the above terms is not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, various embodiments or examples described in the specification, as well as features of various embodiments or examples, may be combined and combined.

Although the embodiments of the present invention have been shown and described, it is understood that the above-described embodiments are illustrative and are not to be construed as limiting the scope of the invention. The embodiments are subject to variations, modifications, substitutions and variations.

Claims

An food material identification system, comprising:

a refrigerator, the refrigerator includes an image capture device, and the image capture device is configured to collect an image of the foodstuff in the refrigerator;

a server that obtains an image of the foodstuff in the refrigerator and identifies the image according to a food product model, the food material model being a neural network obtained by training through a deep learning algorithm.
The food material identification system of claim 1 wherein said image capture device comprises:

a camera module, configured to collect an image of the foodstuff in the refrigerator;

a communication module, configured to send an image of the foodstuff in the refrigerator to the server; and

And a control module, configured to separately control the camera module and the communication module.
The food material identification system of claim 2, wherein the image capture device further comprises:

A lighting module for illuminating the environment in which the ingredients in the refrigerator are located.
The food material identification system according to claim 2, wherein the control module controls the camera module to collect an image of the foodstuff in the refrigerator after receiving a door closing signal from the controller of the refrigerator, and The communication module is controlled to send the image to the server.
The food material identification system according to claim 1, wherein the food material model is obtained by training in the following manner:

An image of the foodstuff in the refrigerator pre-acquired and calibrated is used as a training image, and model parameters are determined according to the depth learning algorithm based on the training image to obtain the food material model.
The food material identification system according to claim 5, wherein the food material model comprises one of a convolutional neural network, a recurrent neural network, and a circulating neural network.
A food material identification method, comprising the steps of:

Obtain an image of the ingredients in the refrigerator;

The image is identified based on a food product model, wherein the food material model is a neural network obtained by training through a deep learning algorithm.
The food material identifying method according to claim 7, wherein the food material model is obtained by training in the following manner:

An image of the foodstuff in the refrigerator pre-acquired and calibrated is used as a training image, and model parameters are determined according to the depth learning algorithm based on the training image to obtain the food material model.
The food material identifying method according to claim 8, wherein the food material model comprises one of a convolutional neural network, a recurrent neural network, and a circulating neural network.
A food material model training method, comprising the steps of:

Obtaining an image of the foodstuff in the refrigerator that is pre-acquired and calibrated as a training image;

The training image is processed according to a depth learning algorithm to determine model parameters to obtain a food product model.
The food material model training method according to claim 10, wherein the food material model comprises one of a convolutional neural network, a recurrent neural network, and a circulating neural network.
The food material model training method according to claim 11, wherein the convolutional neural network comprises a convolution layer, a pooling layer, an excitation layer and a fully connected layer, and an input characteristic of the first layer is the training The image, the output feature of each layer is used as an input feature of the next layer, and processing the training image according to the depth learning algorithm further includes:

The convolution layer performs feature compression on the input features by convolution operations;

The pooling layer performs pooling processing on input features;

The excitation layer obtains an output characteristic by the input characteristic through an excitation function, and normalizes the output feature;

The input features and output features of the fully connected layer have the right to be reconnected between all nodes.
The food material model training method according to claim 12, wherein said convolutional neural network comprises a plurality of convolution layers, a pooling layer and an excitation layer, and a fully connected layer at the tail of said convolutional neural network, Among them, the different convolution kernels of the convolutional layer acquire different output characteristics.
The food material model training method according to claim 10, further comprising: adjusting a model parameter of the food material model based on actual food material information.
A refrigerator, comprising:

An image capture device for taking an image of foodstuffs in a refrigerator and transmitting the image to a server to cause the server to identify the image according to a food product model to obtain food material information, wherein The food material model is a neural network obtained by training through a deep learning algorithm.
The refrigerator according to claim 15, wherein said image capture device comprises:

a camera module, configured to collect an image of the foodstuff in the refrigerator;

a communication module, configured to send an image of the foodstuff in the refrigerator to the server; and

And a control module, configured to separately control the camera module and the communication module.
The refrigerator according to claim 16, wherein the image capture device further comprises:

A lighting module for illuminating the environment in which the ingredients in the refrigerator are located.
The refrigerator according to claim 16, wherein the control module controls the camera module to collect an image of the foodstuff in the refrigerator after receiving the door closing signal from the controller of the refrigerator, and controls the The communication module transmits the image to the server.
A server, wherein the server obtains an image of the foodstuff in the refrigerator, and identifies the image according to the food material model to obtain the food material information, wherein the food material model is obtained by training through a deep learning algorithm Neural network.
The server according to claim 19, wherein said food material model is obtained by training in the following manner:

An image of the foodstuff in the refrigerator pre-captured and calibrated is used as a training image, and model parameters are determined based on the training image according to the depth learning algorithm to obtain the food material model.
The server according to claim 19, wherein said food material model comprises one of a convolutional neural network, a recurrent neural network, and a circulating neural network.