CN115046363B - Intelligent refrigerator based on machine vision and spectrum detection and operation method - Google Patents
Intelligent refrigerator based on machine vision and spectrum detection and operation method Download PDFInfo
- Publication number
- CN115046363B CN115046363B CN202210768651.XA CN202210768651A CN115046363B CN 115046363 B CN115046363 B CN 115046363B CN 202210768651 A CN202210768651 A CN 202210768651A CN 115046363 B CN115046363 B CN 115046363B
- Authority
- CN
- China
- Prior art keywords
- layer
- input
- output
- food material
- food
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001228 spectrum Methods 0.000 title claims abstract description 30
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000001514 detection method Methods 0.000 title claims abstract description 18
- 239000000463 material Substances 0.000 claims abstract description 137
- 230000000007 visual effect Effects 0.000 claims abstract description 64
- 238000004321 preservation Methods 0.000 claims abstract description 29
- 238000013135 deep learning Methods 0.000 claims abstract description 9
- 230000003595 spectral effect Effects 0.000 claims description 33
- 238000010521 absorption reaction Methods 0.000 claims description 27
- 238000013528 artificial neural network Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 18
- 238000010606 normalization Methods 0.000 claims description 17
- 238000004422 calculation algorithm Methods 0.000 claims description 16
- 238000012549 training Methods 0.000 claims description 12
- 230000007246 mechanism Effects 0.000 claims description 10
- 238000009413 insulation Methods 0.000 claims description 9
- 230000009467 reduction Effects 0.000 claims description 9
- 238000000513 principal component analysis Methods 0.000 claims description 8
- 238000007637 random forest analysis Methods 0.000 claims description 8
- 238000004891 communication Methods 0.000 claims description 7
- 238000002835 absorbance Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 238000013139 quantization Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 3
- 230000001678 irradiating effect Effects 0.000 claims description 3
- 230000008447 perception Effects 0.000 claims description 3
- 238000004458 analytical method Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000012544 monitoring process Methods 0.000 abstract description 2
- 238000009659 non-destructive testing Methods 0.000 abstract 1
- 239000010410 layer Substances 0.000 description 163
- 239000013598 vector Substances 0.000 description 10
- 239000011159 matrix material Substances 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 238000011176 pooling Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 239000011229 interlayer Substances 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 235000015219 food category Nutrition 0.000 description 1
- 238000009920 food preservation Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000005057 refrigeration Methods 0.000 description 1
Classifications
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D29/00—Arrangement or mounting of control or safety devices
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D11/00—Self-contained movable devices, e.g. domestic refrigerators
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D19/00—Arrangement or mounting of refrigeration units with respect to devices or objects to be refrigerated, e.g. infrared detectors
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D2400/00—General features of, or devices for refrigerators, cold rooms, ice-boxes, or for cooling or freezing apparatus not covered by any other subclass
- F25D2400/36—Visual displays
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D2700/00—Means for sensing or measuring; Sensors therefor
- F25D2700/06—Sensors detecting the presence of a product
Landscapes
- Engineering & Computer Science (AREA)
- Chemical & Material Sciences (AREA)
- Combustion & Propulsion (AREA)
- Physics & Mathematics (AREA)
- Mechanical Engineering (AREA)
- Thermal Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
The invention discloses an intelligent refrigerator based on machine vision and spectrum detection and an operation method thereof, wherein the intelligent refrigerator comprises at least one vision sensing module and at least one spectrometer module; the visual sensing modules and the spectrometer modules are distributed and installed in the heat preservation box body and face to food materials stored in the heat preservation box body; the output ends of the vision sensing module and the spectrometer module are electrically connected with the core controller. The method for identifying the images by combining machine vision and deep learning and the technology for detecting the freshness of the food materials through spectrum nondestructive testing are utilized to judge the variety and freshness of the food materials stored in the refrigerator, so that the requirements of people on food material management and food material freshness monitoring in daily life are met, and a novel application example is provided for the following intelligent refrigerator market.
Description
Technical Field
The invention relates to the technical field of intelligent refrigerators, in particular to an intelligent refrigerator based on machine vision and spectrum detection and an operation method thereof.
Background
The intelligent refrigerator is a refrigerator type capable of performing intelligent management on food and intelligent control on the refrigerator. In the aspect of food material identification, the conventional intelligent refrigerator generally photographs food materials to be placed in the refrigerator through a visual sensing module arranged in the intelligent refrigerator, then the intelligent refrigerator performs image processing on photographed food material pictures, and after color characteristic values and shape characteristic values of the photographed food materials are extracted, the photographed food material pictures are matched with a local user food information base to determine food material types. In addition, in the aspect of food freshness, the existing intelligent refrigerator generally sets a time threshold after food is put into the refrigerator, and reminds a user when storage time reaches the set threshold. However, the time is not accurately reflected to the freshness of the food materials due to the influence of various complex factors in the transportation, purchase and storage processes, the storage time of the food materials cannot be accurately predicted by fixed time reminding, and the reminding requirement of food preservation cannot be met.
Disclosure of Invention
The invention aims to solve the problems of the existing intelligent refrigerator in the aspects of food material identification and fresh-keeping, and provides an intelligent refrigerator based on machine vision and spectrum detection and an operation method thereof.
In order to solve the problems, the invention is realized by the following technical scheme:
The intelligent refrigerator based on machine vision and spectrum detection comprises an insulation box body, a refrigerating system, an electrical appliance system and a core controller; the refrigerating system, the electric appliance system and the core controller are all positioned in the heat insulation box body, and the refrigerating system is electrically connected with the electric appliance system and the core controller; characterized by further comprising at least one vision sensing module and at least one spectrometer module; the visual sensing modules and the spectrometer modules are distributed and installed in the heat preservation box body and face to food materials stored in the heat preservation box body; the output ends of the vision sensing module and the spectrometer module are electrically connected with the core controller.
Each interlayer of the insulation box body of the intelligent refrigerator is provided with a visual sensing module and a spectrometer module.
The intelligent refrigerator further comprises a communication module which is arranged in the heat preservation box body and is electrically connected with the core controller.
The operation method of the intelligent refrigerator based on machine vision and spectrum detection comprises the following steps:
step 1, shooting given food samples with different types and different freshness under multi-angle and multi-background environments, and performing image preprocessing on the shot images to obtain food sample images;
step 2, manually marking the food material types of the food material sample image obtained in the step 1, and creating a food material sample image data set;
step 3, constructing a food material type identification model based on YOLOv-ViT neural network;
step 4, performing deep learning training on the food material type recognition model in the step 2 by using the food material sample image dataset in the step 2 to obtain a trained food material type recognition model, and disposing the trained food material type recognition model on a core controller of the intelligent refrigerator after model quantization;
Step 5, each visual sensing module of the intelligent refrigerator performs image acquisition and image preprocessing on food materials stored in the heat preservation box body and then sends the food materials into a core controller, and the core controller utilizes the deployed food material type recognition model to recognize the food materials in the heat preservation box body of the intelligent refrigerator to obtain the types of the food materials and count the number of various food materials in the heat preservation box body of the intelligent refrigerator;
Step 6, firstly, irradiating given food material samples with different types and different freshness by utilizing a spectrometer to obtain spectral data of the food material samples; performing data dimension reduction on the spectrum data by using a principal component analysis algorithm, and extracting characteristic wavelengths by using a random forest algorithm; then, utilizing the extracted characteristic wavelength to establish a spectrum index reflecting the characteristic of an absorption peak of the freshness factor of the food material;
Step 7, manually marking the food freshness of the spectral indexes of the food samples obtained in the step 6, and creating a spectral index data set of the food samples;
Step 8, constructing a food freshness identification model based on an LDA neural network;
Step 9, performing deep learning training on the food material freshness identification model in step 8 by using the food material sample spectral index data set in step 7 to obtain a trained food material freshness identification model, and disposing the trained food material freshness identification model on a core controller of the intelligent refrigerator after model quantization;
And 10, each spectrometer module of the intelligent refrigerator collects spectral data of food materials stored in the heat preservation box body, performs data dimension reduction on the spectral data by utilizing a principal component analysis algorithm, performs characteristic wavelength extraction by utilizing a random forest algorithm, obtains a spectral index by utilizing the extracted characteristic wavelength, and then sends the spectral index into a core controller, and the core controller identifies and determines the freshness of the food materials in the heat preservation box body of the intelligent refrigerator by utilizing a deployed food material freshness identification model.
The food material type recognition model constructed in the step 3 is composed of an image slice layer, a position embedding layer, 6 visual coding modules, 2 tensor splicing layers, 3 residual convolution layers, 2 upsampling layers, 8 convolution layers and a multi-scale prediction layer; the input of the image slice layer is used as the input of the food material type identification model, the output of the image slice layer is connected with the input of the position embedding layer, the output of the position embedding layer is connected with the input of the first visual coding module, the output of the first visual coding module is connected with the input of the second visual coding module, the output of the second visual coding module is connected with the input of the third visual coding module, the output of the third visual coding module is connected with the input of the fourth visual coding module, one output of the fourth visual coding module is connected with the input of the fifth visual coding module, and one output of the fifth visual coding module is connected with the input of the sixth visual coding module; the output of the sixth visual coding module is connected with the input of the first residual convolution layer, the other output of the fifth visual coding module is connected with one input of the first tensor splicing layer, and the other output of the fourth visual coding module is connected with one input of the second tensor splicing layer; one output of the first residual error convolution layer is connected with the input of the first convolution layer, the output of the first convolution layer is connected with the input of the first upsampling layer, the input of the first upsampling layer is connected with the other input of the first tensor splicing layer, one output of the first tensor splicing layer is connected with the other input of the second tensor splicing layer, and the output of the second tensor splicing layer is connected with the input of the second residual error convolution layer; the other output of the first tensor splicing layer is connected with the input of a third residual convolution layer, one output of the third residual convolution layer is connected with the input of a second convolution layer, the output of the second convolution layer is connected with the input of the third convolution layer, and the output of the third convolution layer is connected with one input of a multi-scale prediction layer; the other output of the third residual convolution layer is connected with the input of a fourth convolution layer, the output of the fourth convolution layer is connected with the input of a second upsampling layer, and the output of the second upsampling layer is connected with the other input of the second tensor splicing layer; the other output of the first residual convolution layer is connected with the input of a fifth convolution layer, the output of the fifth convolution layer is connected with the input of a sixth convolution layer, and the output of the sixth convolution layer is connected with the other input of the multi-scale prediction layer; the output of the second residual convolution layer is connected with the input of a seventh convolution layer, the output of the seventh convolution layer is connected with the input of an eighth convolution layer, and the output of the eighth convolution layer is connected with the other input of the multi-scale prediction layer; the output of the multi-scale prediction layer is used as the output of the food material type identification model;
In the step 3, each visual coding module of the food material type recognition model consists of a picture embedding layer, 2 normalization function layers, a multi-head attention layer, 2 merging layers and a multi-layer perception mechanism layer; the input of the picture embedding layer is used as the input of the visual coding module; one output of the picture embedding layer is connected with the input of the first normalization function layer, the output of the first normalization function layer is connected with the input of the multi-head attention layer, and the output of the multi-head attention layer is connected with one input of the first merging layer; the other output of the picture embedding layer is connected with the other input of the first merging layer; one output of the first merging layer is connected with the input of the second normalization function layer, the output of the second normalization function layer is connected with the input of the multi-layer sensing mechanism layer, and the output of the multi-layer sensing mechanism layer is connected with one input of the second merging layer; the other output of the first merging layer is connected with the other input of the second merging layer; the output of the second merge layer serves as the output of the visual coding module.
In the step 6, the spectral indexes reflecting the characteristics of the absorption peaks of the freshness factor of the food material include an absorption peak depth index, a relative depth index of the absorption peaks, a normalized intensity difference index and an absorbance difference index; wherein:
The absorption peak depth index a is:
A＝R3+R1-2R2
The relative depth index B of the absorption peak is:
B＝R2/(R3+R1)
The normalized intensity difference index C is:
C＝(R3-R2)/(R3+R2)
The absorbance difference index D is:
D＝log10(R3/R2)
wherein R 1 is a characteristic wavelength of a start point of an absorption peak, R 2 is a characteristic wavelength of a lowest point of the absorption peak, and R 3 is a characteristic wavelength of an end point of the absorption peak.
The step 9 further includes, when the food material in the heat preservation box body of the refrigerator is determined to be not fresh, giving an alarm to the user.
The operation method of the intelligent refrigerator based on machine vision and spectrum detection further comprises the following steps:
And 11, transmitting the type and the quantity of the food materials obtained in the step 5 and the freshness of the food materials obtained in the step 10 to a user terminal by utilizing a communication module.
Compared with the prior art, the invention utilizes the image recognition method combining machine vision and deep learning and the spectrum nondestructive detection technology of the freshness of the food materials to realize the discrimination of the variety and freshness of the food materials stored in the refrigerator, so as to solve the requirements of people on food material management and food material freshness monitoring in daily life, and provide a novel application example for the subsequent intelligent refrigerator market.
Drawings
Fig. 1 is a schematic structural diagram of an intelligent refrigerator based on machine vision and spectrum detection.
Fig. 2 is a flow chart of an operation method of the intelligent refrigerator based on machine vision and spectrum detection.
Fig. 3 is a schematic structural diagram of a food material type identification model based on YOLOv-ViT neural networks.
Fig. 4 is a schematic structural diagram of the visual coding module in fig. 3.
The figures indicate: 1. a thermal insulation box body; 2. a visual sense module; 3. a spectrometer module.
Detailed Description
The present invention will be further described in detail with reference to specific examples in order to make the objects, technical solutions and advantages of the present invention more apparent.
An intelligent refrigerator based on machine vision and spectrum detection is shown in fig. 1, and comprises an insulation box body 1, a refrigerating system, an electrical system and a core controller. The refrigeration system comprises a compression, a condenser, an evaporator, a capillary tube, a connecting pipeline and the like. The electrical system comprises a temperature controller, an electromagnetic valve, an illuminating lamp, a display screen and the like. The heat preservation box body 1 comprises a box body and a door body, wherein the door body is hinged to the box body, and the box body and the door body form a sealed heat preservation box body 1 for storing food materials. The refrigerating system, the electrical system and the core controller are all positioned in the heat preservation box body 1, and the refrigerating system is electrically connected with the electrical system and the core controller. In order to realize machine vision and spectrum detection, at least one vision sensing module 2 and at least one spectrometer module 3 are distributed in the heat preservation box body 1. In the preferred embodiment of the invention, the visual sensing module 2 is a camera module, the spectrometer module 3 is a micro spectrometer module 3, each storage interlayer of the heat insulation box 1 is provided with the visual sensing module 2 and the spectrometer module 3, and the visual sensing module 2 and the spectrometer module 3 face toward food materials stored in the heat insulation box 1. All vision sensing modules 2 and all spectrometer modules 3 are electrically connected to the core controller. In order to realize networking, a communication module is arranged in the heat preservation box body 1. The communication module is electrically connected with the core controller.
The operation method of the intelligent refrigerator based on machine vision and spectrum detection, as shown in fig. 2, specifically comprises the following steps:
And step 1, shooting given food samples with different types and different freshness under multi-angle and multi-background environments, and performing image preprocessing on the shot images to obtain food sample images.
In this embodiment, the camera, the mobile phone or the vision sensing module 2 is used to shoot the food material samples subjected to classification recognition at a plurality of angles and in a plurality of background environments, and the shot images are preprocessed into RGB color images with the size 224×224. The food material sample image comprises 4000 food materials which are common in the market.
And 2, manually labeling the food material types of the food material sample image obtained in the step 1, and creating a food material sample image data set.
And labeling the category of 4000 food material sample images of common food materials in the market by labellmg, and creating a food material sample image dataset. Scaling 7 the created food sample image dataset: 2:1 is divided into a training set, a testing set and a verification set.
And 3, constructing a food material type identification model based on YOLOv-ViT neural network.
The model for identifying the food material type is a YOLOv-ViT neural network obtained by improving a traditional YOLOv-3 neural network, and comprises an image slicing layer, a position embedding layer, 6 visual coding modules, 2 tensor splicing layers, 3 residual convolution layers, 2 upsampling layers, 8 convolution layers and a multi-scale prediction layer. As shown in fig. 3. The input of image section layer is as the input of edible material kind discernment model, the input of position embedding layer is connected in the output of image section layer, the input of first vision coding module is connected in the output of position embedding layer, the input of second vision coding module is connected in the output of first vision coding module, the input of third vision coding module is connected in the output of second vision coding module, the input of fourth vision coding module is connected in the output of third vision coding module, the input of fifth vision coding module is connected in one output of fourth vision coding module, the input of sixth vision coding module is connected in one output of fifth vision coding module. The output of the sixth visual coding module is connected with the input of the first residual convolution layer, the other output of the fifth visual coding module is connected with one input of the first tensor splicing layer, and the other output of the fourth visual coding module is connected with one input of the second tensor splicing layer. One output of the first residual error convolution layer is connected with the input of the first convolution layer, the output of the first convolution layer is connected with the input of the first up-sampling layer, the input of the first up-sampling layer is connected with the other input of the first tensor splicing layer, one output of the first tensor splicing layer is connected with the other input of the second tensor splicing layer, and the output of the second tensor splicing layer is connected with the input of the second residual error convolution layer. The other output of the first tensor splicing layer is connected with the input of a third residual convolution layer, one output of the third residual convolution layer is connected with the input of a second convolution layer, the output of the second convolution layer is connected with the input of the third convolution layer, and the output of the third convolution layer is connected with one input of a multi-scale prediction layer. The other output of the third residual convolution layer is connected to the input of a fourth convolution layer, the output of which is connected to the input of a second upsampling layer, the output of which is connected to the further input of the second tensor stitching layer. The other output of the first residual convolution layer is connected to the input of a fifth convolution layer, the output of which is connected to the input of a sixth convolution layer, the output of which is connected to the other input of the multi-scale prediction layer. The output of the second residual convolution layer is connected to the input of a seventh convolution layer, the output of which is connected to the input of an eighth convolution layer, the output of which is connected to a further input of the multi-scale prediction layer. The output of the multi-scale prediction layer is used as the output of the food material type identification model.
Each visual coding module of the food material type recognition model consists of a picture embedding layer, 2 normalization function layers, a multi-head attention layer, 2 merging layers and a multi-layer perception mechanism layer. As shown in fig. 4. The input of the picture embedding layer serves as the input of the visual coding module. One output of the picture embedding layer is connected with the input of the first normalization function layer, the output of the first normalization function layer is connected with the input of the multi-head attention layer, and the output of the multi-head attention layer is connected with one input of the first merging layer. The other output of the picture embedding layer is connected to the other input of the first merging layer. One output of the first merging layer is connected with the input of the second normalization function layer, the output of the second normalization function layer is connected with the input of the multi-layer sensing mechanism layer, and the output of the multi-layer sensing mechanism layer is connected with one input of the second merging layer. The other output of the first merging layer is connected to the other input of the second merging layer. The output of the second merge layer serves as the output of the visual coding module.
The picture slice layer YOLOv-ViT segments a fixed 224 x 224 image into fixed size tiles of size 16 x 16, then 196 tiles are generated per image, then the input sequence length is 196, each tile dimension is 16 x 3 = 768, the dimension of the linear projection layer is 768 x 768, so the dimension after the input passes through the linear projection layer is still 196 x 768, there are a total of 196 tokens, the dimension of each token is 768, and one special character cls is added, so that the final dimension is 197 x 768. The position embedding layer converts a visual problem into a seq2seq problem by patch embedding, and adds position coding with the same dimension of vector 768 as the dimension of the input sequence embedding, and after adding position coding, the dimension remains 197×768. The multi-head attention mechanism (multi-head attention) of the visual coding module maps the input to Q, K, V vectors with 12 heads (768/12=64), the dimension of Q, K, V vectors is 197×64, there are 12 groups Q, K, V of vectors, finally the outputs of 12 groups Q, K, V of vectors are spliced together, the output dimension is 197×768, the dimension is enlarged and reduced back through the multi-layer perceptron, the 197×768 is enlarged to 197×3072, and the dimension is reduced to 197×768. After one iteration, the dimensions are still the same as the input, and are 197×768, so that the input can be stacked multiple times, and finally the output Z L 0 corresponding to the special character cls is taken as the final output of the visual coding module. The first visual coding module receives the input vector, the output vector is used as the input of the second visual coding module because the dimension is unchanged, the vectors are sequentially stacked for 6 times, the 6-layer visual coding module enables the correlation to be stronger, the recognition accuracy is more accurate, the tensor splicing layer is connected with the feature fusion network, finally 3 feature graphs with different scales are output, different scale targets are detected by adopting multiple scales, 3 feature graphs are output after the three feature graphs are 255, the side lengths are 13:26:52. the up-sampling mode is used for realizing the multi-scale feature map, two tensors connected by tensor splicing are of the same scale, the two splices are 26 multiplied by 26 scale splicing and 52 multiplied by 52 scale splicing, and the tensor splicing of tensor splicing layers is ensured to be the same in scale by (2, 2) up-sampling. The 9 reference frames would be bisected by the three output tensors. According to the method, three scales of large, medium and small are respectively taken as own reference frames, 3 boxes are predicted by each scale, the design mode of the reference frames still uses clustering, 9 clustering centers are obtained, the clustering centers are evenly distributed to 3 scales according to the size, 3 prediction frames are output by each output on own grids, and therefore grid fine granularity becomes more accurate.
Conventional YOLOv employs a full-roll neural network Darknet-53 comprising 53 convolutional layers, each followed by a batch normalization (batch normalization) layer and an activation function (leak ReLU) layer, without a pooling layer, and uses a convolutional layer with a stride of 2 instead of the pooling layer to perform the feature map downsampling process, which effectively prevents the loss of low-level features due to the pooling layer. Conventional YOLOv uses Darknet-53 to extract features, then substitutes the extracted features of the picture into a feature fusion network to perform feature fusion, and in order to identify more objects, especially small objects, YOLOv3 predicts using three different scales (not only 13×13), and the steps of the three different scales are 32, 16 and 8, respectively. The feature extraction neural network Darknet-53 is changed and replaced by the neural network ViT (Vision Transformer) to obtain YOLOv-ViT, and compared with the most advanced convolution network, the vision converter (ViT) can obtain excellent effect, and the calculation resources required by training are greatly reduced. Because the food categories on the market are many, the required data set needs to be established very much, viT is compared with the traditional visual recognition neural network CNN, when the data set is sufficiently large, the comparison of induced bias and Transformer loses the advantages, so that the traditional YOLOv3 is changed, the created data set is processed by utilizing the advantages of simplicity of a ViT model and high operation speed, the improved neural network is utilized to optimize the identification model, the YOLOv-based improved neural network is compared with the identification model obtained by the traditional YOLOv3 neural network, and the identification model obtained by the novel neural network is higher in identification accuracy, higher in detection speed, strong in generalization capability and higher in stability.
And 4, performing deep learning training on the food material type recognition model in the step 2 by using the food material sample image dataset in the step 2 to obtain a trained food material type recognition model, and disposing the trained food material type recognition model on a core controller of the intelligent refrigerator after model quantization.
The ncnn model generated through deep learning algorithm training cannot be directly used by the core controller at this time, a MaixHub model conversion tool is required to be used for quantizing the model, the model is converted into a awnn model which can be directly used, and the model after light weight is deployed on the core controller.
And 5, each visual sensing module 2 of the intelligent refrigerator performs image acquisition and image preprocessing on food materials stored in the heat preservation box body 1, and then sends the food materials into a core controller, and the core controller utilizes the deployed food material type recognition model to recognize the food materials in the heat preservation box body 1 of the intelligent refrigerator to obtain the types of the food materials and count the number of various food materials in the heat preservation box body 1 of the intelligent refrigerator.
The visual sensing module 2 is used for shooting objects needing to be classified and identified, the size of a picture is converted into 224 x 224, the identified image information is converted into BASE64 information, then the visual sensing module 2BASE64 information is sent to the core controller based on the MQTT protocol, the core controller utilizes the deployed food material type identification model to conduct image identification, and the real-time change condition of food materials in the refrigerator and the information of the internal food materials can be obtained, for example: kind, number, etc. And stores the identification information in the core controller and transmits the identification information to a user terminal (e.g., a mobile APP) through the communication module.
Step 6, firstly, irradiating given food material samples with different types and different freshness by utilizing a spectrometer to obtain spectral data of the food material samples; performing data dimension reduction on the spectrum data by using a principal component analysis algorithm, and extracting characteristic wavelengths by using a random forest algorithm; and then, utilizing the extracted characteristic wavelength to establish a spectrum index reflecting the characteristic of the absorption peak of the freshness factor of the food material.
Principal Component Analysis (PCA) is the most commonly used, most efficient algorithm to solve spectral data collineation, extract data characteristic information, and implement data variable compression. Principal component analysis the principal components extracted are linear combinations of the original variables, which are orthogonal. The information contained in the principal components does not overlap, which can eliminate multiple collinearity between variables.
The Random Forest (RF) algorithm is a variable optimization algorithm that determines the number of characteristic wavelengths by setting an appropriate selection threshold and calculates the selection probability of each variable by iterative modeling using a small number of variables.
And establishing a spectral index reflecting the characteristics of the absorption peak of the freshness factor of the food material by utilizing the extracted characteristic wavelength, wherein the spectral index comprises an absorption peak depth index, a relative depth index of the absorption peak, a normalized intensity difference index and an absorbance difference index.
The absorption peak depth index a is:
A＝R3+R1-2R2
The relative depth index B of the absorption peak is:
B＝R2/(R3+R1)
The normalized intensity difference index C is:
C＝(R3-R2)/(R3+R2)
The absorbance difference index D is:
D＝log10(R3/R2)
wherein R 1 is a characteristic wavelength of a start point of an absorption peak, R 2 is a characteristic wavelength of a lowest point of the absorption peak, and R 3 is a characteristic wavelength of an end point of the absorption peak.
And 7, manually marking the spectral indexes of the food material samples obtained in the step 6 for the freshness of the food material, and creating a spectral index data set of the food material samples.
The spectral index of the food material sample is used for reflecting the freshness of the food material, and the freshness degree of the food material can be distinguished through the set corresponding relation criterion of the spectral index and the freshness, so that the specific freshness degree of the food material is determined, and the spectral index of the food material is added with a corresponding freshness or non-freshness label.
Acquiring a spectrum index of the food material, establishing a data set of the spectrum index, marking the spectrum index in the data set to further distinguish the freshness level of the food material, and according to 7:2:1 is divided into a training set, a testing set and a verification set.
And 8, constructing a food material freshness identification model based on an LDA (Linear discriminant analysis) neural network.
LDA is a supervised learning dimension reduction technique, each sample of its dataset being class-output. Basic idea of LDA: given a training sample set, it is sought to project samples onto a straight line such that the projection points of like samples are as close as possible and the projection point centers of heterogeneous samples are as far apart as possible. The specific flow of the LDA is as follows: input: data set d= { (x 1, y 1), (x 2, y 2), (xm, ym) }, arbitrary sample xi is an n-dimensional vector, yi e { C1, C2,., ck }, for k categories in total. Reducing the dimension of the product; and (3) outputting: and D' is a data set after dimension reduction. (1) calculating an inter-class divergence matrix SB; calculating an intra-class divergence matrix SW; (3) Calculating SB and SW through a formula SW-1SBw =lambdaw to obtain a eigenvalue lambda and an eigenvector w, and multiplying the eigenvalue vector lambda 'with the eigenvector w at the first several maximum eigenvalues to obtain a dimensionality reduction conversion matrix lambda' w; (4) Multiplying the original data with a conversion matrix to obtain reduced-dimension data (lambda' w) Tx; classifying it.
And 9, performing deep learning training on the food material freshness identification model in the step 8 by using the food material sample spectral index data set in the step 7 to obtain a trained food material freshness identification model, and disposing the trained food material freshness identification model on a core controller of the intelligent refrigerator after performing model quantization on the trained food material freshness identification model.
Substituting the spectral index data set into the LDA neural network for model training, establishing a food freshness identification model, and comparing the spectral indexes of the food through the model, so that the freshness degree of the food can be judged through the spectral indexes, and the freshness level of the food is obtained.
And 10, each spectrometer module 3 of the intelligent refrigerator collects spectral data of food materials stored in the heat preservation box body 1, and processes the collected spectral data by adopting a method similar to the step 6, namely, performing data dimension reduction on the spectral data by using a principal component analysis algorithm, extracting characteristic wavelengths by using a random forest algorithm, and obtaining a spectral index by using the extracted characteristic wavelengths. And then, the spectrum index is sent to a core controller, the core controller utilizes the deployed food freshness identification model to identify and determine the freshness of the food in the heat preservation box body 1 of the intelligent refrigerator, and when the food in the heat preservation box body 1 of the refrigerator is determined to be stale, an alarm is sent to a user.
And 11, transmitting the type and the quantity of the food materials obtained in the step 5 and the freshness of the food materials obtained in the step 10 to a user terminal by utilizing a communication module.
The intelligent refrigerator performs image recognition and spectrum detection on food stored in the intelligent refrigerator to obtain food information data (type, quantity and freshness), the food information data are transmitted to a cloud server, and the EMQX Cloud cloud server stores identification information and transmits the identification information to a user terminal. The user terminal is internally provided with an app matched with the function of the app, and the app is used for receiving food information data transmitted by the intelligent refrigerator through the server, obtaining food type information according to an image recognition technology, intelligently recommending a menu through the app, and realizing spectral detection of food freshness information in real time to perform an early warning function.
It should be noted that, although the examples described above are illustrative, this is not a limitation of the present invention, and thus the present invention is not limited to the above-described specific embodiments. Other embodiments, which are apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein, are considered to be within the scope of the invention as claimed.
Claims (5)
1. The intelligent refrigerator comprises an insulation box body (1), a refrigerating system, an electrical appliance system and a core controller; the refrigerating system, the electric appliance system and the core controller are all positioned in the heat insulation box body (1), and the refrigerating system is electrically connected with the electric appliance system and the core controller; characterized by further comprising at least one vision sensing module (2) and at least one spectrometer module (3); the vision sensing modules (2) and the spectrometer modules (3) are distributed and arranged in the heat preservation box body (1) and face to food materials stored in the heat preservation box body (1); the output ends of the vision sensing module (2) and the spectrometer module (3) are electrically connected with the core controller;
the method is characterized by comprising the following steps:
step 1, shooting given food samples with different types and different freshness under multi-angle and multi-background environments, and performing image preprocessing on the shot images to obtain food sample images;
step 2, manually marking the food material types of the food material sample image obtained in the step 1, and creating a food material sample image data set;
step 3, constructing a food material type identification model based on YOLOv-ViT neural network;
the food material type identification model consists of an image slice layer, a position embedding layer, 6 visual coding modules, 2 tensor splicing layers, 3 residual error convolution layers, 2 upsampling layers, 8 convolution layers and a multi-scale prediction layer;
The input of the image slice layer is used as the input of the food material type identification model, the output of the image slice layer is connected with the input of the position embedding layer, the output of the position embedding layer is connected with the input of the first visual coding module, the output of the first visual coding module is connected with the input of the second visual coding module, the output of the second visual coding module is connected with the input of the third visual coding module, the output of the third visual coding module is connected with the input of the fourth visual coding module, one output of the fourth visual coding module is connected with the input of the fifth visual coding module, and one output of the fifth visual coding module is connected with the input of the sixth visual coding module; the output of the sixth visual coding module is connected with the input of the first residual convolution layer, the other output of the fifth visual coding module is connected with one input of the first tensor splicing layer, and the other output of the fourth visual coding module is connected with one input of the second tensor splicing layer;
One output of the first residual error convolution layer is connected with the input of the first convolution layer, the output of the first convolution layer is connected with the input of the first upsampling layer, the input of the first upsampling layer is connected with the other input of the first tensor splicing layer, one output of the first tensor splicing layer is connected with the other input of the second tensor splicing layer, and the output of the second tensor splicing layer is connected with the input of the second residual error convolution layer;
The other output of the first tensor splicing layer is connected with the input of a third residual convolution layer, one output of the third residual convolution layer is connected with the input of a second convolution layer, the output of the second convolution layer is connected with the input of the third convolution layer, and the output of the third convolution layer is connected with one input of a multi-scale prediction layer; the other output of the third residual convolution layer is connected with the input of a fourth convolution layer, the output of the fourth convolution layer is connected with the input of a second upsampling layer, and the output of the second upsampling layer is connected with the other input of the second tensor splicing layer; the other output of the first residual convolution layer is connected with the input of a fifth convolution layer, the output of the fifth convolution layer is connected with the input of a sixth convolution layer, and the output of the sixth convolution layer is connected with the other input of the multi-scale prediction layer; the output of the second residual convolution layer is connected with the input of a seventh convolution layer, the output of the seventh convolution layer is connected with the input of an eighth convolution layer, and the output of the eighth convolution layer is connected with the other input of the multi-scale prediction layer; the output of the multi-scale prediction layer is used as the output of the food material type identification model;
Step 4, performing deep learning training on the food material type recognition model in the step 3 by using the food material sample image dataset in the step 2 to obtain a trained food material type recognition model, and disposing the trained food material type recognition model on a core controller of the intelligent refrigerator after model quantization;
Step 5, each visual sensing module (2) of the intelligent refrigerator performs image acquisition and image preprocessing on food materials stored in the heat preservation box body (1) and then sends the food materials into a core controller, and the core controller utilizes the deployed food material type recognition model to recognize the food materials in the heat preservation box body (1) of the intelligent refrigerator to obtain the types of the food materials and count the number of various food materials in the heat preservation box body (1) of the intelligent refrigerator;
Step 6, firstly, irradiating given food material samples with different types and different freshness by utilizing a spectrometer to obtain spectral data of the food material samples; performing data dimension reduction on the spectrum data by using a principal component analysis algorithm, and extracting characteristic wavelengths by using a random forest algorithm; then, utilizing the extracted characteristic wavelength to establish a spectrum index reflecting the characteristic of an absorption peak of the freshness factor of the food material;
Step 7, manually marking the food freshness of the spectral indexes of the food samples obtained in the step 6, and creating a spectral index data set of the food samples;
Step 8, constructing a food freshness identification model based on an LDA neural network;
Step 9, performing deep learning training on the food material freshness identification model in step 8 by using the food material sample spectral index data set in step 7 to obtain a trained food material freshness identification model, and disposing the trained food material freshness identification model on a core controller of the intelligent refrigerator after model quantization;
Step 10, each spectrometer module (3) of the intelligent refrigerator collects spectrum data of food materials stored in the heat preservation box body (1), the main component analysis algorithm is utilized to conduct data dimension reduction on the spectrum data, the random forest algorithm is utilized to conduct characteristic wavelength extraction, the extracted characteristic wavelength is utilized to obtain a spectrum index and then the spectrum index is sent to the core controller, and the core controller utilizes the deployed food material freshness identification model to identify the food materials in the heat preservation box body (1) of the intelligent refrigerator to determine the freshness of the food materials.
2. The method according to claim 1, wherein in step 3, each visual coding module of the food material type recognition model is composed of a picture embedding layer, 2 normalization function layers, a multi-head attention layer, 2 merging layers and a multi-layer perception mechanism layer;
The input of the picture embedding layer is used as the input of the visual coding module; one output of the picture embedding layer is connected with the input of the first normalization function layer, the output of the first normalization function layer is connected with the input of the multi-head attention layer, and the output of the multi-head attention layer is connected with one input of the first merging layer; the other output of the picture embedding layer is connected with the other input of the first merging layer;
One output of the first merging layer is connected with the input of the second normalization function layer, the output of the second normalization function layer is connected with the input of the multi-layer sensing mechanism layer, and the output of the multi-layer sensing mechanism layer is connected with one input of the second merging layer; the other output of the first merging layer is connected with the other input of the second merging layer; the output of the second merge layer serves as the output of the visual coding module.
3. The method according to claim 1, wherein in step 6, the spectral indexes reflecting the characteristics of the absorption peaks of the freshness factor of the food material include an absorption peak depth index, a relative depth index of the absorption peaks, a normalized intensity difference index and an absorbance difference index; wherein the method comprises the steps of
The absorption peak depth index a is:
A＝R3+R1-2R2
The relative depth index B of the absorption peak is:
B＝R2/(R3+R1)
The normalized intensity difference index C is:
C＝(R3-R2)/(R3+R2)
The absorbance difference index D is:
D＝log10(R3/R2)
wherein R 1 is a characteristic wavelength of a start point of an absorption peak, R 2 is a characteristic wavelength of a lowest point of the absorption peak, and R 3 is a characteristic wavelength of an end point of the absorption peak.
4. The method of claim 1, wherein step 9 further comprises alerting the user when the food in the insulated cabinet (1) of the refrigerator is determined to be stale.
5. The method for operating a smart refrigerator based on machine vision and spectral detection of claim 1, further comprising:
And 11, transmitting the type and the quantity of the food materials obtained in the step 5 and the freshness of the food materials obtained in the step 10 to a user terminal by utilizing a communication module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210768651.XA CN115046363B (en) | 2022-06-30 | 2022-06-30 | Intelligent refrigerator based on machine vision and spectrum detection and operation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210768651.XA CN115046363B (en) | 2022-06-30 | 2022-06-30 | Intelligent refrigerator based on machine vision and spectrum detection and operation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115046363A CN115046363A (en) | 2022-09-13 |
| CN115046363B true CN115046363B (en) | 2024-05-07 |
Family
ID=83165028
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210768651.XA Active CN115046363B (en) | 2022-06-30 | 2022-06-30 | Intelligent refrigerator based on machine vision and spectrum detection and operation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115046363B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105181912A (en) * | 2015-06-30 | 2015-12-23 | 江苏大学 | Method for detection of freshness during rice storage |
| CN207164027U (en) * | 2017-06-07 | 2018-03-30 | 浙江凯尔奇电器有限公司 | Food materials freshness intelligent detecting system in a kind of refrigerator-freezer |
| EP3527918A2 (en) * | 2018-02-14 | 2019-08-21 | Whirlpool Corporation | Foodstuff sensing appliance |
| CN112740229A (en) * | 2018-09-21 | 2021-04-30 | 三星电子株式会社 | Method and system for providing information related to state of object in refrigerator |
| CN113792578A (en) * | 2021-07-30 | 2021-12-14 | 北京智芯微电子科技有限公司 | Method, device and system for detecting abnormity of transformer substation |
| CN113807361A (en) * | 2021-08-11 | 2021-12-17 | 华为技术有限公司 | Neural network, target detection method, neural network training method and related products |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20220083349A (en) * | 2020-12-11 | 2022-06-20 | 삼성전자주식회사 | Food monitoring apparatus, refrigerator including the same, and operating method thereof |
-
2022
- 2022-06-30 CN CN202210768651.XA patent/CN115046363B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105181912A (en) * | 2015-06-30 | 2015-12-23 | 江苏大学 | Method for detection of freshness during rice storage |
| CN207164027U (en) * | 2017-06-07 | 2018-03-30 | 浙江凯尔奇电器有限公司 | Food materials freshness intelligent detecting system in a kind of refrigerator-freezer |
| EP3527918A2 (en) * | 2018-02-14 | 2019-08-21 | Whirlpool Corporation | Foodstuff sensing appliance |
| CN112740229A (en) * | 2018-09-21 | 2021-04-30 | 三星电子株式会社 | Method and system for providing information related to state of object in refrigerator |
| CN113792578A (en) * | 2021-07-30 | 2021-12-14 | 北京智芯微电子科技有限公司 | Method, device and system for detecting abnormity of transformer substation |
| CN113807361A (en) * | 2021-08-11 | 2021-12-17 | 华为技术有限公司 | Neural network, target detection method, neural network training method and related products |
Non-Patent Citations (2)
| Title |
|---|
| 基于Transformer改进YOLO v4的火灾检测方法;王国睿;智能计算机与应用;20210731;第11卷(第7期);86-90 * |
| 基于轻量级神经网络的食材识别系统研究与应用;黄颖康;广东工业大学硕士论文;20220315;26-48 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115046363A (en) | 2022-09-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107873101B (en) | Imaging system for object recognition and evaluation | |
| Iqbal et al. | Classification of selected citrus fruits based on color using machine vision system | |
| CN108663331A (en) | Detect the method and refrigerator of food freshness in refrigerator | |
| Zhu et al. | Support vector machine and YOLO for a mobile food grading system | |
| CN110966833B (en) | Method for detecting food material information in refrigerator and refrigerator | |
| US11308348B2 (en) | Methods and systems for processing image data | |
| US11756282B2 (en) | System, method and computer program for guided image capturing of a meal | |
| Asmara et al. | Chicken meat freshness identification using the histogram color feature | |
| Sarkar et al. | Comparative analysis of statistical and supervised learning models for freshness assessment of oyster mushrooms | |
| CN117115571B (en) | Fine-grained intelligent commodity identification method, device, equipment and medium | |
| CN112906780A (en) | Fruit and vegetable image classification system and method | |
| Viet Tran | Efficient image retrieval with statistical color descriptors | |
| TW201822121A (en) | Fruit maturity and quality recognition system and method which include an image pick-up module, fruit feature databases, a data processing module, and a data output module | |
| Zheng et al. | Identifying strawberry appearance quality by vision transformers and support vector machine | |
| CN115046363B (en) | Intelligent refrigerator based on machine vision and spectrum detection and operation method | |
| Xiao et al. | Fast recognition method for citrus under complex environments based on improved YOLOv3 | |
| US20230143130A1 (en) | System and method for identifying fruit shelf life | |
| JP2001092975A (en) | System and method for recognizing article | |
| CN111611921B (en) | Solar panel identification system based on remote sensing big data | |
| Nayak et al. | Fruit recognition using image processing | |
| KR102433779B1 (en) | An analysis system for predictive power usage by learning operation date and method thereof | |
| Sehgal et al. | Auto-annotation of tomato images based on ripeness and firmness classification for multimodal retrieval | |
| Robles‐Kelly et al. | Imaging spectroscopy for scene analysis: challenges and opportunities | |
| Musab et al. | Convolution Neural Network-Based Meat Quality Checker Using Different Optimizers | |
| Sujatha et al. | Implementing deep‐learning techniques for accurate fruit disease identification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |