CN115046363A - Intelligent refrigerator based on machine vision and spectrum detection and operation method - Google Patents
Intelligent refrigerator based on machine vision and spectrum detection and operation method Download PDFInfo
- Publication number
- CN115046363A CN115046363A CN202210768651.XA CN202210768651A CN115046363A CN 115046363 A CN115046363 A CN 115046363A CN 202210768651 A CN202210768651 A CN 202210768651A CN 115046363 A CN115046363 A CN 115046363A
- Authority
- CN
- China
- Prior art keywords
- layer
- input
- output
- food material
- food
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 238000001514 detection method Methods 0.000 title claims abstract description 25
- 238000001228 spectrum Methods 0.000 title claims abstract description 25
- 239000000463 material Substances 0.000 claims abstract description 163
- 230000000007 visual effect Effects 0.000 claims abstract description 74
- 238000013135 deep learning Methods 0.000 claims abstract description 9
- 239000010410 layer Substances 0.000 claims description 237
- 230000003595 spectral effect Effects 0.000 claims description 42
- 238000010521 absorption reaction Methods 0.000 claims description 27
- 238000004321 preservation Methods 0.000 claims description 26
- 238000013528 artificial neural network Methods 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 18
- 238000004422 calculation algorithm Methods 0.000 claims description 16
- 238000010606 normalization Methods 0.000 claims description 16
- 238000012549 training Methods 0.000 claims description 12
- 238000004891 communication Methods 0.000 claims description 10
- 238000009413 insulation Methods 0.000 claims description 10
- 230000007246 mechanism Effects 0.000 claims description 10
- 238000000513 principal component analysis Methods 0.000 claims description 9
- 230000009467 reduction Effects 0.000 claims description 9
- 238000005057 refrigeration Methods 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 9
- 238000002372 labelling Methods 0.000 claims description 8
- 238000007637 random forest analysis Methods 0.000 claims description 8
- 238000013139 quantization Methods 0.000 claims description 7
- 238000002835 absorbance Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 230000001678 irradiating effect Effects 0.000 claims description 3
- 230000008447 perception Effects 0.000 claims description 3
- 239000011229 interlayer Substances 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000012544 monitoring process Methods 0.000 abstract description 2
- 239000013598 vector Substances 0.000 description 9
- 238000006243 chemical reaction Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000011176 pooling Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
Images
Classifications
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D29/00—Arrangement or mounting of control or safety devices
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D11/00—Self-contained movable devices, e.g. domestic refrigerators
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D19/00—Arrangement or mounting of refrigeration units with respect to devices or objects to be refrigerated, e.g. infrared detectors
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D2400/00—General features of, or devices for refrigerators, cold rooms, ice-boxes, or for cooling or freezing apparatus not covered by any other subclass
- F25D2400/36—Visual displays
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D2700/00—Means for sensing or measuring; Sensors therefor
- F25D2700/06—Sensors detecting the presence of a product
Landscapes
- Engineering & Computer Science (AREA)
- Chemical & Material Sciences (AREA)
- Combustion & Propulsion (AREA)
- Physics & Mathematics (AREA)
- Mechanical Engineering (AREA)
- Thermal Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
The invention discloses an intelligent refrigerator based on machine vision and spectrum detection and an operation method thereof, wherein the intelligent refrigerator comprises at least one vision sensing module and at least one spectrometer module; the vision sensing modules and the spectrometer modules are distributed and installed in the heat-insulating box body and face to food materials stored in the heat-insulating box body; the output ends of the visual sensing module and the spectrometer module are electrically connected with the core controller. The method has the advantages that the discrimination of the types and freshness of the food materials stored in the refrigerator is realized by utilizing an image recognition method combining machine vision and deep learning and a spectrum nondestructive detection food material freshness degree technology, so that the requirements of people on food material management and food material freshness monitoring in daily life are met, and a novel application example is provided for the subsequent intelligent refrigerator market.
Description
Technical Field
The invention relates to the technical field of intelligent refrigerators, in particular to an intelligent refrigerator based on machine vision and spectrum detection and an operation method.
Background
The intelligent refrigerator is a refrigerator type capable of intelligently managing food and intelligently controlling the refrigerator. In the aspect of food material identification, in general, a visual sensing module arranged in an intelligent refrigerator is used for photographing food materials to be placed in the refrigerator, then the intelligent refrigerator performs image processing on the photographed food material picture, and after a color characteristic value and a shape characteristic value of the photographed food materials are extracted, the color characteristic value and the shape characteristic value are matched with a local user food information base to determine food material types. In addition, in the aspect of food material freshness, a time threshold value is generally set after food materials are put into the refrigerator, and a user is reminded when the storage time reaches the set threshold value. However, due to the influence of various complex factors in the transportation, purchase and storage processes, the freshness of the food materials cannot be accurately reflected by time, the storage time of the food materials cannot be accurately predicted by fixed time reminding, and the reminding requirement of food fresh keeping cannot be met.
Disclosure of Invention
The invention aims to solve the problems of food material identification and preservation of the existing intelligent refrigerator, and provides an intelligent refrigerator based on machine vision and spectrum detection and an operation method.
In order to solve the problems, the invention is realized by the following technical scheme:
the intelligent refrigerator based on machine vision and spectrum detection comprises a heat preservation box body, a refrigeration system, an electric appliance system and a core controller; the refrigeration system, the electrical system and the core controller are all positioned in the heat insulation box body, and the refrigeration system is electrically connected with the electrical system and the core controller; the device is characterized by further comprising at least one visual sensing module and at least one spectrometer module; the vision sensing modules and the spectrometer modules are distributed and installed in the heat-insulating box body and face to food materials stored in the heat-insulating box body; the output ends of the visual sensing module and the spectrometer module are electrically connected with the core controller.
Each interlayer of the heat preservation box body of the intelligent refrigerator is provided with a vision sensing module and a spectrometer module.
The intelligent refrigerator further comprises a communication module, and the communication module is arranged in the heat preservation box body and is electrically connected with the core controller.
The operation method of the intelligent refrigerator based on the machine vision and the spectrum detection comprises the following steps:
step 4, performing deep learning training on the food material type recognition model in the step 2 by using the food material sample image data set in the step 2 to obtain a trained food material type recognition model, performing model quantization on the trained food material type recognition model, and deploying the trained food material type recognition model to a core controller of an intelligent refrigerator;
step 5, each visual sensing module of the intelligent refrigerator carries out image acquisition and image preprocessing on food materials stored in the heat insulation box body and then sends the food materials into a core controller, the core controller utilizes a deployed food material type identification model to identify the food materials in the heat insulation box body of the intelligent refrigerator to obtain the type of the food materials, and the quantity of the food materials in the heat insulation box body of the intelligent refrigerator is counted;
step 6, irradiating given food material samples of different types and different freshness by using a spectrometer to obtain spectral data of the food material samples; performing data dimensionality reduction on the spectral data by using a principal component analysis algorithm, and extracting characteristic wavelengths by using a random forest algorithm; then establishing a spectral index reflecting the absorption peak characteristics of the food freshness factor by using the extracted characteristic wavelength;
step 7, artificially labeling the food freshness of the food material sample spectral index obtained in the step 6, and creating a food material sample spectral index data set;
step 8, constructing a food material freshness identification model based on an LDA neural network;
step 9, performing deep learning training on the food material freshness identification model in the step 8 by using the food material sample spectral index data set in the step 7 to obtain a trained food material freshness identification model, performing model quantization on the trained food material freshness identification model, and deploying the trained food material freshness identification model to a core controller of an intelligent refrigerator;
step 10, spectrum data of food materials stored in the heat preservation box are collected through each spectrometer module of the intelligent refrigerator, data dimension reduction is conducted on the spectrum data through a principal component analysis algorithm, characteristic wavelengths are extracted through a random forest algorithm, spectrum indexes are obtained through the extracted characteristic wavelengths and then are sent to a core controller, and the core controller identifies the food materials in the heat preservation box of the intelligent refrigerator through a deployed food material freshness identification model to determine food material freshness.
The food material type identification model constructed in the step 3 consists of an image slice layer, a position embedding layer, 6 visual coding modules, 2 tensor splicing layers, 3 residual convolution layers, 2 upsampling layers, 8 convolution layers and a multi-scale prediction layer; the input of the image slice layer is used as the input of the food material type identification model, the output of the image slice layer is connected with the input of the position embedding layer, the output of the position embedding layer is connected with the input of the first visual coding module, the output of the first visual coding module is connected with the input of the second visual coding module, the output of the second visual coding module is connected with the input of the third visual coding module, the output of the third visual coding module is connected with the input of the fourth visual coding module, one output of the fourth visual coding module is connected with the input of the fifth visual coding module, and one output of the fifth visual coding module is connected with the input of the sixth visual coding module; the output of the sixth visual coding module is connected with the input of the first residual convolution layer, the other output of the fifth visual coding module is connected with one input of the first tensor splicing layer, and the other output of the fourth visual coding module is connected with one input of the second tensor splicing layer; one output of the first residual convolution layer is connected with the input of the first convolution layer, the output of the first convolution layer is connected with the input of the first up-sampling layer, the input of the first up-sampling layer is connected with the other input of the first tensor splicing layer, one output of the first tensor splicing layer is connected with the other input of the second tensor splicing layer, and the output of the second tensor splicing layer is connected with the input of the second residual convolution layer; the other output of the first scalar splicing layer is connected with the input of a third residual convolutional layer, one output of the third residual convolutional layer is connected with the input of a second convolutional layer, the output of the second convolutional layer is connected with the input of the third convolutional layer, and the output of the third convolutional layer is connected with one input of the multi-scale prediction layer; the other output of the third residual convolutional layer is connected with the input of a fourth convolutional layer, the output of the fourth convolutional layer is connected with the input of a second upsampling layer, and the output of the second upsampling layer is connected with the other input of the second tensor splicing layer; the other output of the first residual convolutional layer is connected with the input of a fifth convolutional layer, the output of the fifth convolutional layer is connected with the input of a sixth convolutional layer, and the output of the sixth convolutional layer is connected with the other input of the multi-scale prediction layer; the output of the second residual convolutional layer is connected with the input of the seventh convolutional layer, the output of the seventh convolutional layer is connected with the input of the eighth convolutional layer, and the output of the eighth convolutional layer is connected with the other input of the multi-scale prediction layer; the output of the multi-scale prediction layer is used as the output of the food material type identification model;
in the step 3, each visual coding module of the food material type identification model consists of a picture embedding layer, 2 normalization function layers, a multi-head attention layer, 2 merging layers and a multilayer perception mechanism layer; the input of the picture embedding layer is used as the input of the visual coding module; one output of the picture embedding layer is connected with the input of the first normalization function layer, the output of the first normalization function layer is connected with the input of the multi-head attention layer, and the output of the multi-head attention layer is connected with one input of the first merging layer; the other output of the picture embedding layer is connected with the other input of the first merging layer; one output of the first merging layer is connected with the input of the second normalization function layer, the output of the second normalization function layer is connected with the input of the multilayer sensing mechanism layer, and the output of the multilayer sensing mechanism layer is connected with one input of the second merging layer; the other output of the first merging layer is connected with the other input of the second merging layer; the output of the second merging layer is used as the output of the visual coding module.
In the step 6, the spectral index reflecting the characteristic of the absorption peak of the food freshness factor comprises an absorption peak depth index, a relative depth index of the absorption peak, a normalized intensity difference index and an absorbance difference index; wherein:
the absorption peak depth index a is:
A＝R 3 +R 1 -2R 2
the relative depth index B of the absorption peaks is:
B＝R 2 /(R 3 +R 1 )
the normalized intensity difference index C is:
C＝(R 3 -R 2 )/(R 3 +R 2 )
the absorbance difference index D is:
D＝log 10 (R 3 /R 2 )
in the formula, R 1 Characteristic wavelength, R, of the origin of the absorption peak 2 Characteristic wavelength of the lowest point of the absorption peak, R 3 Characteristic wavelength of the end point of the absorption peak.
The step 9 may further include giving an alarm to the user when the food material in the heat-insulating box of the refrigerator is determined to be not fresh.
The operation method of the intelligent refrigerator based on machine vision and spectrum detection further comprises the following steps:
and 11, transmitting the types and the quantity of the food materials obtained in the step 5 and the freshness of the food materials obtained in the step 10 to a user terminal by using a communication module.
Compared with the prior art, the method disclosed by the invention realizes the discrimination of the type and the freshness of the food materials stored in the refrigerator by utilizing an image recognition method combining machine vision and deep learning and a spectrum nondestructive detection technology of the freshness of the food materials, so that the requirements of people on food material management and food material freshness monitoring in daily life are met, and a novel application example is provided for the subsequent intelligent refrigerator market.
Drawings
Fig. 1 is a schematic structural diagram of an intelligent refrigerator based on machine vision and spectrum detection.
Fig. 2 is a flow chart of an operation method of the intelligent refrigerator based on machine vision and spectrum detection.
FIG. 3 is a schematic structural diagram of a food material type identification model based on a YOLOv3-ViT neural network.
Fig. 4 is a schematic structural diagram of the visual coding module in fig. 3.
The following are marked in the figure: 1. a heat preservation box body; 2. a visual sense module; 3. a spectrometer module.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to specific examples.
An intelligent refrigerator based on machine vision and spectrum detection is shown in figure 1 and comprises a heat preservation box body 1, a refrigeration system, an electric appliance system and a core controller. The refrigerating system comprises a compressor, a condenser, an evaporator, a capillary tube, a connecting pipeline and the like. The electric system comprises a temperature controller, an electromagnetic valve, a lighting lamp, a display screen and the like. The heat preservation box body 1 comprises a box body and a door body, wherein the door body is hinged on the box body, and the door body form the closed heat preservation box body 1 for storing food materials. The refrigeration system, the electrical system and the core controller are all located inside the heat preservation box body 1, and the refrigeration system and the electrical system are electrically connected with the core controller. In order to realize machine vision and spectrum detection, at least one vision sensing module 2 and at least one spectrometer module 3 are distributed and installed in the heat preservation box body 1. In the preferred embodiment of the present invention, the vision sensing module 2 is a camera module, the spectrometer module 3 is a micro spectrometer module 3, each storage compartment of the thermal insulation box 1 is provided with one vision sensing module 2 and one spectrometer module 3, and the vision sensing module 2 and the spectrometer module 3 face the food material stored in the thermal insulation box 1. All vision sensing modules 2 and all spectrometer modules 3 are electrically connected to a core controller. In order to realize networking, a communication module is arranged in the heat preservation box body 1. The communication module is electrically connected with the core controller.
As shown in fig. 2, the operation method of the intelligent refrigerator based on machine vision and spectrum detection specifically includes the following steps:
In the present embodiment, a camera, a mobile phone or a vision sensing module 2 is used to capture multiple angles and multiple background environments of the food material sample for classification and identification, and the captured images are preprocessed into RGB color images with a size of 224 × 224. The food material sample image comprises 4000 common food materials on the market.
And 2, manually labeling the food material types of the food material sample images obtained in the step 1, and creating a food material sample image data set.
Labeling 4000 common food materials on the market in terms of categories by using labellmg, and creating a food material sample image dataset. And (3) proportionally comparing the created food material sample image dataset with the image dataset of 7: 2: 1 is divided into a training set, a test set and a verification set.
And 3, constructing a food material type identification model based on a YOLOv3-ViT neural network.
The food material type identification model is a YOLOv3-ViT neural network obtained by improving a traditional YOLOv3 neural network, and consists of an image slice layer, a position embedding layer, 6 visual coding modules, 2 tensor splicing layers, 3 residual error convolutional layers, 2 upsampling layers, 8 convolutional layers and a multi-scale prediction layer. As shown in fig. 3. The input of the image slice layer is used as the input of the food material type identification model, the output of the image slice layer is connected with the input of the position embedding layer, the output of the position embedding layer is connected with the input of the first visual coding module, the output of the first visual coding module is connected with the input of the second visual coding module, the output of the second visual coding module is connected with the input of the third visual coding module, the output of the third visual coding module is connected with the input of the fourth visual coding module, one output of the fourth visual coding module is connected with the input of the fifth visual coding module, and one output of the fifth visual coding module is connected with the input of the sixth visual coding module. The output of the sixth visual coding module is connected with the input of the first residual convolution layer, the other output of the fifth visual coding module is connected with one input of the first tensor splicing layer, and the other output of the fourth visual coding module is connected with one input of the second tensor splicing layer. One output of the first residual convolution layer is connected with the input of the first convolution layer, the output of the first convolution layer is connected with the input of the first up-sampling layer, the input of the first up-sampling layer is connected with the other input of the first tensor splicing layer, one output of the first tensor splicing layer is connected with the other input of the second tensor splicing layer, and the output of the second tensor splicing layer is connected with the input of the second residual convolution layer. The other output of the first scalar concatenation layer is connected with the input of the third residual convolutional layer, one output of the third residual convolutional layer is connected with the input of the second convolutional layer, the output of the second convolutional layer is connected with the input of the third convolutional layer, and the output of the third convolutional layer is connected with one input of the multi-scale prediction layer. Another output of the third residual convolutional layer is connected to an input of a fourth convolutional layer, an output of the fourth convolutional layer is connected to an input of a second upsampling layer, and an output of the second upsampling layer is connected to another input of the second tensor stitching layer. Another output of the first residual convolutional layer is connected to an input of a fifth convolutional layer, an output of the fifth convolutional layer is connected to an input of a sixth convolutional layer, and an output of the sixth convolutional layer is connected to another input of the multi-scale prediction layer. An output of the second residual convolutional layer is connected to an input of the seventh convolutional layer, an output of the seventh convolutional layer is connected to an input of the eighth convolutional layer, and an output of the eighth convolutional layer is connected to yet another input of the multi-scale prediction layer. And outputting the multi-scale prediction layer as the output of the food material type identification model.
Each visual coding module of the food material type identification model consists of a picture embedding layer, 2 normalization function layers, a multi-head attention layer, 2 merging layers and a multilayer perception mechanism layer. As shown in fig. 4. The input of the picture embedding layer is used as the input of the visual coding module. One output of the picture embedding layer is connected with the input of the first normalization function layer, the output of the first normalization function layer is connected with the input of the multi-head attention layer, and the output of the multi-head attention layer is connected with one input of the first merging layer. Another output of the picture embedding layer is connected to another input of the first merging layer. One output of the first merging layer is connected with the input of the second normalization function layer, the output of the second normalization function layer is connected with the input of the multilayer sensing mechanism layer, and the output of the multilayer sensing mechanism layer is connected with one input of the second merging layer. The other output of the first merging layer is connected to the other input of the second merging layer. The output of the second merging layer is used as the output of the visual coding module.
The picture slice layer of yollov 3-ViT divides the fixed 224 × 224 image into small blocks of fixed size, the size of the small block is 16 × 16, 196 small blocks are generated for each image, the input sequence length is 196, the dimension of each small block is 16 × 16 × 3 to 768, the dimension of the linear projection layer is 768 × 768, so the dimension after the input passes through the linear projection layer is still 196 × 768, namely there are 196 tokens in total, the dimension of each token is 768, and a special character cls is added, so that the final dimension is 197 × 768. The position embedding layer converts a visual problem into a seq2seq problem through patch embedding, and adds position codes, the dimension of the vector is 768 the same as that of the input sequence embedding, and after the position codes are added, the dimension is still 197 x 768. The multi-head attention mechanism (multi-head attention) of the visual coding module firstly maps input to Q, K, V vectors, 12 heads (768/12-64) are provided, the dimension of Q, K, V vectors is 197 × 64, and the visual coding module has 12 groups of Q, K, Q, K and,And V vectors, finally splicing the outputs of 12 groups of Q, K, V vectors, wherein the output dimension is 197 multiplied by 768, the dimension is enlarged and reduced back through a multilayer perceptron, the 197 multiplied by 768 is enlarged to be 197 multiplied by 3072, and the 197 multiplied by 768 is reduced. After one iteration, the dimension is still the same as the input dimension and is 197 x 768, so that the dimension can be stacked for multiple times, and finally the output Z corresponding to the special character cls is output L 0 is the final output of the visual coding module. The first visual coding module accepts the input vector, and the output vector is as the input of the second visual coding module because the dimensionality is unchangeable, pile up 6 times in proper order, the visual coding module of 6 layers makes the correlation become stronger, make the recognition accuracy more accurate, and fuse the network connection with its and feature through the tensor concatenation layer, output 3 different yardstick characteristic maps at last, adopt the multiscale to come to detect different yardstick targets, 3 characteristic maps of back output, the degree of depth is 255, the length of side is 13 respectively: 26: 52. the multi-scale characteristic map is realized by using an up-sampling mode, two tensors connected by tensor splicing are of the same scale, the splicing at two positions are 26 × 26-scale splicing and 52 × 52-scale splicing respectively, and the tensor scale of the tensor splicing layer splicing is ensured to be the same by (2, 2) up-sampling. The 9 reference frames would be bisected by the three output tensors. According to the three scales of large, medium and small, the reference frame is taken, 3 boxes are predicted for each scale, the design mode of the reference frame still uses clustering to obtain 9 clustering centers, the clustering centers are uniformly distributed to 3 scales according to the sizes, and each output grid outputs 3 prediction frames, so that the fine granularity of the grid is more accurate.
The conventional YOLOv3 employs a full-volume neural network Darknet-53 which comprises 53 convolutional layers, each of which is followed by a batch normalization layer and an activation function (learu) layer, and does not have a pooling layer, and performs a down-sampling process of a feature map by using a convolutional layer with a step size of 2 instead of the pooling layer, which can effectively prevent the loss of low-level features due to the pooling layer. The traditional YOLOv3 uses Darknet-53 to extract features, and then the extracted picture features are respectively substituted into a feature fusion network to perform feature fusion, in order to identify more objects, especially small objects, the YOLOv3 uses three different scales to perform prediction (not only 13 × 13 is used), and the steps of the three different scales are 32, 16 and 8 respectively. We modified its feature extraction neural network Darknet-53, and replaced it with ViT (Vision Transformer) neural network, to obtain YOLOv3-ViT, compared with the most advanced convolution network, the vision converter (ViT) can obtain excellent effect, at the same time, the computing resource needed by training is greatly reduced. Because food materials on the market have many categories and the required data set needs to be created greatly, ViT has the advantages that compared with the traditional visual recognition neural network CNN, when the data set is large enough, the advantages of induction bias and Transformer comparison are lost, so far, the traditional Yolov3 is changed, the data set created by the people is processed by utilizing the advantages of ViT model simplicity and high operation speed, the improved neural network is utilized to optimize the recognition model of the people, the neural network improved based on the Yolov3 is compared with the recognition model obtained by the traditional Yolov3 neural network, and the recognition model obtained by the novel neural network has the advantages of higher recognition accuracy, higher detection speed, strong generalization capability and higher stability.
And 4, performing deep learning training on the food material type recognition model in the step 2 by using the food material sample image data set in the step 2 to obtain a trained food material type recognition model, performing model quantization on the trained food material type recognition model, and deploying the trained food material type recognition model to a core controller of an intelligent refrigerator.
The ncnn model generated by deep learning algorithm training cannot be directly used by the core controller at this time, a MaxHub model conversion tool is needed to be used for carrying out quantization model conversion to be an awnn model which can be directly used, and the lightweight model is deployed on the core controller.
And 5, carrying out image acquisition and image preprocessing on food materials stored in the heat preservation box body 1 by each vision sensing module 2 of the intelligent refrigerator, and then sending the food materials into a core controller, identifying the food materials in the heat preservation box body 1 of the intelligent refrigerator by the core controller by using the deployed food material type identification model to obtain the types of the food materials, and counting the number of various food materials in the heat preservation box body 1 of the intelligent refrigerator.
The method comprises the steps that the visual sensing module 2 is used for shooting articles needing to be classified and identified, the size of a picture is converted into 224, identified image information is converted into BASE64 information, then the information of the visual sensing module 2BASE64 is sent to a core controller based on an MQTT protocol, the core controller conducts image identification by using a deployed food material type identification model, and the real-time change situation of food materials in the refrigerator and the information of the food materials in the refrigerator can be obtained, for example: type, amount, etc. And stores the identification information in the core controller and transmits the identification information to the user terminal (such as a mobile phone APP) through the communication module.
Step 6, irradiating given food material samples of different types and different freshness by using a spectrometer to obtain spectral data of the food material samples; performing data dimensionality reduction on the spectral data by using a principal component analysis algorithm, and extracting characteristic wavelengths by using a random forest algorithm; and then establishing a spectral index reflecting the absorption peak characteristics of the food freshness factor by using the extracted characteristic wavelength.
Principal Component Analysis (PCA) is the most common and effective algorithm for solving the problem of spectral data collinearity, extracting data characteristic information and realizing data variable compression. Principal component analysis the extracted principal components are linear combinations of the original variables, which are orthogonal. The information contained in the principal components does not overlap, which may eliminate multiple collinearity between the variables.
A Random Forest (RF) algorithm is a variable optimization algorithm that determines the number of characteristic wavelengths by setting an appropriate selection threshold and calculates the selection probability of each variable by iterative modeling using a small number of variables.
And establishing a spectral index reflecting the characteristic of the absorption peak of the food freshness factor by using the extracted characteristic wavelength, wherein the spectral index comprises an absorption peak depth index, a relative depth index of the absorption peak, a normalized intensity difference index and an absorbance difference index.
The absorption peak depth index a is:
A＝R 3 +R 1 -2R 2
the relative depth index B of the absorption peaks is:
B＝R 2 /(R 3 +R 1 )
the normalized intensity difference index C is:
C＝(R 3 -R 2 )/(R 3 +R 2 )
the absorbance difference index D is:
D＝log 10 (R 3 /R 2 )
in the formula, R 1 Characteristic wavelength, R, of the origin of the absorption peak 2 Characteristic wavelength of the lowest point of the absorption peak, R 3 Characteristic wavelength of the end point of the absorption peak.
And 7, artificially labeling the food freshness of the food sample spectral index obtained in the step 6, and creating a food sample spectral index data set.
The spectral index of the food material sample is used for reflecting the freshness of the food material, and the freshness of the food material can be distinguished through the set corresponding relation criterion of the spectral index and the freshness, so that the specific freshness of the food material is determined, and a corresponding fresh or stale label is added to the spectral index of the food material.
Acquiring the spectral index of the food material, establishing a data set for the spectral index, labeling the spectral index in the data set to further distinguish the freshness level of the food material, and according to the following steps of 7: 2: 1 is divided into a training set, a test set and a verification set.
And 8, constructing a food material freshness identification model based on an LDA (Linear discriminant analysis) neural network.
LDA is a supervised learning dimension reduction technique in which each sample of the dataset is output by class. The basic idea of LDA: given a training sample set, the samples are projected on a straight line, so that the projection points of the same type of samples are as close as possible, and the centers of the projection points of the different types of samples are as far away as possible. The specific process of LDA is as follows: inputting: the data set D { (x1, y1), (x2, y2),. · (xm, ym) }, where any sample xi is an n-dimensional vector, yi ∈ { C1, C2.., Ck }, for k classes. Reducing the dimension of the product; and (3) outputting: and D', reducing the dimension of the data set. (1) Calculating an inter-class divergence matrix SB; calculating an intra-class divergence matrix SW; (3) calculating SB and SW through a formula SW-1SBw which is equal to lambda w to obtain an eigenvalue lambda and an eigenvector w, and multiplying the previous several largest eigenvalue vectors lambda 'with the eigenvector w to obtain a dimension reduction conversion matrix lambda' w; (4) multiplying the original data by a conversion matrix to obtain data (lambda' w) Tx after dimensionality reduction; thereby classifying it.
And 9, performing deep learning training on the food material freshness identification model in the step 8 by using the food material sample spectral index data set in the step 7 to obtain a trained food material freshness identification model, performing model quantization on the trained food material freshness identification model, and deploying the trained food material freshness identification model to a core controller of an intelligent refrigerator.
Substituting the spectral index dataset into the LDA neural network to perform model training, establishing a food freshness identification model, and comparing the spectral indexes of food materials through the model, so that the freshness degree of the food materials can be judged through the spectral indexes, and the food freshness level is obtained.
And step 10, collecting spectral data of food materials stored in the heat preservation box body 1 by each spectrometer module 3 of the intelligent refrigerator, and processing the collected spectral data by the same method as the step 6, namely performing data dimension reduction on the spectral data by using a principal component analysis algorithm, extracting characteristic wavelengths by using a random forest algorithm, and obtaining a spectral index by using the extracted characteristic wavelengths. And then, the spectral index is sent to a core controller, the core controller identifies the food materials in the insulation box body 1 of the intelligent refrigerator by utilizing the deployed food material freshness identification model to determine the food material freshness, and when the food materials in the insulation box body 1 of the intelligent refrigerator are determined to be not fresh, an alarm is sent to a user.
And 11, transmitting the types and the quantity of the food materials obtained in the step 5 and the freshness of the food materials obtained in the step 10 to a user terminal by utilizing a communication module.
The intelligent refrigerator carries out image recognition and spectrum detection on food materials stored in the intelligent refrigerator to obtain food material information data (type, quantity and freshness), and transmits the food material information data to the Cloud server, and the EMQX Cloud server stores the identification information and transmits the identification information to the user terminal. The user terminal is internally provided with app matched with functions of the user terminal and used for receiving food material information data transmitted by the server from the intelligent refrigerator, food material type information is obtained according to an image recognition technology, a menu is recommended through app intelligence, spectral detection is achieved in real time to obtain food material freshness information, an early warning function is performed, when the food material freshness reaches a warning value, the user is warned through app, the user is warned as soon as possible to process food materials, and the problems of overdue food health and the like are prevented.
It should be noted that, although the above-mentioned embodiments of the present invention are illustrative, the present invention is not limited thereto, and thus the present invention is not limited to the above-mentioned embodiments. Other embodiments, which can be made by those skilled in the art in light of the teachings of the present invention, are considered to be within the scope of the present invention without departing from its principles.
Claims (9)
1. The intelligent refrigerator based on machine vision and spectrum detection comprises a heat preservation box body (1), a refrigeration system, an electric appliance system and a core controller; the refrigeration system, the electrical system and the core controller are all positioned in the heat insulation box body (1), and the refrigeration system is electrically connected with the electrical system and the core controller; the device is characterized by further comprising at least one visual sensing module (2) and at least one spectrometer module (3); the vision sensing modules (2) and the spectrometer modules (3) are distributed and installed in the heat preservation box body (1) and face towards food materials stored in the heat preservation box body (1); the output ends of the visual sensing module (2) and the spectrometer module (3) are electrically connected with the core controller.
2. The intelligent refrigerator based on machine vision and spectrum detection as claimed in claim 1, wherein each interlayer of the thermal insulation box body (1) is provided with a vision sensing module (2) and a spectrometer module (3).
3. The intelligent refrigerator based on machine vision and spectral detection as claimed in claim 1, further comprising a communication module, wherein the communication module is arranged in the heat preservation box body (1) and is electrically connected with the core controller.
4. The method for operating an intelligent refrigerator based on machine vision and spectral detection as claimed in claim 1, wherein the method comprises the steps of:
step 1, shooting given food material samples of different types and different freshness in a multi-angle and multi-background environment, and carrying out image preprocessing on the shot images to obtain food material sample images;
step 2, manually labeling food material types of the food material sample images obtained in the step 1, and creating a food material sample image data set;
step 3, constructing a food material type identification model based on a YOLOv3-ViT neural network;
step 4, performing deep learning training on the food material type recognition model in the step 2 by using the food material sample image data set in the step 2 to obtain a trained food material type recognition model, performing model quantization on the trained food material type recognition model, and deploying the trained food material type recognition model to a core controller of an intelligent refrigerator;
step 5, carrying out image acquisition and image preprocessing on food materials stored in the heat preservation box body (1) by each visual sensing module (2) of the intelligent refrigerator, and then sending the food materials into a core controller, identifying the food materials in the heat preservation box body (1) of the intelligent refrigerator by the core controller by using a deployed food material type identification model to obtain the types of the food materials, and counting the number of the various food materials in the heat preservation box body (1) of the intelligent refrigerator;
step 6, irradiating given food material samples of different types and different freshness by using a spectrometer to obtain spectral data of the food material samples; performing data dimension reduction on the spectral data by using a principal component analysis algorithm, and extracting characteristic wavelengths by using a random forest algorithm; then establishing a spectral index reflecting the absorption peak characteristics of the food freshness factor by using the extracted characteristic wavelength;
step 7, artificially labeling the food freshness of the food material sample spectral index obtained in the step 6, and creating a food material sample spectral index data set;
step 8, constructing a food material freshness identification model based on an LDA neural network;
step 9, performing deep learning training on the food material freshness identification model in the step 8 by using the food material sample spectral index data set in the step 7 to obtain a trained food material freshness identification model, performing model quantization on the trained food material freshness identification model, and deploying the trained food material freshness identification model to a core controller of an intelligent refrigerator;
step 10, spectrum data of food materials stored in the heat preservation box body (1) are collected through each spectrometer module (3) of the intelligent refrigerator, data dimensionality reduction is conducted on the spectrum data through a principal component analysis algorithm, characteristic wavelengths are extracted through a random forest algorithm, spectrum indexes are obtained through the extracted characteristic wavelengths and then are sent to a core controller, and the core controller identifies the food materials in the heat preservation box body (1) of the intelligent refrigerator through a deployed food material freshness identification model to determine food material freshness.
5. The method as claimed in claim 4, wherein the food material type recognition model constructed in step 3 comprises an image slice layer, a position embedding layer, 6 visual coding modules, 2 tensor splicing layers, 3 residual convolution layers, 2 upsampling layers, 8 convolution layers, and a multi-scale prediction layer;
the input of the image slice layer is used as the input of the food material type identification model, the output of the image slice layer is connected with the input of the position embedding layer, the output of the position embedding layer is connected with the input of the first visual coding module, the output of the first visual coding module is connected with the input of the second visual coding module, the output of the second visual coding module is connected with the input of the third visual coding module, the output of the third visual coding module is connected with the input of the fourth visual coding module, one output of the fourth visual coding module is connected with the input of the fifth visual coding module, and one output of the fifth visual coding module is connected with the input of the sixth visual coding module; the output of the sixth visual coding module is connected with the input of the first residual convolution layer, the other output of the fifth visual coding module is connected with one input of the first tensor splicing layer, and the other output of the fourth visual coding module is connected with one input of the second tensor splicing layer;
one output of the first residual convolution layer is connected with the input of the first convolution layer, the output of the first convolution layer is connected with the input of the first up-sampling layer, the input of the first up-sampling layer is connected with the other input of the first tensor splicing layer, one output of the first tensor splicing layer is connected with the other input of the second tensor splicing layer, and the output of the second tensor splicing layer is connected with the input of the second residual convolution layer;
the other output of the first scalar splicing layer is connected with the input of a third residual convolutional layer, one output of the third residual convolutional layer is connected with the input of a second convolutional layer, the output of the second convolutional layer is connected with the input of the third convolutional layer, and the output of the third convolutional layer is connected with one input of the multi-scale prediction layer; the other output of the third residual convolutional layer is connected with the input of a fourth convolutional layer, the output of the fourth convolutional layer is connected with the input of a second upsampling layer, and the output of the second upsampling layer is connected with the other input of the second tensor splicing layer; the other output of the first residual convolutional layer is connected with the input of a fifth convolutional layer, the output of the fifth convolutional layer is connected with the input of a sixth convolutional layer, and the output of the sixth convolutional layer is connected with the other input of the multi-scale prediction layer; the output of the second residual convolutional layer is connected with the input of the seventh convolutional layer, the output of the seventh convolutional layer is connected with the input of the eighth convolutional layer, and the output of the eighth convolutional layer is connected with the other input of the multi-scale prediction layer; and outputting the multi-scale prediction layer as the output of the food material type identification model.
6. The method for operating an intelligent refrigerator based on machine vision and spectrum detection as claimed in claim 5, wherein in step 3, each visual coding module of the food material type identification model is composed of a picture embedding layer, 2 normalization function layers, a multi-head attention layer, 2 merging layers and a multi-layer perception mechanism layer;
the input of the picture embedding layer is used as the input of the visual coding module; one output of the picture embedding layer is connected with the input of the first normalization function layer, the output of the first normalization function layer is connected with the input of the multi-head attention layer, and the output of the multi-head attention layer is connected with one input of the first merging layer; the other output of the picture embedding layer is connected with the other input of the first merging layer;
one output of the first merging layer is connected with the input of the second normalization function layer, the output of the second normalization function layer is connected with the input of the multilayer sensing mechanism layer, and the output of the multilayer sensing mechanism layer is connected with one input of the second merging layer; the other output of the first merging layer is connected with the other input of the second merging layer; the output of the second merging layer is used as the output of the visual coding module.
7. The method as claimed in claim 4, wherein in step 6, the spectral indexes reflecting the characteristic of the absorption peak of the food freshness factor include an absorption peak depth index, a relative depth index of the absorption peak, a normalized intensity difference index and an absorbance difference index; wherein
The absorption peak depth index a is:
A＝R 3 +R 1 -2R 2
the relative depth index B of the absorption peaks is:
B＝R 2 /(R 3 +R 1 )
the normalized intensity difference index C is:
C＝(R 3 -R 2 )/(R 3 +R 2 )
the absorbance difference index D is:
D＝log 10 (R 3 /R 2 )
in the formula, R 1 Characteristic wavelength, R, of the origin of the absorption peak 2 Characteristic wavelength of the lowest point of the absorption peak, R 3 Characteristic wavelength of the end point of the absorption peak.
8. The method for operating the intelligent refrigerator based on machine vision and spectral detection as claimed in claim 4, wherein the step 9 further comprises giving an alarm to the user when the food material in the refrigerator body (1) is determined to be not fresh.
9. The method for operating an intelligent refrigerator based on machine vision and spectral detection as claimed in claim 4, further comprising:
and 11, transmitting the types and the quantity of the food materials obtained in the step 5 and the freshness of the food materials obtained in the step 10 to a user terminal by utilizing a communication module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210768651.XA CN115046363B (en) | 2022-06-30 | 2022-06-30 | Intelligent refrigerator based on machine vision and spectrum detection and operation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210768651.XA CN115046363B (en) | 2022-06-30 | 2022-06-30 | Intelligent refrigerator based on machine vision and spectrum detection and operation method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115046363A true CN115046363A (en) | 2022-09-13 |
| CN115046363B CN115046363B (en) | 2024-05-07 |
Family
ID=83165028
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210768651.XA Active CN115046363B (en) | 2022-06-30 | 2022-06-30 | Intelligent refrigerator based on machine vision and spectrum detection and operation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115046363B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105181912A (en) * | 2015-06-30 | 2015-12-23 | 江苏大学 | Method for detection of freshness during rice storage |
| CN207164027U (en) * | 2017-06-07 | 2018-03-30 | 浙江凯尔奇电器有限公司 | Food materials freshness intelligent detecting system in a kind of refrigerator-freezer |
| EP3527918A2 (en) * | 2018-02-14 | 2019-08-21 | Whirlpool Corporation | Foodstuff sensing appliance |
| US20200097776A1 (en) * | 2018-09-21 | 2020-03-26 | Samsung Electronics Co., Ltd. | Method and system for providing information related to a status of an object in a refrigerator |
| CN113792578A (en) * | 2021-07-30 | 2021-12-14 | 北京智芯微电子科技有限公司 | Method, device and system for detecting abnormity of transformer substation |
| CN113807361A (en) * | 2021-08-11 | 2021-12-17 | 华为技术有限公司 | Neural network, target detection method, neural network training method and related products |
| US20220191432A1 (en) * | 2020-12-11 | 2022-06-16 | Samsung Electronics Co., Ltd. | Food monitoring apparatus, refrigerator including the same, and operating method thereof |
-
2022
- 2022-06-30 CN CN202210768651.XA patent/CN115046363B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105181912A (en) * | 2015-06-30 | 2015-12-23 | 江苏大学 | Method for detection of freshness during rice storage |
| CN207164027U (en) * | 2017-06-07 | 2018-03-30 | 浙江凯尔奇电器有限公司 | Food materials freshness intelligent detecting system in a kind of refrigerator-freezer |
| EP3527918A2 (en) * | 2018-02-14 | 2019-08-21 | Whirlpool Corporation | Foodstuff sensing appliance |
| US20200097776A1 (en) * | 2018-09-21 | 2020-03-26 | Samsung Electronics Co., Ltd. | Method and system for providing information related to a status of an object in a refrigerator |
| CN112740229A (en) * | 2018-09-21 | 2021-04-30 | 三星电子株式会社 | Method and system for providing information related to state of object in refrigerator |
| US20220191432A1 (en) * | 2020-12-11 | 2022-06-16 | Samsung Electronics Co., Ltd. | Food monitoring apparatus, refrigerator including the same, and operating method thereof |
| CN113792578A (en) * | 2021-07-30 | 2021-12-14 | 北京智芯微电子科技有限公司 | Method, device and system for detecting abnormity of transformer substation |
| CN113807361A (en) * | 2021-08-11 | 2021-12-17 | 华为技术有限公司 | Neural network, target detection method, neural network training method and related products |
Non-Patent Citations (2)
| Title |
|---|
| 王国睿: "基于Transformer改进YOLO v4的火灾检测方法", 智能计算机与应用, vol. 11, no. 7, 31 July 2021 (2021-07-31), pages 86 - 90 * |
| 黄颖康: "基于轻量级神经网络的食材识别系统研究与应用", 广东工业大学硕士论文, 15 March 2022 (2022-03-15), pages 26 - 48 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115046363B (en) | 2024-05-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9367730B2 (en) | Method and system for automated face detection and recognition | |
| CN111291809A (en) | Processing device, method and storage medium | |
| CN111274972B (en) | Dish identification method and device based on measurement learning | |
| CN109492528A (en) | A kind of recognition methods again of the pedestrian based on gaussian sum depth characteristic | |
| CN117115571B (en) | Fine-grained intelligent commodity identification method, device, equipment and medium | |
| CN112906780A (en) | Fruit and vegetable image classification system and method | |
| CN114898158A (en) | Small sample traffic abnormity image acquisition method and system based on multi-scale attention coupling mechanism | |
| CN113642646B (en) | Image threat object classification and positioning method based on multi-attention and semantics | |
| CN117058517A (en) | Helmet detection method, device and medium based on YOLOv5 optimization model | |
| CN111310531B (en) | Image classification method, device, computer equipment and storage medium | |
| US11394929B2 (en) | System and method for language-guided video analytics at the edge | |
| CN115046363A (en) | Intelligent refrigerator based on machine vision and spectrum detection and operation method | |
| CN112257506A (en) | Fruit and vegetable size identification method and device, electronic equipment and computer readable medium | |
| Nayak et al. | Fruit recognition using image processing | |
| Steffens et al. | A texture driven approach for visible spectrum fire detection on mobile robots | |
| US20220335725A1 (en) | Monitoring presence or absence of an object using local region matching | |
| CN113465251B (en) | Intelligent refrigerator and food material identification method | |
| CN110646350B (en) | Product classification method, device, computing equipment and computer storage medium | |
| Sehgal et al. | Auto-annotation of tomato images based on ripeness and firmness classification for multimodal retrieval | |
| Saikia et al. | Colour neural descriptors for instance retrieval using CNN features and colour models | |
| CN110097135B (en) | Holographic diffraction label image recognition algorithm based on double tensors | |
| Singh | Single Stage Facial Recognition based on YOLOv5 | |
| Nawfal et al. | Performance Evaluation Between Tiny Yolov3 and MobileNet SSDv1 for Object Detection | |
| Mitrea et al. | Ensemble-based learning using few training samples for video surveillance scenarios | |
| Musab et al. | Convolution Neural Network-Based Meat Quality Checker Using Different Optimizers |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |