CN113536989A - Refrigerator frosting monitoring method and system based on camera video frame-by-frame analysis - Google Patents
Refrigerator frosting monitoring method and system based on camera video frame-by-frame analysis Download PDFInfo
- Publication number
- CN113536989A CN113536989A CN202110727931.1A CN202110727931A CN113536989A CN 113536989 A CN113536989 A CN 113536989A CN 202110727931 A CN202110727931 A CN 202110727931A CN 113536989 A CN113536989 A CN 113536989A
- Authority
- CN
- China
- Prior art keywords
- image
- model
- neural network
- convolutional neural
- refrigerator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 34
- 238000012544 monitoring process Methods 0.000 title claims abstract description 34
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 60
- 239000010410 layer Substances 0.000 claims description 36
- 230000006870 function Effects 0.000 claims description 28
- 238000004364 calculation method Methods 0.000 claims description 20
- 230000004913 activation Effects 0.000 claims description 19
- 238000011423 initialization method Methods 0.000 claims description 18
- 238000007781 pre-processing Methods 0.000 claims description 17
- 238000005070 sampling Methods 0.000 claims description 15
- 230000002457 bidirectional effect Effects 0.000 claims description 10
- 238000003709 image segmentation Methods 0.000 claims description 7
- 238000013136 deep learning model Methods 0.000 claims description 6
- 239000002346 layers by function Substances 0.000 claims description 6
- 238000011176 pooling Methods 0.000 claims description 6
- 230000004927 fusion Effects 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000005192 partition Methods 0.000 claims description 3
- 230000017105 transposition Effects 0.000 claims description 3
- 230000007613 environmental effect Effects 0.000 description 16
- 235000013305 food Nutrition 0.000 description 10
- 238000003860 storage Methods 0.000 description 9
- 238000005057 refrigeration Methods 0.000 description 8
- 238000005265 energy consumption Methods 0.000 description 7
- 238000003745 diagnosis Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000018109 developmental process Effects 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 235000013399 edible fruits Nutrition 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 235000013622 meat product Nutrition 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000010223 real-time analysis Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 238000010257 thawing Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000009827 uniform distribution Methods 0.000 description 2
- 235000013311 vegetables Nutrition 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 230000007797 corrosion Effects 0.000 description 1
- 238000005260 corrosion Methods 0.000 description 1
- 235000013365 dairy product Nutrition 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 235000013372 meat Nutrition 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000000275 quality assurance Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
The invention provides a refrigerator frosting monitoring method and system based on camera video frame-by-frame analysis, wherein the method comprises the following steps: step 1, acquiring video data of an area where a condenser in a refrigerator is located; step 2, separating a condenser area image from the video data; step 3, determining whether the separated condenser area image needs to be preprocessed, if so, entering step 4, and if not, entering step 6; step 4, judging whether the separated condenser area image needs image quality enhancement or not, if so, performing super-resolution image enhancement and then entering step 5, otherwise, entering step 6; step 5, constructing a convolutional neural network model and predicting the frosting condition of the refrigerating machine in real time by using the constructed convolutional neural network model; and 6, constructing a ConvLSTM model and predicting the frosting condition of the refrigerating machine in real time by using the constructed ConvLSTM model. The invention can accurately diagnose the frosting condition of the refrigerating machine.
Description
Technical Field
The invention relates to the technical field of refrigeration equipment frosting diagnosis, in particular to a refrigerator frosting monitoring method and system based on camera video frame-by-frame analysis.
Background
With the development of society and the improvement of living standard of residents, the quality assurance and food safety problems of food are more and more emphasized by society, and the quality guarantee, storage and circulation of food are also one of the most important research contents in the fields of agriculture and food industry. In 2019, the import quantity of frozen and refrigerated aquatic products and meat products in China rises to 1000 ten thousand tons, the total output of fruits, vegetables, meat products, aquatic products and dairy products is expected to break through 13 hundred million tons, the cold chain market demand is huge, but the comprehensive cold chain circulation rate is only 17% (2.2563 million tons), the corrosion loss rate is high (the fruit, vegetable, meat and aquatic products respectively reach 20% -30%, 12% and 15%), the direct economic loss exceeds 6800 million yuan (about 1% of GDP in China), and the great waste of social resources is caused. Cold chain logistics, which provide a suitable temperature environment for perishable food storage and circulation, are critical to reduce spoilage rates, maintain food quality and safety.
The cold storage is an important infrastructure for food freezing processing, storage and circulation, is a key node of a whole cold chain, has no replaceable function in the aspects of cold chain commodity storage and quality guarantee, plays an important role in national economy and has huge total amount. The total amount of the cold storage in the country in 2018 reaches 5238 ten thousand tons (about 1.3 billion cubic meters), the total logistics amount exceeds 4 trillion yuan, and the cold storage construction and technical research and development are important research contents in the fields of food industry and logistics management. However, with the rapid development of cold-chain logistics represented by a cold storage, the energy consumption thereof is rapidly increased, and taking the food industry as an example, the energy consumption of a refrigeration system including production, circulation and storage links accounts for 35.0% of the total energy consumption of the food industry, and the total energy consumption reaches 1300 TWh/year in the global scope, which is a serious challenge. And because the environment in the logistics process needs to be controlled, the cost of cold-chain logistics is higher than that of common logistics by more than 40.0%, and the problems of high cost and low efficiency are particularly obvious. In each link of cold-chain logistics, the working efficiency of a refrigerating system is a key factor for increasing the refrigerating effect and reducing the energy consumption, and because the air outlet of the refrigerator is very easy to frost in a low-temperature environment, the refrigerating efficiency is severely restricted, the product refrigeration of the logistics system can be influenced, and the energy consumption cost is greatly increased.
Therefore, the design of the intelligent refrigerator working state diagnosis method has important significance for improving the refrigeration efficiency and reducing the energy consumption cost. However, the conventional sensor monitoring method can only reflect the temperature and humidity of the environment, and cannot accurately diagnose the frosting condition of the refrigerator.
Disclosure of Invention
Aiming at the defects of the prior art, the invention provides a refrigerator frosting monitoring method based on camera video frame-by-frame analysis, which can accurately diagnose the frosting condition of a refrigerator.
In order to achieve the purpose, the invention provides a refrigerator frosting monitoring method based on camera video frame-by-frame analysis, which comprises the following steps:
step 1, acquiring video data of an area where a condenser in a refrigerator is located;
step 2, separating a condenser area image from the video data according to the video data;
step 3, determining whether the separated condenser area image needs to be preprocessed, if so, entering step 4, and if not, entering step 6;
step 4, judging whether the separated condenser area image needs image quality enhancement or not, if so, performing super-resolution image enhancement on the separated condenser area image, and then entering step 5, otherwise, entering step 6;
step 5, constructing a convolutional neural network model according to the processed image data of the condenser area, and then predicting the frosting condition of the refrigerating machine in real time by using the constructed convolutional neural network model;
and 6, constructing a ConvLSTM model according to the separated condenser area image, and then predicting the frosting condition of the refrigerating machine in real time by using the constructed ConvLSTM model.
Further, in step 2, the step of separating the condenser area image from the video data includes:
step 201, constructing a running environment of a segmented convolutional neural network model;
step 202, performing convolution calculation on an input image to realize down-sampling of the image, then performing multi-scale feature fusion calculation, and inputting the calculated abstract features into the next convolution layer;
and 203, performing transposition convolution calculation on the input of the next convolution layer to realize up-sampling of the abstract features, reducing the size of the abstract features to the size of the original image, and reserving the data area of the abstract features so as to partition the condenser area image.
Further, in step 3, the step of determining whether the separated condenser area image needs to be preprocessed includes:
step 301, calculating noise points existing in an image, wherein the noise points include but are not limited to noise points formed by Gaussian noise and impulse noise;
step 302, counting the area of the noise image according to the noise points, and judging whether the area of the noise image is within a preset threshold range; if so, preprocessing is required, otherwise, preprocessing is not required.
Further, in step 4, the step of determining whether the separated condenser area image needs image enhancement includes: and counting the average pixel value of the separated condenser area image, and judging whether the average pixel value is within a preset pixel average value range, if so, performing image enhancement, otherwise, not performing image enhancement.
Further, in step 4, the step of performing super-resolution image enhancement on the separated condenser region image includes:
step 401, constructing a segmented convolutional neural network model operating environment;
step 402, extracting target image features by setting a convolution layer in a deep learning model, and constructing a target feature set;
step 403, calculating the distance between the target feature set and the generated feature set in a high-dimensional feature space through a perceptual loss function, calculating an optimal optimization gradient by using a back propagation algorithm, and optimizing the target image features;
in step 404, the optimized features are up-sampled and calculated, and the resolution enhanced image is output.
Further, in step 5, the step of constructing a convolutional neural network model from the processed condenser region image data includes:
step 501, constructing a convolutional neural network model with a sequential structure, wherein the convolutional neural network model comprises 2 groups of convolutional modules, and each convolutional module comprises 2 convolutional layers and 1 pooling sampling layer;
502, selecting the optimal items of a model initialization method, an activation function and an optimizer;
step 503, calculating and determining the optimal convolution kernel number.
Further, in step 6, the step of constructing the ConvLSTM model from the separated condenser region image includes:
step 601, constructing a ConvLSTM model with a sequential structure, and setting a ConvLSTM functional layer in the model;
step 602, calculating the convolution operation of the image under each time sequence by using a TimeDistributed wrapper;
step 603, calculating the logic relation of the image in the time sequence direction by using a Bidirectional wrapper of Bidirectional, and extracting the time sequence characteristics of the image data;
step 604, selecting the optimal items of the model initialization method, the activation function and the optimizer;
step 605, calculating and determining other model parameters.
On the other hand, the invention also provides a refrigerator frosting monitoring system based on the camera video frame-by-frame analysis, which comprises
The video data acquisition module is used for acquiring video data of an area where a condenser in the refrigerator is located;
the image segmentation module is used for separating a condenser area image from the video data according to the video data;
the image preprocessing module is used for determining whether the separated condenser area image needs preprocessing or not, and if so, preprocessing is carried out;
the image quality enhancement module is used for judging whether the separated condenser area image needs image quality enhancement or not, and if so, performing super-resolution image enhancement on the separated condenser area image;
the first prediction module is used for constructing a convolutional neural network model for the image data of the condenser area subjected to super-resolution image enhancement by the image quality enhancement module and then predicting the frosting condition of the refrigerating machine in real time by utilizing the constructed convolutional neural network model;
and the second prediction module is used for constructing a ConvLSTM model for the condenser area image which does not need to be preprocessed and/or the image quality enhancement module, and then predicting the frosting condition of the refrigerating machine in real time by utilizing the constructed ConvLSTM model.
Further, the first prediction module comprises
The convolutional neural network model building unit is used for building a convolutional neural network module with a sequential structure, wherein the convolutional neural network model comprises 2 groups of convolutional modules, and each convolutional module comprises 2 convolutional layers and 1 pooling sampling layer;
the first optimal item selection unit is used for selecting optimal items of the model initialization method, the activation function and the optimizer;
and the convolution kernel number calculation unit is used for calculating and determining the optimal convolution kernel number.
Further, the second prediction module comprises
The ConvLSTM model building unit is used for building a ConvLSTM model of a sequential structure and setting a ConvLSTM functional layer in the model;
the convolution operation calculating unit is used for calculating the convolution operation of the image under each time sequence through the TimeDistributed wrapper;
the time sequence characteristic determining unit is used for calculating the logic relation of the image in the time sequence direction by using the Bidirectional wrapper and extracting the time sequence characteristic of the image data;
the second optimal item selection unit is used for selecting optimal items of the model initialization method, the activation function and the optimizer;
and the model parameter calculation unit is used for calculating and determining other model parameters.
Compared with the prior art, the invention has the following advantages: according to the method, the camera video monitoring mode is adopted, the image of the air outlet condenser area is collected, the frosting area segmentation and the image super-resolution enhancement are carried out on the image quality through the video image frame-by-frame analysis method, and the accurate judgment method for the frosting condition of the air outlet condenser of the refrigerator system is realized. Specifically, when environmental interference exists in the image, the frosting condition of the refrigerating machine is predicted by constructing a convolutional neural network model; when the image is not interfered, the frosting condition of the refrigerator is predicted by constructing the ConvLSTM model, the defects that monitoring equipment is easily influenced by the environment of a refrigeration house, the definition of image quality is not enough, video information is difficult to extract and the like are overcome, and the purpose of real-time and accurate monitoring of the working state of the refrigerator is achieved at low cost.
In addition, the convolutional neural network group model based on deep learning can effectively process video time sequence data, accurate real-time frosting diagnosis and analysis can be realized through video image content change analysis in a small range of time and by combining with real-time analysis information, and a timely and accurate intelligent early warning function is provided for defrosting and manual intervention of an air outlet condenser.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart of a refrigerator frosting monitoring method based on a camera video frame-by-frame analysis according to the present invention;
fig. 2 is a block diagram of a refrigerator frosting monitoring system based on a camera video frame-by-frame analysis according to the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, the embodiment of the invention discloses a refrigerator frosting monitoring method based on camera video frame-by-frame analysis, which comprises the following steps:
step 1, acquiring video data of an area where a condenser in a refrigerator is located;
step 2, separating a condenser area image from the video data according to the video data;
step 3, determining whether the separated condenser area image needs to be preprocessed, if so, performing preprocessing, and then entering step 4, otherwise, entering step 6;
step 4, judging whether the separated condenser area image needs image quality enhancement or not, if so, performing super-resolution image enhancement on the separated condenser area image, and then entering step 5, otherwise, entering step 6;
step 5, constructing a convolutional neural network model according to the processed image data of the condenser area, and then predicting the frosting condition of the refrigerating machine in real time by using the constructed convolutional neural network model;
and 6, constructing a ConvLSTM model according to the separated condenser area image, and then predicting the frosting condition of the refrigerating machine in real time by using the constructed ConvLSTM model.
Correspondingly, referring to fig. 2, the embodiment of the invention also discloses a refrigerator frosting monitoring system based on the camera video frame-by-frame analysis, which comprises
The video data acquisition module is used for acquiring video data of an area where a condenser in the refrigerator is located;
the image segmentation module is used for separating a condenser area image from the video data according to the video data;
the image preprocessing module is used for determining whether the separated condenser area image needs preprocessing or not, and if so, preprocessing is carried out;
the image quality enhancement module is used for judging whether the separated condenser area image needs image quality enhancement or not, and if so, performing super-resolution image enhancement on the separated condenser area image;
the first prediction module is used for constructing a convolutional neural network model for the image data of the condenser area subjected to super-resolution image enhancement by the image quality enhancement module and then predicting the frosting condition of the refrigerating machine in real time by utilizing the constructed convolutional neural network model;
and the second prediction module is used for constructing a ConvLSTM model for the condenser area image which does not need to be preprocessed and/or the image quality enhancement module, and then predicting the frosting condition of the refrigerating machine in real time by utilizing the constructed ConvLSTM model.
In this embodiment, the refrigerator frosting monitoring method based on the camera video frame-by-frame analysis takes the refrigerator frosting monitoring system based on the camera video frame-by-frame analysis as an execution object of the step, or takes the components in the refrigerator frosting monitoring system based on the camera video frame-by-frame analysis as an execution object of the step. Specifically, step 1 takes a video data acquisition module as an execution object of the step, step 2 takes an image segmentation module as an execution object of the step, step 3 takes an image preprocessing module as an execution object of the step, step 4 takes an image quality enhancement module as an execution object of the step, step 5 takes a first prediction module as an execution object of the step, and step 6 takes a second prediction module as an execution object of the step.
Because the frosting of the condenser (or the evaporator) can block the air channel and obviously affect the working efficiency of the refrigerating machine, the invention uses the camera to monitor the frosting condition of the condenser area, thereby diagnosing the working efficiency of the refrigerating machine. However, images obtained by using the camera are easily interfered by environmental factors in a low-temperature environment of a refrigeration house, and the problems of low resolution, poor imaging effect, incapability of intelligent identification and the like occur; aiming at the situation, the method analyzes the video data collected by the camera, judges whether the image collected by the camera in real time has environmental interference, builds a convolutional neural network model by utilizing the image data of the condenser area after performing super-resolution image enhancement on the separated image of the condenser area if the image has the environmental interference, and then predicts the frosting condition of the refrigerator in real time by utilizing the built convolutional neural network model; if the image collected by the camera in real time does not have environmental interference, a ConvLSTM model is constructed according to the separated condenser area image, and then the frosting condition of the refrigerating machine is predicted in real time by utilizing the constructed ConvLSTM model; according to the invention, different deep learning models are adopted to predict the frosting condition of the refrigerator aiming at the existence of environmental interference, so that the frosting condition of the refrigerator can be accurately diagnosed.
In the step 1, video data of an area where a condenser in a refrigerator is located, which is acquired by a camera, is acquired.
In the step 2, the video frame collected by the camera is segmented to separate the image of the condenser area, so that interference of other irrelevant factors in the image on the later analysis is avoided.
Specifically, in step 2, the step of separating the condenser area image from the video data includes:
step 201, constructing a running environment of a segmented convolutional neural network model;
202, after convolution calculation is carried out on an input image to realize down-sampling of the image, multi-scale feature fusion calculation is carried out through feature point-by-point addition and feature channel dimension splicing, and then calculated abstract features are input into a next convolution layer;
and 203, performing transposition convolution calculation on the next convolution layer input to realize up-sampling of the abstract features, reducing the size of the abstract features to the size of the original image, and reserving the data area of the abstract features so as to partition the condenser area image.
In the embodiment of the invention, a segmentation convolutional neural network model is constructed to perform down sampling and up sampling on an image, collected multi-scale features are fused through feature point-by-point addition and feature channel dimension splicing, and each pixel point is judged to be classified, so that a pixel-level image segmentation result is realized.
Since the network is allowed to accept a picture of an arbitrary size and output a division map of the same size as the original or larger, a transposed convolution layer is provided in the network structure, and a feature map is mapped back to the original size or larger, and a large feature map is scrolled from a small feature map.
When the conventional convolutional neural network processes image segmentation, the rounding approximation is used to find the corresponding model parameters of the image region in the network structure, which may cause the corresponding relationship to deviate from the actual situation. This offset is generally negligible in the image classification task, but may have a large impact in the more refined image segmentation task. Therefore, in the present embodiment, an area feature aggregation manner is used, and the downsampling of the image is implemented by performing convolution calculation on any area in the input image to the corresponding area in the neural network feature map in step 202, so as to aggregate the area features, thereby solving the problem of area mismatch caused by two quantization in the conventional convolutional neural network.
In step 3, after the segmented condenser area image is obtained, whether the separated condenser area image needs to be preprocessed or not is determined, namely whether noise exists or not, and if the noise does not exist, the frosting condition can be predicted by constructing a ConvLSTM model; if noise exists, further judgment is needed.
Specifically, in step 3, the step of determining whether the separated condenser area image needs to be preprocessed includes:
step 301, calculating noise points existing in an image, wherein the noise points include but are not limited to noise points formed by Gaussian noise and impulse noise;
step 302, counting the area of the noise image according to the noise points, and judging whether the area of the noise image is within a preset threshold range; if so, preprocessing is required, otherwise, preprocessing is not required.
In the embodiment of the invention, after the segmented condenser area image is acquired, noise points existing in the image are calculated, the area of the noise image is counted, and when the area of the noise image is larger than the range of a preset threshold value, the acquired image is possibly interfered by the environment and needs to be further judged; and if the area of the noise image is within the preset threshold value, the collected image is proved to have no environmental interference, so that the frosting condition can be directly predicted by constructing a ConvLSTM model.
In step 4, after removing noise from the separated condenser area image, it is further determined whether the image has environmental interference, and specifically, it is determined whether the environmental interference exists by determining whether the separated condenser area image needs image enhancement.
Specifically, in step 4, the step of determining whether the separated condenser area image needs image enhancement includes: and counting the average pixel value of the separated condenser area image, and judging whether the average pixel value is within a preset pixel average value range, if so, performing image enhancement, otherwise, not performing image enhancement.
In the embodiment of the invention, the determination of the range of the preset pixel average value is performed by calculating the pixel average value of a plurality of images with environmental interference; if the average pixel value of the separated condenser area image is within the preset pixel average value range, the image is proved to have the condition of environmental interference, so that the super-resolution image enhancement needs to be carried out on the image, and then a convolutional neural network model is constructed to predict the frosting condition.
Specifically, in step 4, the step of performing super-resolution image enhancement on the separated condenser region image includes:
step 401, constructing a segmented convolutional neural network model operating environment;
step 402, extracting target image features by setting a convolution layer in a deep learning model, and constructing a target feature set;
step 403, calculating the distance between the target feature set and the generated feature set in a high-dimensional feature space through a perceptual loss function, calculating an optimal optimization gradient by using a back propagation algorithm, and optimizing the target image features;
in step 404, the optimized features are up-sampled and calculated, and the resolution enhanced image is output.
Since the conventional convolutional neural network usually uses the mean square error as a loss function when training the network, although a high peak signal-to-noise ratio can be ensured, the generated image usually loses high-frequency details and cannot be applied to identifying the frosting degree of a condenser (or an evaporator). In the embodiment of the invention, in order to improve the characteristic details of the restored picture, the difference between the current model weight and the target expectation is calculated by adopting the perception loss, and the training effect is achieved by continuously correcting the model coefficient; specifically, by utilizing the image characteristics of the condenser (or evaporator) area extracted by the convolutional layer, the generated picture and the target picture are more similar in semantics and style by comparing the difference between the characteristics of the generated picture after passing through the convolutional neural network and the characteristics of the target picture after passing through the convolutional neural network.
In step 402, convolutional layers need to be set in the deep learning model, and a plurality of convolutional layers are set in the convolution module in consideration of the pixel size of the evaporator (or condenser) region, and the sizes of convolution kernels are set to 3 × 3, 5 × 5, and 7 × 7, respectively.
In the step 5, if the acquired image has environmental interference, a convolution neural network model is constructed to predict the frosting condition of the refrigerating machine after the super-resolution enhancement is carried out on the condenser area image with the environmental interference.
Specifically, in step 5, the step of constructing the convolutional neural network model according to the processed condenser region image data includes:
step 501, constructing a convolutional neural network model with a sequential structure, wherein the convolutional neural network model comprises 2 groups of convolutional modules, and each convolutional module comprises 2 convolutional layers and 1 pooling sampling layer;
502, selecting the optimal items of a model initialization method, an activation function and an optimizer;
step 503, calculating and determining the optimal convolution kernel number.
Correspondingly, in the refrigerator frosting monitoring system based on the frame-by-frame analysis of the camera video, the first prediction module comprises
The convolutional neural network model building unit is used for building a convolutional neural network module with a sequential structure, wherein the convolutional neural network model comprises 2 groups of convolutional modules, and each convolutional module comprises 2 convolutional layers and 1 pooling sampling layer;
the first optimal item selection unit is used for selecting optimal items of the model initialization method, the activation function and the optimizer;
and the convolution kernel number calculation unit is used for calculating and determining the optimal convolution kernel number.
In this embodiment, step 5 takes the first prediction module as the execution object of the step, or takes the component of the first prediction module as the execution object of the step. Specifically, step 501 takes a convolutional neural network model building unit as an execution object of the step, step 502 takes a first optimal item selection unit as an execution object of the step, and step 503 takes a convolutional kernel number calculation unit as an execution object of the step.
In the embodiment of the present invention, considering that much work in the CNN model in this embodiment is based on the 2/4 grouping structure, the performance of 2/4 is superior and stable in large-size image analysis, so in the case of a convolutional neural network model with a sequential structure, the convolutional neural network model includes 2 sets of convolution modules, each convolution module includes 2 convolutional layers and 1 pooled sampling layer, so as to improve the stability of the prediction model.
And constructing a convolutional neural network model, determining an initialization method, an activation function and an optimizer of the model, and then calculating the optimal convolutional kernel number by using the determined initialization method, activation function and optimizer, thereby finally determining the convolutional neural network model.
Specifically, the initialization method comprises uniform distribution initialization, all-0 initialization, all-1 initialization, a fixed value initialization method, normal distribution initialization, random uniform distribution initialization, truncated Gaussian distribution initialization, random orthogonal matrix initialization, identity matrix initialization and the like, 2000 training image data are trained by the model through iteration of each optimizer for 2000 rounds, and influences of different initialization methods on the performance of the model are researched; and determining the optimal initialization method by comprehensively considering the loss function and the accuracy of the test set.
For the selection of the activation function, the embodiment of the invention compares softmax, relu, sigmoid and LeakyReLu functions commonly used in a deep learning model as the activation functions to evaluate the optimal activation function performance. The optimal activation function is determined by comparing whether there is a non-0 output when the activation function is inactive.
For the selection of the optimizer, the embodiment of the invention respectively performs the performance analysis of the optimizer on seven algorithms such as an RMSprop algorithm, an Adam algorithm, a random gradient descent algorithm, an Adagrad algorithm, an Adadelta algorithm, an Adamax algorithm, a Nadam algorithm and the like.
And 6, if the acquired image has no environmental interference, constructing a ConvLSTM model to predict the frosting condition of the refrigerating machine.
Specifically, in step 6, the step of constructing the ConvLSTM model from the separated condenser region image includes:
step 601, constructing a ConvLSTM model with a sequential structure, and setting a ConvLSTM functional layer in the model;
step 602, calculating the convolution operation of the image under each time sequence by using a TimeDistributed wrapper;
step 603, calculating the logic relation of the image in the time sequence direction by using a Bidirectional wrapper of Bidirectional, and extracting the time sequence characteristics of the image data;
step 604, selecting the optimal items of the model initialization method, the activation function and the optimizer;
step 605, calculating and determining other model parameters.
Correspondingly, in the refrigerator frosting monitoring system based on the frame-by-frame analysis of the camera video, the second prediction module comprises
The ConvLSTM model building unit is used for building a ConvLSTM model of a sequential structure and setting a ConvLSTM functional layer in the model;
the convolution operation calculating unit is used for calculating the convolution operation of the image under each time sequence through the TimeDistributed wrapper;
the time sequence characteristic determining unit is used for calculating the logic relation of the image in the time sequence direction by using the Bidirectional wrapper and extracting the time sequence characteristic of the image data;
the second optimal item selection unit is used for selecting optimal items of the model initialization method, the activation function and the optimizer;
and the model parameter calculation unit is used for calculating and determining other model parameters.
In this embodiment, step 6 uses the second prediction module as the execution target of the step, or uses the component of the second prediction module as the execution target of the step. Specifically, step 601 takes a ConvLSTM model building unit as an execution object of the step, step 602 takes a convolution operation calculation unit as an execution object of the step, step 603 takes a time-series characteristic determination unit as an execution object of the step, step 604 takes a second optimal item selection unit as an execution object of the step, and step 605 takes a model parameter calculation unit as an execution object of the step.
In the embodiment of the invention, in order to increase the learning effect of the model on the time dimension, the visual time representation with strong robustness is provided by adding the top-down feedback and the transverse connection to the bottom-up feedforward connection; the temporal image content correlation analysis (temporal characteristics) is performed by using an LSTM layer, the image content characteristics are extracted (spatial characteristics) by using a convolutional layer, and the LSTM layer and a CNN layer are combined to form a ConvLSTM layer by using a keras deep learning framework, so that the temporal-spatial characteristics can be simultaneously utilized. The ConvLSTM kernel is essentially the same as LSTM, taking the output of the previous layer as the input of the next layer, but its input transformation and circular transformation are implemented by convolution. The difference from CNN is that after the LSTM is added with the convolution operation, not only the timing relationship can be obtained, but also the features can be extracted like a convolutional layer, the spatial features can be extracted, and the switching between states is also changed into the convolution calculation. In the ConvLSTM model, a layer is also required to be applied to each input time slice, each time sequence of the time dimension is independently subjected to convolution operation to extract features, and the features are realized by a TimeDistributed wrapper provided by a keras; the method is characterized in that unidirectional LSTM is expanded, learning parameters are added during forward propagation, information extraction features of a later sequence (future) are utilized, and a Bidirectional wrapper is used for realizing the method.
In the embodiment of the invention, for the collected monitoring video, on one hand, the brightness and the chromatic value of the pixels in the two adjacent frames of images are relatively close, and because the frosting is an image gradual change process which is as long as tens of seconds or even hundreds of seconds in the video, the condition that the picture content is greatly different due to short-time mutation can not occur, the early warning of the frosting can be carried out by a frame memory prediction method, namely a convolutional neural network model, at the early stage of the frosting; on the other hand, when the environment suddenly interferes, the influence of the environmental interference factor can be preliminarily eliminated through the image super-resolution processing, and for the interference which cannot be eliminated, the frame memory prediction technology, namely the ConvLSTM model, is also needed to carry out frosting condition diagnosis under the interference condition. Due to the fact that the deep learning network lacks time variables existing in video streams and smooth transition of scenes in videos cannot be achieved; thus, feed-forward deep neural networks that use data from a large number of static images and labels for supervised training are not suitable for frost diagnosis of the condenser region in a refrigeration unit.
In summary, the invention collects the image of the air outlet condenser area by adopting a camera video monitoring mode, and performs frosting area segmentation and image super-resolution enhancement on the image quality by a video image frame-by-frame analysis method, thereby realizing an accurate method for judging the frosting condition of the air outlet condenser of the refrigerator system. Specifically, when environmental interference exists in the image, the frosting condition of the refrigerating machine is predicted by constructing a convolutional neural network model; when the image is not interfered, the frosting condition of the refrigerator is predicted by constructing the ConvLSTM model, the defects that monitoring equipment is easily influenced by the environment of a refrigeration house, the definition of image quality is not enough, video information is difficult to extract and the like are overcome, and the purpose of real-time and accurate monitoring of the working state of the refrigerator is achieved at low cost.
In addition, the convolutional neural network group model based on deep learning can effectively process video time sequence data, accurate real-time frosting diagnosis and analysis can be realized through video image content change analysis in a small range of time and by combining with real-time analysis information, and a timely and accurate intelligent early warning function is provided for defrosting and manual intervention of an air outlet condenser.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A refrigerator frosting monitoring method based on camera video frame-by-frame analysis is characterized by comprising the following steps:
step 1, acquiring video data of an area where a condenser in a refrigerator is located;
step 2, separating a condenser area image from the video data according to the video data;
step 3, determining whether the separated condenser area image needs to be preprocessed, if so, entering step 4, and if not, entering step 6;
step 4, judging whether the separated condenser area image needs image quality enhancement or not, if so, performing super-resolution image enhancement on the separated condenser area image, and then entering step 5, otherwise, entering step 6;
step 5, constructing a convolutional neural network model according to the processed image data of the condenser area, and then predicting the frosting condition of the refrigerating machine in real time by using the constructed convolutional neural network model;
and 6, constructing a ConvLSTM model according to the separated condenser area image, and then predicting the frosting condition of the refrigerating machine in real time by using the constructed ConvLSTM model.
2. A refrigerator frost monitoring method based on camera video frame-by-frame analysis according to claim 1, wherein in step 2, the step of separating the image of the condenser area from the video data comprises:
step 201, constructing a running environment of a segmented convolutional neural network model;
step 202, performing convolution calculation on an input image to realize down-sampling of the image, then performing multi-scale feature fusion calculation, and inputting the calculated abstract features into the next convolution layer;
and 203, performing transposition convolution calculation on the input of the next convolution layer to realize up-sampling of the abstract features, reducing the size of the abstract features to the size of the original image, and reserving the data area of the abstract features so as to partition the condenser area image.
3. A refrigerator frosting monitoring method based on camera video frame-by-frame analysis according to claim 1, wherein in step 3, the step of determining whether the separated condenser area image needs to be preprocessed comprises:
step 301, calculating noise points existing in an image, wherein the noise points include but are not limited to noise points formed by Gaussian noise and impulse noise;
step 302, counting the area of the noise image according to the noise points, and judging whether the area of the noise image is within a preset threshold range; if so, preprocessing is required, otherwise, preprocessing is not required.
4. A refrigerator frosting monitoring method based on camera video frame-by-frame analysis according to claim 1, wherein in step 4, the step of determining whether the separated condenser area image needs image enhancement comprises: and counting the average pixel value of the separated condenser area image, and judging whether the average pixel value is within a preset pixel average value range, if so, performing image enhancement, otherwise, not performing image enhancement.
5. The refrigerator frosting monitoring method based on camera video frame-by-frame analysis according to claim 1, wherein in step 4, the step of performing super-resolution image enhancement on the separated condenser area image comprises:
step 401, constructing a segmented convolutional neural network model operating environment;
step 402, extracting target image features by setting a convolution layer in a deep learning model, and constructing a target feature set;
step 403, calculating the distance between the target feature set and the generated feature set in a high-dimensional feature space through a perceptual loss function, calculating an optimal optimization gradient by using a back propagation algorithm, and optimizing the target image features;
in step 404, the optimized features are up-sampled and calculated, and the resolution enhanced image is output.
6. A refrigerator frosting monitoring method based on camera video frame-by-frame analysis according to claim 1, wherein in step 5, the step of constructing a convolutional neural network model from the processed condenser region image data comprises:
step 501, constructing a convolutional neural network model with a sequential structure, wherein the convolutional neural network model comprises 2 groups of convolutional modules, and each convolutional module comprises 2 convolutional layers and 1 pooling sampling layer;
502, selecting the optimal items of a model initialization method, an activation function and an optimizer;
step 503, calculating and determining the optimal convolution kernel number.
7. A refrigerator frosting monitoring method based on camera video frame-by-frame analysis according to claim 1, wherein in step 6, the step of constructing the ConvLSTM model from the separated condenser region image includes:
step 601, constructing a ConvLSTM model with a sequential structure, and setting a ConvLSTM functional layer in the model;
step 602, calculating the convolution operation of the image under each time sequence by using a TimeDistributed wrapper;
step 603, calculating the logic relation of the image in the time sequence direction by using a Bidirectional wrapper of Bidirectional, and extracting the time sequence characteristics of the image data;
step 604, selecting the optimal items of the model initialization method, the activation function and the optimizer;
step 605, calculating and determining other model parameters.
8. The refrigerator frost monitoring system based on camera video frame-by-frame analysis of claim 1, comprising
The video data acquisition module is used for acquiring video data of an area where a condenser in the refrigerator is located;
the image segmentation module is used for separating a condenser area image from the video data according to the video data;
the image preprocessing module is used for determining whether the separated condenser area image needs preprocessing or not, and if so, preprocessing is carried out;
the image quality enhancement module is used for judging whether the separated condenser area image needs image quality enhancement or not, and if so, performing super-resolution image enhancement on the separated condenser area image;
the first prediction module is used for constructing a convolutional neural network model for the image data of the condenser area subjected to super-resolution image enhancement by the image quality enhancement module and then predicting the frosting condition of the refrigerating machine in real time by utilizing the constructed convolutional neural network model;
and the second prediction module is used for constructing a ConvLSTM model for the condenser area image which does not need to be preprocessed and/or the image quality enhancement module, and then predicting the frosting condition of the refrigerating machine in real time by utilizing the constructed ConvLSTM model.
9. The refrigerator frost monitoring system based on camera video frame-by-frame analysis of claim 1, wherein the first prediction module comprises
The convolutional neural network model building unit is used for building a convolutional neural network module with a sequential structure, wherein the convolutional neural network model comprises 2 groups of convolutional modules, and each convolutional module comprises 2 convolutional layers and 1 pooling sampling layer;
the first optimal item selection unit is used for selecting optimal items of the model initialization method, the activation function and the optimizer;
and the convolution kernel number calculation unit is used for calculating and determining the optimal convolution kernel number.
10. The refrigerator frost monitoring system based on camera video frame-by-frame analysis of claim 1, wherein the second prediction module comprises
The ConvLSTM model building unit is used for building a ConvLSTM model of a sequential structure and setting a ConvLSTM functional layer in the model;
the convolution operation calculating unit is used for calculating the convolution operation of the image under each time sequence through the TimeDistributed wrapper;
the time sequence characteristic determining unit is used for calculating the logic relation of the image in the time sequence direction by using the Bidirectional wrapper and extracting the time sequence characteristic of the image data;
the second optimal item selection unit is used for selecting optimal items of the model initialization method, the activation function and the optimizer;
and the model parameter calculation unit is used for calculating and determining other model parameters.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110727931.1A CN113536989A (en) | 2021-06-29 | 2021-06-29 | Refrigerator frosting monitoring method and system based on camera video frame-by-frame analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110727931.1A CN113536989A (en) | 2021-06-29 | 2021-06-29 | Refrigerator frosting monitoring method and system based on camera video frame-by-frame analysis |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113536989A true CN113536989A (en) | 2021-10-22 |
Family
ID=78097155
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110727931.1A Pending CN113536989A (en) | 2021-06-29 | 2021-06-29 | Refrigerator frosting monitoring method and system based on camera video frame-by-frame analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113536989A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116989510A (en) * | 2023-09-28 | 2023-11-03 | 广州冰泉制冷设备有限责任公司 | Intelligent refrigeration method combining frosting detection and hot gas defrosting |
-
2021
- 2021-06-29 CN CN202110727931.1A patent/CN113536989A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116989510A (en) * | 2023-09-28 | 2023-11-03 | 广州冰泉制冷设备有限责任公司 | Intelligent refrigeration method combining frosting detection and hot gas defrosting |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10345046B2 (en) | Fault diagnosis device based on common information and special information of running video information for electric-arc furnace and method thereof | |
| Trebing et al. | Wind speed prediction using multidimensional convolutional neural networks | |
| CN114484731B (en) | Central air conditioner fault diagnosis method and device based on stacking fusion algorithm | |
| CN113449660B (en) | Abnormal event detection method of space-time variation self-coding network based on self-attention enhancement | |
| CN113435593B (en) | Refrigeration equipment frosting prediction method based on sensor time sequence data analysis | |
| CN114120093B (en) | Coal gangue target detection method based on improved YOLOv algorithm | |
| Chen et al. | Unsupervised curriculum domain adaptation for no-reference video quality assessment | |
| CN115358281B (en) | Machine learning-based cold and hot all-in-one machine monitoring control method and system | |
| CN111339892B (en) | Swimming pool drowning detection method based on end-to-end 3D convolutional neural network | |
| CN111191610A (en) | People flow detection and processing method in video monitoring | |
| CN112598144A (en) | CNN-LSTM burst fault early warning method based on correlation analysis | |
| CN113536989A (en) | Refrigerator frosting monitoring method and system based on camera video frame-by-frame analysis | |
| CN114821155A (en) | Multi-label classification method and system based on deformable NTS-NET neural network | |
| CN117349583A (en) | Intelligent detection method and system for low-temperature liquid storage tank | |
| CN116977807A (en) | Multi-sensor fusion-based intelligent monitoring system and method for refrigerator | |
| Daogang et al. | Anomaly identification of critical power plant facilities based on YOLOX-CBAM | |
| CN117541534A (en) | Power transmission line inspection method based on unmanned plane and CNN-BiLSTM model | |
| Deng et al. | Intelligent prediction of rolling bearing remaining useful life based on probabilistic DeepAR-Transformer model | |
| CN114973102A (en) | Video anomaly detection method based on multipath attention time sequence | |
| CN113591982A (en) | Performance evaluation method and system of fault detection and diagnosis algorithm | |
| CN117579513B (en) | Visual operation and maintenance system and method for convergence and diversion equipment | |
| CN113312968B (en) | Real abnormality detection method in monitoring video | |
| CN117676099B (en) | Security early warning method and system based on Internet of things | |
| SANGAM et al. | Rotten Fruit Detection Using Artificial Intelligence | |
| Xu et al. | A multi-component chiller status prediction method using e-lstm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |