CN107527060B - Refrigerating device storage management system and refrigerating device - Google Patents
Refrigerating device storage management system and refrigerating device Download PDFInfo
- Publication number
- CN107527060B CN107527060B CN201610442232.1A CN201610442232A CN107527060B CN 107527060 B CN107527060 B CN 107527060B CN 201610442232 A CN201610442232 A CN 201610442232A CN 107527060 B CN107527060 B CN 107527060B
- Authority
- CN
- China
- Prior art keywords
- module
- target storage
- neural network
- storage object
- refrigerating device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000005057 refrigeration Methods 0.000 claims abstract description 49
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 42
- 238000012360 testing method Methods 0.000 claims abstract description 40
- 238000001514 detection method Methods 0.000 claims abstract description 33
- 238000012549 training Methods 0.000 claims abstract description 30
- 238000005070 sampling Methods 0.000 claims abstract description 15
- 230000006870 function Effects 0.000 claims abstract description 14
- 230000003068 static effect Effects 0.000 claims description 33
- 238000013528 artificial neural network Methods 0.000 claims description 22
- 238000012545 processing Methods 0.000 claims description 15
- 238000012795 verification Methods 0.000 claims description 15
- 238000010586 diagram Methods 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 8
- 239000013598 vector Substances 0.000 claims description 8
- 238000011156 evaluation Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 5
- 238000002372 labelling Methods 0.000 claims description 3
- 239000011232 storage material Substances 0.000 claims description 3
- 230000003247 decreasing effect Effects 0.000 claims description 2
- 230000008859 change Effects 0.000 abstract description 16
- 230000003993 interaction Effects 0.000 abstract description 6
- 235000013305 food Nutrition 0.000 description 17
- 239000000463 material Substances 0.000 description 16
- 238000011176 pooling Methods 0.000 description 13
- 238000000034 method Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 10
- 230000009471 action Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 5
- 239000000284 extract Substances 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000002123 temporal effect Effects 0.000 description 3
- 238000000354 decomposition reaction Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 241000894006 Bacteria Species 0.000 description 1
- 206010012735 Diarrhoea Diseases 0.000 description 1
- 206010028813 Nausea Diseases 0.000 description 1
- 206010047700 Vomiting Diseases 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000009432 framing Methods 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000003706 image smoothing Methods 0.000 description 1
- 230000008693 nausea Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000008673 vomiting Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Business, Economics & Management (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Economics (AREA)
- Social Psychology (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Image Analysis (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
The invention provides a refrigerating device storage management system and a refrigerating device, comprising: the training module based on the convolutional neural network is used for learning and detecting the target storage object; the identification module is based on a convolutional neural network and is used for identifying whether a target storage object exists or not and the type of the target storage object; and the detection module is used for capturing and detecting whether the target storage object exists or not and whether the target storage object is stored in or taken out of the refrigerating device or not. The storage management system of the refrigeration device provided by the invention can establish the convolutional neural network through the steps of training, verifying, testing and the like so as to realize automatic learning, interaction, identification and statistics of the target storage objects in the refrigeration device, the types of the target storage objects and the change of the quantity of each target storage object according to the sampling video without changing the use habit of the traditional refrigeration device, realize intelligent automatic statistics and interaction functions, and has the advantages of high management precision, accurate statistical data and good use flexibility.
Description
Technical Field
The invention relates to the technical field of refrigeration equipment, in particular to a refrigerating device storage management system and a refrigerating device.
Background
Many people believe that food is safe to place in a freezer and does not go stale or go bad. In fact, the refrigerating device only reduces the temperature to inhibit the propagation speed of bacteria. However, food is stored for a long time, and is deteriorated, and nausea, vomiting, diarrhea and the like may occur after the food is eaten. The household refrigerating device is not generally provided with the function of counting food materials, the storage period of food is determined through user experience, and the food deterioration caused by omission is easy to occur, so that the body health is influenced. For the large-scale refrigeration and storage industry, food material statistics needs the responsibility of specially-assigned persons, and the cost is high. Once missing, the food in batches can be deteriorated and destroyed, and high economic loss is caused.
In order to solve the above problems, a management system for food materials in a refrigerator is proposed in the prior art, and as disclosed in the invention patent (application No. 2014106605313), when it is detected that a door of the refrigerator is opened, voice information input by a user is received. The voice information includes basic food material changing information corresponding to the changing operation of the user on the food material in the refrigerating device. The refrigeration device identifies voice information input by a user, carries out preprocessing, generates change information corresponding to the change operation of the user on the food materials in the refrigeration device, and transmits the change information to the terminal, so that the terminal generates food material management information after the food materials in the refrigeration device are changed. Not only can be seen that, in the above technical solution, in order to count the information of the food material in the refrigeration device, a step of voice input must be added, which actually makes the whole operation more complicated and does not conform to the habit of using the refrigeration device in daily life. If the input voice information is accidentally forgotten, the accuracy of the statistical information is greatly reduced.
In summary, the food material management system in the refrigeration device in the prior art has the problems that the use habit of the user is not met and the accuracy of statistical information is low.
Disclosure of Invention
The invention provides a management system for stored materials of a refrigerating device, aiming at overcoming the defects that the stored material statistics in the prior art does not conform to the traditional use habit and the management cost is high. The specific technical scheme provided by the invention comprises the following steps:
a refrigeration unit storage management system, comprising:
the training module based on the convolutional neural network is used for learning and detecting the target storage object;
the identification module is based on a convolutional neural network and is used for identifying whether a target storage object exists or not and the type of the target storage object;
the detection module is based on a convolutional neural network and is used for capturing and detecting whether a target storage object exists or not and whether the target storage object is stored in the refrigerating device or taken out of the refrigerating device or not;
further, the training module comprises:
a database for storing a still picture of a target storage;
the processing module is used for distinguishing the types of the storage objects on the static pictures in the database and labeling the storage objects according to different types to form a labeled original image;
and the first convolution neural network is used for receiving the marked original image output by the processing module, and extracting the characteristics, positioning and classifying in the marked original image to learn the detection target storage object.
Further, the first convolutional neural network comprises:
the characteristic extraction layer is used for extracting pixel values of a marked detection area on the marked original image and extracting characteristics to obtain a characteristic diagram of the detection area;
the characteristic sampling layer is used for generating a low-dimensional vector by sliding a window through the characteristic map of the detection area;
a feature mapping layer for mapping the low-dimensional vectors to a fully connected layer;
and the full connection layer comprises a regression layer for positioning and a classification layer for classification, and the full connection layer is used for outputting a result and determining whether the target storage object is detected.
Further, the identification module comprises:
the optimization module is used for adjusting the hyper-parameters of the first convolution network according to the loss function curve, the error rate curve and the learning curve generated by the training module;
a verification image database for storing verification images;
the verification module is used for inputting a verification image to the optimized first convolution neural network and obtaining an optimized first convolution neural network;
and the test module is used for processing the video into a single frame, inputting the single frame as a test image into the optimized first convolution neural network for identification, determining whether a target storage object exists and the type of the target storage object, outputting an identification result and obtaining an optimized network model at the same time.
Preferably, the hyper-parameters include a learning rate, a regularization term coefficient, and a number of layers of the convolutional neural network.
Further, the detection module comprises:
the input module is used for inputting the video sets and the static images at the inlet of the refrigerating device and in the refrigerating device;
the detection module inputs the static and motion components of the video set and the static images to the optimized network model to detect whether a target storage item is stored in or removed from the refrigeration unit.
The system further comprises an estimation module, a storage module and a control module, wherein the estimation module is used for determining the number of the target storage objects stored in or taken out of the refrigerating device; the estimation module comprises:
a first estimation module for estimating a profile area of a target storage item for storage or retrieval based on a refrigeration unit portal video set and an output of the optimized network model;
a second estimation module for estimating again the contour area of the stored or taken out target storage object according to the static image,
the calibration module is used for determining the quantity of the stored materials according to the comparison of the output of the first estimation module and the output of the second estimation module;
when the output result of the optimized network model confirms that the target storage object is stored or taken out, a first estimation module is used for estimating the contour area of the target storage object according to the video set at the inlet of the refrigerating device and taking the contour area as a standard value; the second estimation module is used for estimating the contour area of the target storage object again according to the static image in the refrigerating device to serve as a test value; the calibration module is used for comparing the test value with the standard value to determine the quantity of the storage materials.
Further, the device also comprises a statistic module which is used for outputting the type and the quantity of the target storage objects in the refrigerating device according to the output value of the estimation module; the statistical module is used for recording the types of the target storage objects and increasing or decreasing the number of the target storage objects when the test value is equal to the standard value.
Further, the device also comprises a display module, wherein the display module receives the output result of the statistic module and generates a display value.
The storage management system of the refrigeration device provided by the invention can establish the convolutional neural network through a plurality of steps of training, verifying, testing and the like so as to realize automatic learning, interaction, identification and statistics of the target storage objects in the refrigeration device, the types of the target storage objects and the change of the quantity of each target storage object according to the sampling video without changing the use habit of the traditional refrigeration device, realize intelligent automatic statistics and interaction functions, and has the advantages of high management precision, accurate statistical data and good use flexibility.
The invention also discloses a refrigerating device which comprises a refrigerating device storage management system. The refrigeration unit storage management system includes:
the training module based on the convolutional neural network is used for learning and detecting the target storage object;
the identification module is based on a convolutional neural network and is used for identifying whether a target storage object exists or not and the type of the target storage object;
the detection module is based on a convolutional neural network and is used for capturing and detecting whether a target storage object exists or not and whether the target storage object is stored in the refrigerating device or taken out of the refrigerating device or not;
an evaluation module for determining the amount of the target storage item stored in or removed from the refrigeration device; and
and the counting module is used for outputting the type and the quantity of the target storage objects in the refrigerating device according to the output value of the estimation module.
The refrigerating device disclosed by the invention has the functions of automatically identifying and interactively counting stored objects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow diagram of a refrigeration unit storage management system in accordance with the present invention;
FIG. 2 is a schematic diagram of an embodiment of a first convolutional neural network in a training module;
FIG. 3 is a flow diagram of a first embodiment of a refrigeration unit storage management system;
FIG. 4 is a flow diagram of a second embodiment of a refrigeration unit storage management system;
FIG. 5 is an example of a loss function curve in an identification module;
FIG. 6 is an example of an error rate curve in an identification module;
FIG. 7 is an example of a learning curve in an identification module.
Detailed Description
To make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific embodiments of the present invention and the accompanying drawings. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a flow chart illustrating an embodiment of a refrigeration storage management system according to the present invention. In this embodiment, the management system for the stored materials of the refrigeration device comprises a training module, an identification module, a detection module, an estimation module and a statistic module, wherein the training module, the identification module and the detection module are all realized based on a convolutional neural network, and the management system further comprises a display module for displaying the detection value of the statistic module in real time. As shown in fig. 3, the storage management system of the refrigeration device provided in the present embodiment specifically includes:
the training module learns to detect the target stores. In this embodiment, the target storage object is learned and detected, that is, the first convolutional neural network is learned and distinguished to be the target storage object. In the case of a cold storage device, the target storage material may be a food product, a medicine, or other items requiring cold storage, such as laboratory preparations, specimens, etc. Firstly, a database for storing static pictures of a target storage object is established, the database comprises a large number of pictures to form a training set, and the order of magnitude of the static pictures in the training set can reach one hundred thousand or even higher. The static picture comprises images of a plurality of target storage objects which are mutually shielded and distinguish each part.
The processing module processes the static pictures in the database, and specifically, the processing comprises dividing a plurality of rectangular frames on one static picture, and framing a target storage object in each rectangular frame. Classifying the framed target storage objects, labeling according to different categories, adding labels, and making labeled files aiming at different categories on each static picture to form labeled original images. The number of still pictures in the database for each type of target storage object is on average over 500 sheets to prevent overfitting. The selected area is regarded as a detection area in the training module, and the labeled original image is preferably preprocessed before being input into the first convolutional neural network to eliminate some interference factors in the image, wherein specific preprocessing methods include but are not limited to gray scale transformation, histogram modification, image smoothing and denoising and the like.
The first convolutional neural network is the core of the training module and the whole storage management system. Referring to fig. 2, the first convolutional neural network includes a feature extraction layer L1. The feature extraction layer L1 extracts features from pixel values of a frame region, i.e., a detection region, in the labeled original image, and the feature extraction layer L1 converts the pixel values of the detection region into data of a plurality of channels. Information for each channel is obtained independently. The number of channels may be plural. And performing convolution pooling on each channel to obtain a feature map of a frame selection area, namely a frame selection feature map for short, wherein the feature sampling layer L2 respectively slides through the frame selection feature map one by using a window, namely, the frame selection feature map is sampled one by using a convolution kernel sampling mode to obtain a low-dimensional vector. The feature mapping layer L3 maps the low-dimensional vectors to the fully-connected layer. The fully-connected layer comprises a regression layer L41 for positioning and a classification layer L42 for classification, so that the first convolution neural network learns the position of the target stock on the labeled original image and the type of the target stock through interaction. And the full connection layer outputs a result and determines whether the target storage object is detected.
The first convolution neural network is trained and adjusted, and the identification module of the management system is used for enabling the first convolution neural network to learn and identify whether the target storage object exists in the input image or not and the type of the target storage object. The identification module comprises an optimization module, and the optimization module adjusts the hyper-parameters of the first convolution network according to the loss function curve, the error rate curve and the learning curve generated by the training module. Specifically, the learning rate in the hyper-parameters is adjusted by a loss function curve, and the loss function generated by the training module has various forms. As shown in fig. 5, the loss function curve oscillates in the first mode, indicating that the learning rate is too high, and the falling rate is too low in the second mode, indicating that the learning rate is too low. The regularization term coefficients can also be adjusted by an error rate curve, as shown in fig. 6, a Train curve represents the training error rate, and a vali curve represents the validation error rate. In the training module, the setting value of the regular term coefficient is usually 1, and the value of the regular term coefficient can be adjusted according to a train curve and a vali curve generated after training. And adjusting the size of the first convolutional neural network, namely the number of layers of the first convolutional neural network according to the learning curve, wherein the direction of an arrow in the figure represents the trend of timely layer change. As shown in fig. 7, the number of verification images and training time may also be adjusted according to the learning curve. The first convolution neural network adjusted according to the loss function curve, the error rate curve and the learning curve is obviously superior to the first convolution neural network before training. The adjustment of the hyper-parameters is not limited to the three items, and other hyper-parameters may be adjusted according to the training result.
The verification module inputs the images in the verification image database to the optimized first convolution neural network, and the first convolution neural network identifies whether the verification images input by the verification image database have the target storage object and the type of the target storage object and outputs the result. The verification image is similar to the real image needing to be identified and is not processed by the processing module any more. And adjusting the hyper-parameters for the second time according to the output result to obtain the optimized first convolution neural network.
The test module tests the optimized first convolutional neural network. Processing the collected videos at the inlet of the refrigerating equipment and in the refrigerating equipment into independent frames, inputting the independent frames into an optimized first convolution neural network for identification as test images in a frame mode, determining whether a target storage object exists or not and the type of the target storage object, and outputting an identification result. And storing the recognition result in a video format or a text format, analyzing again, and optimizing the first convolution neural network again by using the analysis result to obtain an optimized network model.
And managing the quantity and the type of the stored materials in the refrigerating device through the trained, verified and tested refrigerating device stored material management system. Specifically, the change in the number of the target stored objects is mainly achieved by putting in and taking out the target stored objects. The shape and the form of the target storage object can not be changed excessively in the refrigerating or freezing process, so that the change caused by the deformation can be ignored in the image interaction processing in the storage object management system, and the most key difficulty of the refrigerating device storage object management system after training, verification and testing is that the target storage objects are mutually shielded in the storage process. Therefore, accurate recognition statistics are achieved in the following manner.
A camera device for taking a video is provided at the entrance of the refrigeration device. The camera device can be static or can shake along with the movement of the inlet of the refrigerating device, and the camera device collects videos at the inlet of the refrigerating device, and the visual field range of the camera device covers the whole inlet and continuous scenes near the inlet. For household refrigeration equipment, the camera device mainly collects the action state that the hand is inserted into or pulled out of the inlet of the refrigeration device. In the case of a large refrigerator, the video captured by the camera device includes the movement of a person entering or exiting the refrigerator, the movement of a hand when the person enters or exits, and a target storage object in the hand. The camera device can be arranged on the refrigerating device and also can be arranged on a fixing structure near the refrigerating device, so that the working stability is ensured. The interior of the refrigerating device is also provided with a camera device for shooting the storage state of the target storage object in the refrigerating device, such as a static scene on a shelf in the refrigerating device. The camera device can be realized by a plurality of independent cameras, and can also realize the image acquisition at the inlet of the refrigerating device and in the refrigerating device through one camera arranged at the inlet of the refrigerating device or on the door body.
The camera device stores the collected images into a video set, and inputs the video set at the inlet of the refrigerating device and the video set in the refrigerating device into a video input module of the storage management system. In order to respectively identify whether the action is to put in or take out the target storage object and the type and the number of the target storage objects, the image decomposition module divides each frame in the video file in the video set into a static component and a motion component. Different from unpredictable moving objects and moving modes in human body movement recognition in the prior art, video image processing at the inlet of a refrigerating device and in the refrigerating device has a relatively fixed detection area, a detection area background and a relatively stable moving mode, so that a recognition mode with higher recognition precision and higher processing speed is required to realize accurate management statistics.
In this embodiment, the image decomposition module decomposes and processes the static component and the motion component of the continuous frames to form a spatial stream and a temporal stream. The static component is a segmented static video frame, and the motion component is an image with a velocity vector reflecting the behavior of the target, such as an optical flow density map. In order to improve the precision and speed of recognition processing, the optimization network model adopts a first convolutional neural network A and a second convolutional neural network B which are independently arranged and are trained, verified and tested to respectively process space flow and time flow. The super-parameters of the first convolutional neural network A and the second convolutional neural network B are optimal values formed through three steps of training, verifying and testing. The first convolutional neural network A and the second convolutional neural network B are preferably of the same scale, and the hyper-parameters may be set according to specific needs and are slightly different.
Specifically, the first convolutional neural network a further includes an input layer, a multi-layer convolutional pooling layer, and a classifier, wherein the input layer is used for inputting the spatial stream. Since the spatial stream is composed of consecutive still video frames, the background in the still video frames is relatively stable in practice, and the foreground object changes more obviously. Thus, the static component input through the input layer can be represented by the pixel values in the detection area. And extracting effective pixel values in the detection area by the multilayer convolution pooling layer in the first convolution neural network A, extracting characteristics and forming a characteristic diagram of the detection area. And selecting a convolution kernel or a window on the feature map, performing convolution by using the convolution kernel, and then further performing pooling. A preferred mode is to use multiple convolution kernels to perform convolution, that is, static components are converted into data of multiple channels, each channel data is independently acquired and is convolved by a convolution kernel, wherein the learned features of each layer of convolution are local, but the features after multiple layers of convolution are close to global, so that better accuracy is achieved. And inputting the output result output by the multilayer convolution pooling layer into a classifier for further learning analysis. The classifier preferably comprises a fully connected layer and a Softmax layer, so that the first convolutional neural network a learns how to determine whether there is a target storage in the spatial stream generated from the video set.
Correspondingly, the second convolutional neural network B also further comprises an input layer, a multi-layer convolutional pooling layer and a classifier, wherein the input layer is used for inputting the time stream. The time flow is composed of an optical flow density map, and the characteristics and the velocity vector of the time flow can also be represented by pixel values in a detection area. And extracting the characteristics of the dynamic components in the detection area in the optical flow density graph by using the multilayer convolution pooling layer in the second convolution neural network B, selecting a convolution kernel or a window on the characteristic graph, performing convolution by using the convolution kernel, and then performing pooling. The dynamic component is also converted into data of a plurality of channels, and each channel data is independently obtained and is respectively convolved by a convolution kernel. And inputting the output result output by the multilayer convolution pooling layer into a classifier for further learning analysis. The classifier preferably comprises a fully connected layer and a Softmax layer, so that the second convolutional neural network B can learn how to determine whether there is an action to take or store the target storage from or into the cold storage device.
The output results of the first convolutional neural network A and the second convolutional neural network B are input into the same classifier, preferably SVM, and then the output results of the optimized network model can accurately determine whether the target storage object, the type of the target storage object and whether the target storage object is stored in the refrigerating device or taken out of the refrigerating device.
In many cases, the number of target storage items put into the refrigerator at a time is different, which may cause a deviation if the output result of the direct statistical detection module is deviated, and therefore, an estimation module is further provided in the management system. The function of the evaluation module is primarily to determine the amount of the target storage product to be stored in or removed from the refrigeration device. The estimation module comprises a first estimation module and a second estimation module, when the output result of the detection module determines that the target storage object, the type of the target storage object and the target storage object are placed in the refrigerating device in a single pick-and-place action, the estimation module firstly estimates the contour area of the storage object for the first time according to the space flow in the dynamic scene video set at the inlet of the refrigerating device and uses the contour area as a standard value A1At this time, the standard value A is a value at which the target stored object in the default hand motion is not mutually hidden by different types of target stored objects1The accuracy of (2) is higher. The second estimation module continuously receives the static image in the refrigerating device, generates a change value when the outline area of the target storage object in the static image changes, and takes the change value as a test value A of the target storage object stored in the refrigerating device2. The estimation module is also provided with a calibration module which compares the standard value with the test value, if the test value is not equal to the standard value, the second estimation module generates the test value again until the standard value is equal to the test value, and the quantity of the target storage object stored in the refrigeration device is determined.
Similarly, when the target storage object is taken out of the refrigerating device, the first estimation module is used for estimating the outline area of the taken-out target storage object according to the space flow of the dynamic scene video set at the inlet of the refrigerating device and taking the outline area as a standard value. The second estimation module estimates the change value of the contour area of the target storage object of the static image in the refrigerating device again according to the static scene in the refrigerating device, and the change value is used as the test value of the target storage object taken out of the refrigerating device. The calibration module compares the standard value to the test value. If the test value is not equal to the standard value, the second evaluation module again generates the test value until the standard value is equal to the test value, and determines the quantity of the target storage object taken out of the refrigeration device.
The estimation module outputs the output result to the statistic module. And the statistical module records the types of the target storage objects and increases or decreases the number of the target storage objects when the test value is equal to the standard value. The output result of the statistical module can be directly output to the display module. The display module receives the output result and generates a display value according to the use habit, wherein the display value can comprise information such as the type, storage period and quantity of the target object. The display generated by the display module is displayed through a display screen, and the display screen can be arranged on the refrigerating device or can be arranged on other terminals communicated with the refrigerating device, so that the type of the target storage object in the refrigerating device and the quantity of each type of target storage object can be known and inquired through the display device anytime and anywhere. The whole process does not need to change the traditional use habit, realizes automatic statistics, automatic judgment and automatic display, effectively simplifies the process of the storage statistics and reduces the use cost of the detection statistics.
Referring to fig. 4, a schematic structural diagram of a second embodiment of the refrigerator storage management system according to the present disclosure is shown, in which the basic implementation of the training module and the recognition module is substantially the same as that of the first embodiment. Please refer to the detailed description of the first embodiment. In this embodiment, the optimized network model is a 3D convolutional neural network. The 3D convolutional neural network provided in this embodiment extracts temporal and spatial features in a video set composed of dynamic videos at the entrance of the refrigeration apparatus input by the test module through a 3D convolutional kernel, and processes in both temporal and spatial dimensions.
Specifically, the 3D convolutional neural network specifically includes an original processing layer, a feature extraction layer, a spatio-temporal convolutional layer, a feature sampling layer, and a classifier. Wherein the original processing layer composes a plurality of successive original frames in the video set into a convolution cube. The feature extraction layer extracts a plurality of channel information for each original frame. In order to realize the sampling of two dimensions of space and time, the feature extraction layer extracts five independent channel information including gray scale, X-direction gradient, Y-direction gradient, X-direction optical flow and Y-direction optical flow. The space-time convolutional layer performs convolution on each channel separately. The space-time convolutional layer comprises a plurality of layers, and the output result of each space-time convolutional layer is pooled through an independent characteristic sampling layer. The preferred 3D convolution kernel for convolution is a 3 x 3 window. The pooling windows in the first feature sampling layer were windows of 1 x 2, wherein the pooling windows in the feature sampling layer were windows of 2 x 2. The selection of the pooling window is to realize an optimal sampling result, the dimensionality of the pooling window in the first feature sampling layer is 1, and the sampling is prevented from being performed too early so as to keep more image input information.
And inputting the output result of the characteristic sampling layer into a classifier for classification learning. The classifier includes two fully connected layers and a softmax classifier to simultaneously determine whether there is a target storage item and whether there is a target storage item stored in or removed from the refrigeration appliance through a 3D convolutional network.
Also, in many cases, the number of target storage items put into the refrigerator at a time is different, which may cause a deviation if the output result of the direct statistical detection module is deviated, and therefore, an estimation module is further provided in the management system. The function of the evaluation module is primarily to determine the amount of the target storage product to be stored in or removed from the refrigeration device. Specifically, the estimation module comprises a first estimation module and a second estimation module, when the output result of the 3D convolutional neural network determines that the target storage object, the type of the target storage object and the target storage object are placed in the refrigerating device in a single pick-and-place action, the estimation module firstly estimates the contour area of the storage object for the first time according to the space flow in the dynamic scene video set at the inlet of the refrigerating device and takes the contour area as a standard value A1At this time, the standard value A is a value at which the target stored object in the default hand motion is not mutually hidden by different types of target stored objects1The accuracy of (2) is higher. The second estimation module continuously receives the static image in the refrigerating device, and when the contour area of the target storage object in the static image changes, the second estimation moduleGenerating a variation value and using the variation value as a test value A of the target storage object stored in the refrigerating device2. The estimation module is also provided with a calibration module which compares the standard value with the test value, if the test value is not equal to the standard value, the second estimation module generates the test value again until the standard value is equal to the test value, and the quantity of the target storage object stored in the refrigeration device is determined.
Similarly, when the target storage object is taken out of the refrigerating device, the first estimation module is used for estimating the outline area of the taken-out target storage object according to the space flow of the dynamic scene video set at the inlet of the refrigerating device and taking the outline area as a standard value. The second estimation module estimates the change value of the contour area of the target storage object of the static image in the refrigerating device again according to the static scene in the refrigerating device, and the change value is used as the test value of the target storage object taken out of the refrigerating device. The calibration module compares the standard value to the test value. If the test value is not equal to the standard value, the second evaluation module again generates the test value until the standard value is equal to the test value, and determines the quantity of the target storage object taken out of the refrigeration device.
The estimation module outputs the output result to the statistic module. And the statistical module records the types of the target storage objects and increases or decreases the number of the target storage objects when the test value is equal to the standard value. The output result of the statistical module can be directly output to the display module. The display module receives the output result and generates a display value according to the use habit, wherein the display value can comprise information such as the type, storage period and quantity of the target object. The display generated by the display module is displayed through a display screen, and the display screen can be arranged on the refrigerating device or can be arranged on other terminals communicated with the refrigerating device, so that the type of the target storage object in the refrigerating device and the quantity of each type of target storage object can be known and inquired through the display device anytime and anywhere. The whole process does not need to change the traditional use habit, realizes automatic statistics, automatic judgment and automatic display, effectively simplifies the process of the storage statistics and reduces the use cost of the detection statistics.
The invention also provides a refrigerating device adopting the refrigerating device storage management system specifically disclosed by the embodiment. Referring to the detailed description of the first and second embodiments, the refrigeration device disclosed in the present invention has the technical effect of the refrigeration device storage management system.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (7)
1. A refrigeration unit storage management system, comprising:
the training module based on the convolutional neural network is used for learning and detecting the target storage object; the training module comprises a first convolution neural network, wherein the first convolution neural network is used for receiving an original image to be marked, extracting the characteristics, positioning and classifying in the original image to learn and detect a target storage object;
the identification module is based on a convolutional neural network and is used for identifying whether a target storage object exists or not and the type of the target storage object; the identification module comprises an optimization module, and the optimization module is used for adjusting the hyper-parameters of the first convolution neural network according to the loss function curve, the error rate curve and the learning curve generated by the training module; a verification image database for storing verification images; the verification module is used for inputting a verification image to the adjusted first convolution network and obtaining an optimized first convolution neural network; the testing module is used for processing the video into a single frame, inputting the single frame as a testing image into the optimized first convolution neural network for identification, determining whether a target storage object exists or not and the type of the target storage object, outputting an identification result and obtaining an optimized network model at the same time;
the detection module is based on a convolutional neural network and is used for capturing and detecting whether a target storage object exists or not and whether the target storage object is stored in the refrigerating device or taken out of the refrigerating device or not; the detection module comprises an input module for inputting a video set and a static image at an inlet of the refrigeration device and in the refrigeration device, and the detection module inputs static components and motion components in the video set and the static image into the optimized network model to detect whether a target storage object is stored in or taken out of the refrigeration device; and
an evaluation module for determining the amount of the target storage item stored in or removed from the refrigeration device; the estimation module comprises: a first estimation module for estimating a profile area of a target storage item for storage or retrieval based on a refrigeration unit portal video set and an output of the optimized network model; the calibration module is used for determining the number of the stored objects according to the output comparison of the first estimation module and the second estimation module; when the output result of the optimized network model confirms that the target storage object is stored or taken out, a first estimation module is used for estimating the contour area of the target storage object according to the video set at the inlet of the refrigerating device and taking the contour area as a standard value; the second estimation module is used for estimating the contour area of the target storage object again according to the static image in the refrigerating device to serve as a test value; the calibration module is used for comparing the test value with the standard value to determine the quantity of the storage materials.
2. A refrigeration unit storage management system as recited in claim 1 wherein the training module comprises:
a database for storing a still picture of a target storage;
and the processing module is used for distinguishing the types of the storage objects on the static pictures in the database, labeling the storage objects according to different types to form a labeled original image and outputting the labeled original image to the first convolution neural network.
3. A refrigeration unit storage management system as recited in claim 2 wherein the first convolutional neural network comprises:
the characteristic extraction layer is used for extracting pixel values of a marked detection area on the marked original image and extracting characteristics to obtain a characteristic diagram of the detection area;
the characteristic sampling layer is used for generating a low-dimensional vector by sliding a window through the characteristic map of the detection area;
a feature mapping layer for mapping the low-dimensional vectors to a fully connected layer;
and the full connection layer comprises a regression layer for positioning and a classification layer for classification, and the full connection layer is used for outputting a result and determining whether the target storage object is detected.
4. A refrigeration unit storage management system as recited in claim 1 wherein the hyper-parameters include learning rate, regularization term coefficient, and number of convolutional neural network layers.
5. A refrigeration unit storage management system as recited in claim 1 further comprising:
the counting module is used for outputting the type and the quantity of the target storage objects in the refrigerating device according to the output value of the estimation module;
the statistical module is used for recording the types of the target storage objects and increasing or decreasing the number of the target storage objects when the test value is equal to the standard value.
6. A refrigeration unit storage management system as recited in claim 5 further comprising a display module that receives the output of the statistics module and generates a display value.
7. A cold storage appliance comprising a cold storage appliance storage management system as claimed in any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610442232.1A CN107527060B (en) | 2016-06-20 | 2016-06-20 | Refrigerating device storage management system and refrigerating device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610442232.1A CN107527060B (en) | 2016-06-20 | 2016-06-20 | Refrigerating device storage management system and refrigerating device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107527060A CN107527060A (en) | 2017-12-29 |
| CN107527060B true CN107527060B (en) | 2022-01-25 |
Family
ID=60733806
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610442232.1A Active CN107527060B (en) | 2016-06-20 | 2016-06-20 | Refrigerating device storage management system and refrigerating device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107527060B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110795959A (en) * | 2018-08-01 | 2020-02-14 | 周维 | Intelligent identification method and device for sliced food materials |
| KR20200037657A (en) * | 2018-10-01 | 2020-04-09 | 삼성전자주식회사 | Refrigerator, server and method for recognizing object thereof |
| CN109657694A (en) * | 2018-10-26 | 2019-04-19 | 平安科技（深圳）有限公司 | Picture automatic classification method, device and computer readable storage medium |
| CN111658291B (en) * | 2020-06-11 | 2022-06-24 | 四川大学华西医院 | Circulating cold compress device is used in orthopedics nursing |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105531715A (en) * | 2013-06-26 | 2016-04-27 | 亚马逊科技公司 | Detecting item interaction and movement |
| CN105654270A (en) * | 2014-11-18 | 2016-06-08 | 博西华家用电器有限公司 | Refrigerator, terminal, and management system and management method for food materials in refrigerator |
| CN105678216A (en) * | 2015-12-21 | 2016-06-15 | 中国石油大学(华东) | Spatio-temporal data stream video behavior recognition method based on deep learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7627171B2 (en) * | 2003-07-03 | 2009-12-01 | Videoiq, Inc. | Methods and systems for detecting objects of interest in spatio-temporal signals |
-
2016
- 2016-06-20 CN CN201610442232.1A patent/CN107527060B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105531715A (en) * | 2013-06-26 | 2016-04-27 | 亚马逊科技公司 | Detecting item interaction and movement |
| CN105654270A (en) * | 2014-11-18 | 2016-06-08 | 博西华家用电器有限公司 | Refrigerator, terminal, and management system and management method for food materials in refrigerator |
| CN105678216A (en) * | 2015-12-21 | 2016-06-15 | 中国石油大学(华东) | Spatio-temporal data stream video behavior recognition method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107527060A (en) | 2017-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110532970B (en) | Age and gender attribute analysis method, system, equipment and medium for 2D images of human faces | |
| CN104732413B (en) | A kind of intelligent personalized video ads method for pushing and system | |
| CN107527060B (en) | Refrigerating device storage management system and refrigerating device | |
| KR101964397B1 (en) | Information processing apparatus and information processing method | |
| CN111340126B (en) | Article identification method, apparatus, computer device, and storage medium | |
| CN105022982B (en) | Hand motion recognition method and apparatus | |
| US7822275B2 (en) | Method for detecting water regions in video | |
| US9443144B2 (en) | Methods and systems for measuring group behavior | |
| CN107527363B (en) | Refrigerating device storage management system and refrigerating device | |
| CN108600865B (en) | A kind of video abstraction generating method based on super-pixel segmentation | |
| CN104240264B (en) | The height detection method and device of a kind of moving object | |
| CN110807434A (en) | Pedestrian re-identification system and method based on combination of human body analysis and coarse and fine particle sizes | |
| US11308348B2 (en) | Methods and systems for processing image data | |
| CN108596187B (en) | Commodity purity detection method and display cabinet | |
| CN110909672A (en) | Smoking action recognition method based on double-current convolutional neural network and SVM | |
| CN113766330A (en) | Method and device for generating recommendation information based on video | |
| CN107526991B (en) | Refrigerating device storage management system and refrigerating device | |
| CN113468914A (en) | Method, device and equipment for determining purity of commodities | |
| US20220325946A1 (en) | Selective image capture using a plurality of cameras in a refrigerator appliance | |
| CN113947770A (en) | Method for identifying object placed in different areas of intelligent cabinet | |
| WO2021091481A1 (en) | System for object identification and content quantity estimation through use of thermal and visible spectrum images | |
| Hashemi | A survey of visual attention models | |
| Wang | Recognition and Analysis of Behavior Features of School-Age Children Based on Video Image Processing. | |
| CN112818168B (en) | Method for controlling elevator media playing sequence based on video face recognition | |
| CN111488831B (en) | Food association identification method and refrigerator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20211215 Address after: 266101 Haier Road, Laoshan District, Qingdao, Qingdao, Shandong Province, No. 1 Applicant after: QINGDAO HAIER SMART TECHNOLOGY R&D Co.,Ltd. Applicant after: Haier Smart Home Co., Ltd. Address before: 266101 Haier Road, Laoshan District, Qingdao, Qingdao, Shandong Province, No. 1 Applicant before: QINGDAO HAIER SMART TECHNOLOGY R&D Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |