CN107526991B - Refrigerating device storage management system and refrigerating device - Google Patents
Refrigerating device storage management system and refrigerating device Download PDFInfo
- Publication number
- CN107526991B CN107526991B CN201610442235.5A CN201610442235A CN107526991B CN 107526991 B CN107526991 B CN 107526991B CN 201610442235 A CN201610442235 A CN 201610442235A CN 107526991 B CN107526991 B CN 107526991B
- Authority
- CN
- China
- Prior art keywords
- convolutional neural
- neural network
- target storage
- module
- refrigerating device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 81
- 230000003068 static effect Effects 0.000 claims abstract description 38
- 238000012545 processing Methods 0.000 claims abstract description 16
- 238000000354 decomposition reaction Methods 0.000 claims abstract description 7
- 230000002123 temporal effect Effects 0.000 claims abstract description 5
- 238000005057 refrigeration Methods 0.000 claims description 43
- 238000011176 pooling Methods 0.000 claims description 23
- 238000012360 testing method Methods 0.000 claims description 22
- 238000012549 training Methods 0.000 claims description 19
- 238000001514 detection method Methods 0.000 claims description 18
- 239000000463 material Substances 0.000 claims description 17
- 238000012795 verification Methods 0.000 claims description 14
- 239000013598 vector Substances 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 5
- 238000005457 optimization Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 230000003247 decreasing effect Effects 0.000 claims description 2
- 230000004927 fusion Effects 0.000 claims description 2
- 230000008859 change Effects 0.000 abstract description 13
- 230000006870 function Effects 0.000 abstract description 11
- 230000003993 interaction Effects 0.000 abstract description 5
- 235000013305 food Nutrition 0.000 description 16
- 238000000034 method Methods 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 230000009471 action Effects 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000009432 framing Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000012706 support-vector machine Methods 0.000 description 3
- 239000003814 drug Substances 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 241000894006 Bacteria Species 0.000 description 1
- 206010012735 Diarrhoea Diseases 0.000 description 1
- 206010028813 Nausea Diseases 0.000 description 1
- 206010047700 Vomiting Diseases 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000006866 deterioration Effects 0.000 description 1
- 229940079593 drug Drugs 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000003706 image smoothing Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000008693 nausea Effects 0.000 description 1
- 239000011232 storage material Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 230000008673 vomiting Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Economics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Social Psychology (AREA)
- Entrepreneurship & Innovation (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Accounting & Taxation (AREA)
- Finance (AREA)
- Development Economics (AREA)
- Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Image Analysis (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
The invention provides a refrigerating device storage management system and a refrigerating device, comprising a video input module, a storage module and a storage module, wherein the video input module is used for inputting video sets at an inlet of the refrigerating device and in the refrigerating device; the image decomposition module is used for splitting and processing a single video in the video set into a static component and a motion component and respectively forming a spatial stream and a temporal stream; and the convolutional neural network is used for determining whether the target storage object exists or not and the type of the target storage object according to the spatial stream and determining whether the target storage object is stored in or taken out or not according to the time stream. The invention utilizes the convolutional neural network to automatically identify and count the target storage objects, the types of the target storage objects and the change of the quantity of each target storage object in the refrigerating device through independent space flow and time flow in a video set without changing the use habit of the traditional refrigerating device, realizes intelligent automatic counting and interaction functions, and has the advantages of high management precision, accurate statistical data and good use flexibility.
Description
Technical Field
The invention relates to the technical field of refrigeration equipment, in particular to a refrigerating device storage management system and a refrigerating device.
Background
Many people believe that food is safe to place in a freezer and does not go stale or go bad. In fact, the refrigerating device only reduces the temperature to inhibit the propagation speed of bacteria. However, food is stored for a long time, and is deteriorated, and nausea, vomiting, diarrhea and the like may occur after the food is eaten. The household refrigerating device is not generally provided with the function of counting food materials, the storage period of food is determined through user experience, and the food deterioration caused by omission is easy to occur, so that the body health is influenced. For the large-scale refrigeration and storage industry, the statistics of stored materials such as medicines or experimental preparations needs to be more responsible for special people, and the cost is higher. Once missing, the food in batches can be deteriorated and destroyed, and high economic loss is caused.
In order to solve the above problems, a management system for food materials in a refrigerator is proposed in the prior art, and as disclosed in the invention patent (application No. 2014106605313), when it is detected that a door of the refrigerator is opened, voice information input by a user is received. The voice information includes basic food material changing information corresponding to the changing operation of the user on the food material in the refrigerating device. The refrigeration device identifies voice information input by a user, carries out preprocessing, generates change information corresponding to the change operation of the user on the food materials in the refrigeration device, and transmits the change information to the terminal, so that the terminal generates food material management information after the food materials in the refrigeration device are changed. Not only can be seen that, in the above technical solution, in order to count the information of the food material in the refrigeration device, a step of voice input must be added, which actually makes the whole operation more complicated and does not conform to the habit of using the refrigeration device in daily life. If the input voice information is accidentally forgotten, the accuracy of the statistical information is greatly reduced.
In summary, the food material management system in the refrigeration device in the prior art has the problems that the use habit of the user is not met and the accuracy of statistical information is low.
Disclosure of Invention
The invention provides a management system for stored materials of a refrigerating device, aiming at overcoming the defects that the stored material statistics in the prior art does not conform to the traditional use habit and the management cost is high. The specific technical scheme provided by the invention comprises the following steps:
a refrigeration unit storage management system, comprising:
the video input module is used for inputting video sets at an inlet of the refrigerating device and in the refrigerating device;
the image decomposition module is used for splitting and processing a single video in the video set into a static component and a motion component and respectively forming a spatial stream and a temporal stream;
and the convolutional neural network is used for determining whether the target storage object exists or not and the type of the target storage object according to the spatial stream and determining whether the target storage object is stored in or taken out or not according to the time stream.
The system further comprises a first estimation module, a second estimation module and a control module, wherein the first estimation module is used for estimating the contour area of the stored or taken target storage object according to the inlet video set of the refrigerating device and the output result of the convolutional neural network;
a second estimation module for estimating the contour area of the target storage object again according to the static scene,
the calibration module is used for determining the quantity of the stored materials according to the comparison of the output of the first estimation module and the output of the second estimation module;
the video set comprises dynamic scenes at the entrance of the refrigeration apparatus and static scenes in the refrigeration apparatus; when the convolutional neural network determines that a target storage object is stored in or taken out according to time flow, the first estimation module is used for estimating the contour area of the target storage object according to the space flow of dynamic scene splitting processing at the inlet of the refrigerating device and taking the contour area as a standard value; the second estimation module is used for estimating the contour area of the target storage object again according to the static scene in the refrigerating device as a test value; the calibration module is used for comparing the areas of the target storage objects acquired twice and determining the quantity of the storage objects.
The device further comprises a statistic module, wherein the statistic module is used for recording the types of the target storage matters and increasing or decreasing the number of the target storage matters when the test value is equal to the standard value.
Further, the convolutional neural network comprises a first convolutional neural network; the first convolutional neural network further comprises:
an input layer for inputting the spatial stream;
the multilayer convolution pooling layer is used for extracting pixel values of static components forming the space flow, preprocessing the pixel values of a detection area in the static components and extracting features to obtain a feature map;
and the classifier is used for determining and outputting whether the target storage object exists and the type of the target storage object according to the output result of the convolution pooling layer.
Further, the convolutional neural network further comprises a second convolutional neural network; the second convolutional neural network further comprises;
an input layer for inputting the time stream;
the multilayer convolution pooling layer is used for extracting the characteristics of the dynamic components forming the time flow and obtaining a characteristic map with the speed vector characteristics;
and the classifier is used for determining and outputting whether a target storage object is stored in or taken out according to the feature diagram output by the convolution pooling layer.
Further, the convolution pooling layer of the first convolutional neural network and the convolution pooling layer of the second convolutional neural network convert the static component and the dynamic component into data of a plurality of channels, and the data of each channel is acquired independently.
Further, the first convolutional neural network and the second convolutional neural network are the same in scale, and the classifier of the first convolutional neural network and the classifier of the second convolutional neural network respectively comprise a full connection layer and a Softmax layer.
Further, the outputs of the first convolutional neural network and the second convolutional neural network are input into the same classifier and fused; the classifier used for fusion is an SVM.
Furthermore, the video set is a dynamic scene of human hand actions at the entrance of the refrigerating device and a static scene on a shelf in the refrigerating device.
The storage management system of the refrigeration device can automatically identify and count the target storage objects, the types of the target storage objects and the quantity change of each target storage object in the refrigeration device through independent space flow and time flow in a video set by utilizing a convolutional neural network, and can realize intelligent automatic counting and interaction functions through calibration of an estimation module and counting of a counting module without changing the use habit of the traditional refrigeration device, and the refrigeration device has the advantages of high management precision, accurate statistical data and good use flexibility.
The invention also discloses a refrigerating device with the refrigerating device storage management system, wherein the refrigerating device is provided with the refrigerating device storage management system which comprises a video input module for inputting video sets at an inlet of the refrigerating device and in the refrigerating device; the image decomposition module is used for splitting and processing a single video in the video set into a static component and a motion component and respectively forming a spatial stream and a temporal stream; and the convolutional neural network is used for determining whether the target storage object exists or not and the type of the target storage object according to the spatial stream and determining whether the target storage object is stored in or taken out or not according to the time stream.
The refrigerating device disclosed by the invention has the functions of automatically identifying and counting stored objects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flow chart of one embodiment of a refrigeration unit storage management system in accordance with the present invention;
FIG. 2 is a flow diagram of another embodiment of a refrigeration unit storage management system according to the present invention;
FIG. 3 is an example of a loss function curve in a refrigeration unit storage management system identification module;
FIG. 4 is an example of an error rate profile in a refrigeration unit storage management system identification module;
FIG. 5 is an example of a learning curve in a refrigeration unit storage management system identification module.
Detailed Description
To make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the specific embodiments of the present invention and the accompanying drawings. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 is a flow chart illustrating an embodiment of a refrigeration storage management system according to the present invention. In this embodiment, the management system for the stored materials of the refrigeration device may include a training module, an identification module, a detection module, an estimation module, and a statistic module, wherein the training module, the identification module, and the detection module are implemented based on a convolutional neural network, and may further include a display module for displaying the detection value of the statistic module in real time. As shown in the drawings, the storage management system of the refrigeration device provided in this embodiment specifically includes:
the training module learns to detect the target stores. In this embodiment, learning to detect the target storage object allows the convolutional neural network to learn to distinguish whether the target storage object is the target storage object. In the case of a cold storage device, the target storage material may be a food product, a medicine, or other items requiring cold storage, such as laboratory preparations, specimens, etc. Firstly, a database for storing static pictures of a target storage object is established, the database comprises a large number of pictures to form a training set, and the order of magnitude of the static pictures in the training set can reach one hundred thousand or even higher. The static picture comprises images of a plurality of target storage objects which are mutually shielded and distinguish each part.
The processing module processes the static pictures in the database, and specifically, the processing comprises dividing a plurality of rectangular frames on one static picture, and framing a target storage object in each rectangular frame. Classifying the framed target storage objects, labeling according to different categories, adding labels, and making labeled files aiming at different categories on each static picture to form labeled original images. The number of still pictures in the database for each type of target storage object is on average over 500 sheets to prevent overfitting. The selected area is regarded as a detection area in the training module, and the labeled original image is preferably preprocessed before being input into the convolutional neural network to eliminate some interference factors in the image, wherein specific preprocessing methods include but are not limited to gray scale transformation, histogram modification, image smoothing and denoising and the like.
The convolutional neural network is the core of the training module and the whole storage management system. The convolutional neural network includes a feature extraction layer. The feature extraction layer extracts features according to pixel values of a framing area, namely a detection area, in the marked original image, and converts the pixel values of the detection area into data of a plurality of channels. Information for each channel is obtained independently. The number of channels may be plural. And performing convolution pooling on each channel to obtain a feature map of a frame selection area, namely a frame selection feature map for short, wherein the feature sampling layer slides the frame selection feature map one by using a window, namely, the frame selection feature map is sampled one by using a convolution kernel sampling mode to obtain a low-dimensional vector. The feature mapping layer maps the low-dimensional vectors to the fully-connected layer. The fully-connected layer comprises a regression layer for positioning and a classification layer for classification, so that the convolutional neural network learns the position of the target storage object on the labeled original image and the type of the target storage object through interaction. And the full connection layer outputs a result and determines whether the target storage object is detected.
Training and adjusting the convolutional neural network, wherein an identification module of the management system is used for enabling the convolutional neural network to learn and identify whether the target storage object exists in the input image and the type of the target storage object. The identification module comprises an optimization module, and the optimization module adjusts the hyper-parameters of the convolutional neural network according to the loss function curve, the error rate curve and the learning curve generated by the training module. Specifically, the learning rate in the hyper-parameters is adjusted by a loss function curve, and the loss function generated by the training module has various forms. As shown in fig. 3, the loss function curve oscillates in the first mode, indicating that the learning rate is too high, and the falling rate is too low in the second mode, indicating that the learning rate is too low. The regularization term coefficients can also be adjusted by an error rate curve, as shown in fig. 4, a Train curve represents the training error rate, and a vali curve represents the validation error rate. In the training module, the setting value of the regular term coefficient is usually 1, and the value of the regular term coefficient can be adjusted according to a train curve and a vali curve generated after training. And adjusting the size of the convolutional neural network according to the learning curve, namely the number of layers of the convolutional neural network, and the trend of timely layer number change represented by the arrow direction in the figure. As shown in fig. 5, the number of verification images and training time may also be adjusted according to the learning curve. The convolutional neural network adjusted according to the loss function curve, the error rate curve and the learning curve is obviously superior to the convolutional neural network before training. The adjustment of the hyper-parameters is not limited to the three items, and other hyper-parameters may be adjusted according to the training result.
The verification module inputs the images in the verification image database to the optimized convolutional neural network, and the convolutional neural network identifies whether the target storage object exists in the verification images input by the verification image database and the type of the target storage object and outputs a result. And (5) verifying that the image is similar to a real image needing to be identified, and processing and selecting the image by the processing module. And adjusting the hyper-parameters for the second time according to the output result to obtain the optimized convolutional neural network.
The test module tests the optimized convolutional neural network. Processing the collected videos at the inlet of the refrigeration equipment and in the refrigeration equipment into independent frames, inputting the independent frames as test images into an optimized convolutional neural network in a framing mode for identification, determining whether the target storage object exists or not and the type of the target storage object, and outputting an identification result. And storing the recognition result in a video format or a text format, analyzing again, and optimizing the convolutional neural network again by using the analysis result to obtain an optimized network model.
And managing the quantity and the type of the stored materials in the refrigerating device through the trained, verified and tested refrigerating device stored material management system. Specifically, the change in the number of the target stored objects is mainly achieved by putting in and taking out the target stored objects. The shape and the form of the target storage object can not be changed excessively in the refrigerating or freezing process, so that the change caused by the deformation can be ignored in the image interaction processing in the storage object management system, and the most key difficulty of the refrigerating device storage object management system after training, verification and testing is that the target storage objects are mutually shielded in the storage process. Therefore, accurate recognition statistics are achieved in the following manner.
A camera device for taking a video is provided at the entrance of the refrigeration device. The camera device can be static or can shake along with the movement of the inlet of the refrigerating device, and the camera device collects videos at the inlet of the refrigerating device, and the visual field range of the camera device covers the whole inlet and continuous scenes near the inlet. For household refrigeration equipment, the camera device mainly collects the action state that the hand is inserted into or pulled out of the inlet of the refrigeration device. In the case of a large refrigerator, the video captured by the camera device includes the movement of a person entering or exiting the refrigerator, the movement of a hand when the person enters or exits, and a target storage object in the hand. The camera device can be arranged on the refrigerating device and also can be arranged on a fixing structure near the refrigerating device, so that the working stability is ensured. The interior of the refrigerating device is also provided with a camera device for shooting the storage state of the target storage object in the refrigerating device, such as a static scene on a shelf in the refrigerating device. The camera device can be realized by a plurality of independent cameras, and can also realize the image acquisition at the inlet of the refrigerating device and in the refrigerating device through one camera arranged at the inlet of the refrigerating device or on the door body.
The camera device stores the collected images into a video set, and inputs the video set at the inlet of the refrigerating device and the video set in the refrigerating device into a video input module of the storage management system. In order to respectively identify whether the action is to put in or take out the target storage object and the type and the number of the target storage objects, the image decomposition module divides each frame in the video file in the video set into a static component and a motion component. Different from unpredictable moving objects and moving modes in human body movement recognition in the prior art, video image processing at the inlet of a refrigerating device and in the refrigerating device has a relatively fixed detection area, a detection area background and a relatively stable moving mode, so that a recognition mode with higher recognition precision and higher processing speed is required to realize accurate management statistics.
In this embodiment, the image decomposition module decomposes and processes the static component and the motion component of the continuous frames to form a spatial stream and a temporal stream. The static component is a segmented static video frame, and the motion component is an image with a velocity vector reflecting the behavior of the target, such as an optical flow density map. In order to improve the precision and speed of recognition processing, the optimization network model adopts a first convolutional neural network and a second convolutional neural network which are independently arranged and are trained, verified and tested to respectively process spatial streams and time streams. The hyper-parameters of the first convolutional neural network and the second convolutional neural network are optimal values formed by three steps of training, verifying and testing. The first convolutional neural network and the second convolutional neural network are preferably of the same scale, and the hyper-parameters may be set according to specific needs and are slightly different.
Specifically, the first convolutional neural network further comprises an input layer, a multi-layer convolutional pooling layer, and a classifier, wherein the input layer is used for inputting the spatial stream. Since the spatial stream is composed of consecutive still video frames, the background in the still video frames is relatively stable in practice, and the target storage object, i.e., the foreground object, changes significantly during access. Thus, the static component input through the input layer can be represented by the pixel values in the detection area. And extracting effective pixel values in the detection area by the multilayer convolution pooling layer in the first convolution neural network, extracting characteristics and forming a characteristic diagram of the detection area. And selecting a convolution kernel or a window on the feature map, performing convolution by using the convolution kernel, and then further performing pooling. A preferred mode is to use multiple convolution kernels to perform convolution, that is, static components are converted into data of multiple channels, each channel data is independently acquired and is convolved by a convolution kernel, wherein the learned features of each layer of convolution are local, but the features after multiple layers of convolution are close to global, so that better accuracy is achieved. And inputting the output result output by the multilayer convolution pooling layer into a classifier for further learning analysis. Specifically, the output results of the multi-layer convolution pooling layer are mapped to the fully-connected layer and the Softmax layer, so that the first convolution neural network can learn how to determine whether the target storage object exists in the spatial stream generated according to the video set.
Correspondingly, the second convolutional neural network also further comprises an input layer, a multi-layer convolutional pooling layer and a classifier, wherein the input layer is used for inputting the time stream. The time flow is composed of an optical flow density map, and the characteristics and the velocity vector of the time flow can also be represented by pixel values in a detection area. And extracting the characteristics of the dynamic components in the detection area in the optical flow density graph by using the multilayer convolution pooling layer in the second convolution neural network, selecting a convolution kernel or a window on the characteristic graph, performing convolution by using the convolution kernel, and then performing pooling. The dynamic component is also converted into data of a plurality of channels, and each channel data is independently obtained and is respectively convolved by a convolution kernel. And inputting the output result output by the multilayer convolution pooling layer into a classifier for further learning analysis. In particular, the output of the multi-layer convolution pooling layer is mapped to the fully connected layer and the Softmax layer, which enables the second convolutional neural network to learn how to determine whether there is an action to cause the target storage to be removed from or stored in the refrigeration device.
And inputting the output results of the first convolutional neural network and the second convolutional neural network into the same classifier, and fusing the output results of the first convolutional neural network and the second convolutional neural network, wherein the classifier is preferably an SVM (support vector machine), so that whether the target storage object, the type of the target storage object and whether the target storage object is stored in the refrigerating device or taken out of the refrigerating device can be accurately determined according to the output results of the optimized network model.
In many cases, the number of target storage items put into the refrigerator at a time is different, which may cause a deviation if the output result of the direct statistical detection module is deviated, and therefore, an estimation module is further provided in the management system. The function of the evaluation module is primarily to determine the amount of the target storage product to be stored in or removed from the refrigeration device. The estimation module comprises a first estimation module and a second estimation module, when the output result of the detection module determines that the target storage object, the type of the target storage object and the target storage object are placed in the refrigerating device in a single pick-and-place action, the estimation module firstly estimates the contour area of the storage object for the first time according to the space flow in the dynamic scene video set at the inlet of the refrigerating device and uses the contour area as a standard value A1At this time, the standard value A is a value at which the target stored object in the default hand motion is not mutually hidden by different types of target stored objects1The accuracy of (2) is higher. The second estimation module continuously receives the static image in the refrigerating device, generates a change value when the contour area of the target storage object in the static image changes, and takes the change value as the measurement of the target storage object stored in the refrigerating deviceTest value A2. The estimation module is also provided with a calibration module which compares the standard value with the test value, if the test value is not equal to the standard value, the second estimation module generates the test value again until the standard value is equal to the test value, and the quantity of the target storage object stored in the refrigeration device is determined.
Similarly, when the target storage object is taken out of the refrigerating device, the first estimation module is used for estimating the outline area of the taken-out target storage object according to the space flow of the dynamic scene video set at the inlet of the refrigerating device and taking the outline area as a standard value. The second estimation module estimates the change value of the contour area of the target storage object of the static image in the refrigerating device again according to the static scene in the refrigerating device, and the change value is used as the test value of the target storage object taken out of the refrigerating device. The calibration module compares the standard value to the test value. If the test value is not equal to the standard value, the second evaluation module again generates the test value until the standard value is equal to the test value, and determines the quantity of the target storage object taken out of the refrigeration device.
The estimation module outputs the output result to the statistic module. And the statistical module records the types of the target storage objects and increases or decreases the number of the target storage objects when the test value is equal to the standard value. The output result of the statistical module can be directly output to the display module. The display module receives the output result and generates a display value according to the use habit, wherein the display value can comprise information such as the type, storage period and quantity of the target object. The display generated by the display module is displayed through a display screen, and the display screen can be arranged on the refrigerating device or can be arranged on other terminals communicated with the refrigerating device, so that the type of the target storage object in the refrigerating device and the quantity of each type of target storage object can be known and inquired through the display device anytime and anywhere. The whole process does not need to change the traditional use habit, realizes automatic statistics, automatic judgment and automatic display, effectively simplifies the process of the storage statistics and reduces the use cost of the detection statistics.
Referring to fig. 2, which is a schematic structural diagram of a second embodiment of the storage management system of a refrigeration device disclosed in the present invention, the first convolutional neural network and the second convolutional neural network used in this embodiment may also be directly built according to requirements. The first convolutional neural network and the second convolutional neural network are built and provided with an input layer, a plurality of layers of convolutional pooling layers and a classifier. And further optimized during use. In the second embodiment, the interaction, identification, and statistical processes of the video input module, the image decomposition module, the convolutional neural network, the first estimation module, the second estimation module, and the calibration module of the storage management system are the same as those of the first embodiment, and are not described herein again.
The invention also provides a refrigerating device adopting the refrigerating device storage management system specifically disclosed by the embodiment. Referring to the detailed description of the first and second embodiments, the refrigeration device disclosed in the present invention has the technical effect of the refrigeration device storage management system.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A refrigeration unit storage management system, comprising:
the video input module is used for inputting video sets at an inlet of the refrigerating device and in the refrigerating device;
the image decomposition module is used for splitting and processing a single video in the video set into a static component and a motion component and respectively forming a spatial stream and a temporal stream;
the convolutional neural network comprises an optimization network model and is used for determining whether a target storage object exists or not and the type of the target storage object according to the spatial stream and determining whether the target storage object is stored or taken out or not according to the time stream;
the generation of the optimized network model comprises:
the training module is configured to establish a database of static pictures of the target storage objects, wherein the static pictures comprise images of a plurality of target storage objects which mutually occlude and distinguish each part;
the recognition module is configured to enable the convolutional neural network to learn and recognize whether a target storage object exists in an input image or not and the type of the target storage object, and train and obtain the convolutional neural network optimized by the recognition module;
the verification module is configured to input the images in the verification image database into the convolutional neural network optimized by the identification module, so that the convolutional neural network identifies whether the target storage object and the type of the target storage object exist in the verification images input by the verification image database and outputs a result, and readjustment is performed according to the output result to obtain the convolutional neural network optimized by the verification module; and
and the testing module is used for testing the convolutional neural network optimized by the verification module, processing the acquired videos at the inlet of the refrigeration equipment and in the refrigeration equipment into independent frames and inputting the independent frames as test images into the convolutional neural network optimized by the verification module in a frame-by-frame mode for identification, determining whether a target storage object and the type of the target storage object exist or not, outputting an identification result, and optimizing the convolutional neural network optimized by the verification module again according to the identification result to obtain the optimized network model.
2. A cold storage device storage management system according to claim 1,
further comprising: the first estimation module is used for estimating the contour area of the stored object or the extracted object according to the inlet video set of the refrigerating device and the output result of the convolutional neural network;
a second estimation module for estimating the contour area of the target storage object again according to the static scene,
the calibration module is used for determining the quantity of the stored materials according to the comparison of the output of the first estimation module and the output of the second estimation module;
the video set comprises dynamic scenes at the entrance of the refrigeration apparatus and static scenes in the refrigeration apparatus; when the convolutional neural network determines that a target storage object is stored in or taken out according to time flow, the first estimation module is used for estimating the contour area of the target storage object according to the space flow of dynamic scene splitting processing at the inlet of the refrigerating device and taking the contour area as a standard value; the second estimation module is used for estimating the contour area of the target storage object again according to the static scene in the refrigerating device as a test value; the calibration module is used for comparing the areas of the target storage objects acquired twice and determining the quantity of the storage objects.
3. A refrigerator storage management system according to claim 2, further comprising a statistical module for recording the kind of the target storage and increasing or decreasing the number of the target storage when the test value and the standard value are equal.
4. A refrigeration unit storage management system according to any one of claims 1 to 3, wherein the convolutional neural network comprises a first convolutional neural network; the first convolutional neural network further comprises:
an input layer for inputting the spatial stream;
the multilayer convolution pooling layer is used for extracting pixel values of static components forming the space flow, preprocessing the pixel values of a detection area in the static components and extracting features to obtain a feature map;
and the classifier is used for determining and outputting whether the target storage object exists and the type of the target storage object according to the output result of the multilayer convolution pooling layer.
5. A refrigeration unit storage management system as recited in claim 4 wherein said convolutional neural network further comprises a second convolutional neural network; the second convolutional neural network further comprises:
an input layer for inputting the time stream;
the multilayer convolution pooling layer is used for extracting the characteristics of the dynamic components forming the time flow and obtaining a characteristic map with the speed vector characteristics;
and the classifier is used for determining and outputting whether a target storage object is stored in or taken out according to the characteristic diagram output by the multilayer convolution pooling layer.
6. A cold storage device storage management system according to claim 5, wherein the multiple convolutional pooling layers of the first convolutional neural network and the second convolutional neural network convert static and dynamic components into data for a plurality of channels, respectively, with data for each channel being acquired independently.
7. A refrigerator storage management system as recited in claim 6 wherein the first and second convolutional neural networks are the same size, the classifiers of the first and second convolutional neural networks comprising a fully-connected layer and a Softmax layer, respectively.
8. A refrigeration unit storage management system as recited in claim 6 wherein the output of the first convolutional neural network and the second convolutional neural network are input into the same classifier and merged; the classifier used for fusion is an SVM.
9. A refrigerator storage management system as recited in claim 8, wherein the video set includes dynamic scenes of human hand motion at the refrigerator entrance and static scenes on shelves within the refrigerator.
10. A cold storage appliance comprising a cold storage appliance storage management system as claimed in any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610442235.5A CN107526991B (en) | 2016-06-20 | 2016-06-20 | Refrigerating device storage management system and refrigerating device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610442235.5A CN107526991B (en) | 2016-06-20 | 2016-06-20 | Refrigerating device storage management system and refrigerating device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107526991A CN107526991A (en) | 2017-12-29 |
| CN107526991B true CN107526991B (en) | 2022-01-21 |
Family
ID=60734492
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610442235.5A Active CN107526991B (en) | 2016-06-20 | 2016-06-20 | Refrigerating device storage management system and refrigerating device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107526991B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108510116B (en) * | 2018-03-29 | 2020-06-30 | 哈尔滨工业大学 | Case and bag space planning system based on mobile terminal |
| CN111540448A (en) * | 2020-04-20 | 2020-08-14 | 深圳市华智信息科技有限公司 | Intelligent detection method for user behavior, electronic device and computer-readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105531715A (en) * | 2013-06-26 | 2016-04-27 | 亚马逊科技公司 | Detecting item interaction and movement |
| CN105654270A (en) * | 2014-11-18 | 2016-06-08 | 博西华家用电器有限公司 | Refrigerator, terminal, and management system and management method for food materials in refrigerator |
| CN105678216A (en) * | 2015-12-21 | 2016-06-15 | 中国石油大学(华东) | Spatio-temporal data stream video behavior recognition method based on deep learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7627171B2 (en) * | 2003-07-03 | 2009-12-01 | Videoiq, Inc. | Methods and systems for detecting objects of interest in spatio-temporal signals |
-
2016
- 2016-06-20 CN CN201610442235.5A patent/CN107526991B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105531715A (en) * | 2013-06-26 | 2016-04-27 | 亚马逊科技公司 | Detecting item interaction and movement |
| CN105654270A (en) * | 2014-11-18 | 2016-06-08 | 博西华家用电器有限公司 | Refrigerator, terminal, and management system and management method for food materials in refrigerator |
| CN105678216A (en) * | 2015-12-21 | 2016-06-15 | 中国石油大学(华东) | Spatio-temporal data stream video behavior recognition method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107526991A (en) | 2017-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| St-Charles et al. | Universal background subtraction using word consensus models | |
| De Geest et al. | Online action detection | |
| US9443144B2 (en) | Methods and systems for measuring group behavior | |
| CN108600865B (en) | A kind of video abstraction generating method based on super-pixel segmentation | |
| CN107527060B (en) | Refrigerating device storage management system and refrigerating device | |
| CN107527363B (en) | Refrigerating device storage management system and refrigerating device | |
| CN102722698A (en) | Method and system for detecting and tracking multi-pose face | |
| US11308348B2 (en) | Methods and systems for processing image data | |
| US8922674B2 (en) | Method and system for facilitating color balance synchronization between a plurality of video cameras and for obtaining object tracking between two or more video cameras | |
| CN110532970A (en) | Age-sex's property analysis method, system, equipment and the medium of face 2D image | |
| CN110807434A (en) | Pedestrian re-identification system and method based on combination of human body analysis and coarse and fine particle sizes | |
| CN113766330A (en) | Method and device for generating recommendation information based on video | |
| CN109544592A (en) | For the mobile moving object detection algorithm of camera | |
| CN105979366A (en) | Smart television and content recommending method and content recommending device thereof | |
| US20140270388A1 (en) | Methods and systems for measuring group behavior | |
| CN114332911A (en) | Head posture detection method and device and computer equipment | |
| CN107526991B (en) | Refrigerating device storage management system and refrigerating device | |
| CN113468914A (en) | Method, device and equipment for determining purity of commodities | |
| US20220325946A1 (en) | Selective image capture using a plurality of cameras in a refrigerator appliance | |
| Hashemi | A survey of visual attention models | |
| EP4399689A1 (en) | Object detection systems and methods including an object detection model using a tailored training dataset | |
| CN113936237A (en) | No-reference video quality evaluation prediction method and system based on contrast self-supervision | |
| Izadi et al. | Abnormal event detection in indoor video using feature coding | |
| CN112580526A (en) | Student classroom behavior identification system based on video monitoring | |
| Takahara et al. | Making background subtraction robust to various illumination changes |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20211216 Address after: 266101 Haier Road, Laoshan District, Qingdao, Qingdao, Shandong Province, No. 1 Applicant after: QINGDAO HAIER SMART TECHNOLOGY R&D Co.,Ltd. Applicant after: Haier Smart Home Co., Ltd. Address before: 266101 Haier Road, Laoshan District, Qingdao, Qingdao, Shandong Province, No. 1 Applicant before: QINGDAO HAIER SMART TECHNOLOGY R&D Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |