CN109631486A - A kind of food monitoring method, refrigerator and the device with store function - Google Patents
A kind of food monitoring method, refrigerator and the device with store function Download PDFInfo
- Publication number
- CN109631486A CN109631486A CN201811549015.8A CN201811549015A CN109631486A CN 109631486 A CN109631486 A CN 109631486A CN 201811549015 A CN201811549015 A CN 201811549015A CN 109631486 A CN109631486 A CN 109631486A
- Authority
- CN
- China
- Prior art keywords
- food
- maturity
- refrigerator
- image
- date
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 235000013305 food Nutrition 0.000 title claims abstract description 529
- 238000000034 method Methods 0.000 title claims abstract description 109
- 238000012544 monitoring process Methods 0.000 title claims abstract description 42
- 230000008569 process Effects 0.000 claims description 40
- 230000009471 action Effects 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 13
- 230000008859 change Effects 0.000 claims description 12
- 238000003062 neural network model Methods 0.000 claims description 10
- 238000005457 optimization Methods 0.000 claims description 7
- 230000004044 response Effects 0.000 claims description 7
- 230000007613 environmental effect Effects 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 6
- 238000013135 deep learning Methods 0.000 claims description 5
- 230000006386 memory function Effects 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 11
- 230000006870 function Effects 0.000 description 11
- 241000220225 Malus Species 0.000 description 4
- 239000000463 material Substances 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 235000013399 edible fruits Nutrition 0.000 description 3
- 239000010410 layer Substances 0.000 description 3
- 235000016709 nutrition Nutrition 0.000 description 3
- 238000005192 partition Methods 0.000 description 3
- 238000004321 preservation Methods 0.000 description 3
- 235000013311 vegetables Nutrition 0.000 description 3
- 235000021016 apples Nutrition 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000011229 interlayer Substances 0.000 description 2
- 235000013622 meat product Nutrition 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000010411 cooking Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 235000005911 diet Nutrition 0.000 description 1
- 230000037213 diet Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000037406 food intake Effects 0.000 description 1
- 235000012631 food intake Nutrition 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000000050 nutritive effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
- F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
- F25D29/00—Arrangement or mounting of control or safety devices
- F25D29/005—Mounting of control devices
Landscapes
- Engineering & Computer Science (AREA)
- Chemical & Material Sciences (AREA)
- Combustion & Propulsion (AREA)
- Physics & Mathematics (AREA)
- Mechanical Engineering (AREA)
- Thermal Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)
Abstract
This application discloses a kind of food monitoring method, refrigerator and with the device of store function, this method comprises: obtaining the image for the food being put into refrigerator；The image is identified, to obtain the maturity of food；Obtain the maturity corresponding edible boundary date of the food；In the preset time range before this eats boundary date or the edible boundary date, corresponding prompt information is issued, to remind user to handle the food in time.By the above-mentioned means, the application can simplify user's operation, realizes intelligent monitoring of the refrigerator to food, improve the intelligence degree of refrigerator.
Description
Technical Field
The application relates to the technical field of intelligent home, in particular to a food monitoring method, a refrigerator and a device with a storage function.
Background
With the development of the technology and the improvement of the living standard of people, more and more intelligent household appliances begin to go into every family, and the intelligent technology influences the aspects of daily life. The refrigerator is used as an indispensable household appliance for cooking food in modern families, and the existing intelligent refrigerator can only realize a simple food digital recording function, particularly can only manually input the placing time and the expiration time by a user and prompt the expiration time. This method requires the user to manually input the placement time and expiration time of each food, and is cumbersome to operate and not intelligent enough.
Disclosure of Invention
The technical problem mainly solved by the application is to provide a food monitoring method, a refrigerator and a device with a storage function, which can simplify user operation and realize intelligent food monitoring.
In order to solve the technical problem, the application adopts a technical scheme that: there is provided a food monitoring method comprising: acquiring an image of food put in a refrigerator; identifying the image to obtain the maturity of the food; acquiring an edible boundary date corresponding to the maturity of the food; and sending corresponding prompt information in the edible boundary date or a preset time range before the edible boundary date so as to remind a user of processing the food in time.
In order to solve the above technical problem, another technical solution adopted by the present application is: provided is a refrigerator including: an image acquisition device and a processor connected to each other; the image acquisition device is used for acquiring images of food put in the refrigerator; the processor is configured to execute instructions to implement the food monitoring method as described above.
In order to solve the above technical problem, the present application adopts another technical solution: there is provided an apparatus having a storage function, having stored therein program instructions that are executed to implement the food monitoring method as described above.
The beneficial effect of this application is: different from the prior art, in the embodiment of the application, the image of the food put into the refrigerator is acquired, the maturity of the food is obtained after the image is identified, the corresponding eating boundary date is acquired according to the maturity of the food, and corresponding prompt information is sent within a preset time range before the eating boundary date or the eating boundary date to prompt a user to process the food in time, so that the user only needs to put the food into the refrigerator, the refrigerator can automatically judge the maturity of the food and the corresponding eating boundary date according to the acquired food image and make a corresponding prompt to prompt the user to process food materials in time without manually setting eating time and the like by the user, the user operation is simplified, the intelligent monitoring of the food by the refrigerator is realized, and the intelligent degree of the refrigerator is improved.
Drawings
FIG. 1 is a schematic flow chart diagram of a first embodiment of a food monitoring method of the present application;
FIG. 2 is a detailed flowchart of step S11 in FIG. 1;
FIG. 3 is a schematic diagram of a refrigerator to which the present food monitoring method is applied;
FIG. 4 is a schematic flow chart diagram of a second embodiment of the food monitoring method of the present application;
FIG. 5 is a schematic flow chart diagram of a third embodiment of the food monitoring method of the present application;
FIG. 6 is a graph of a maturity curve for a food item A;
FIG. 7 is a schematic flow chart diagram of a fourth embodiment of the food monitoring method of the present application;
FIG. 8 is a schematic flow chart diagram of a fifth embodiment of the food monitoring method of the present application;
FIG. 9 is a schematic flow chart diagram of a sixth embodiment of a food monitoring method of the present application;
FIG. 10 is a schematic flow chart diagram of a seventh embodiment of a food monitoring method of the present application;
FIG. 11 is a schematic flow chart diagram of an eighth embodiment of a food monitoring method of the present application;
FIG. 12 is a schematic structural diagram of an embodiment of a refrigerator according to the present application;
FIG. 13 is a schematic structural diagram of an embodiment of a device with a storage function according to the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
As shown in fig. 1, a first embodiment of the food monitoring method of the present application includes:
s11: an image of food placed in a refrigerator is acquired.
Specifically, an image of food put in the refrigerator may be photographed by an image pickup device installed in the refrigerator. For example, a camera is installed on the top of the refrigerator, or both the top and the inside of the refrigerator, and the installed camera can be used for acquiring images of the food put in by the user when the user puts in the food.
Alternatively, the camera may trigger shooting when the refrigerator is opened, specifically as shown in fig. 2, step S11 includes:
s111: receiving the trigger signal of the opening of the refrigerator door.
Specifically, the refrigerator door is installed with a sensing device, such as a distance sensor, etc., when the refrigerator door is opened, the sensing device generates a trigger signal, and the refrigerator receives the trigger signal, so that the refrigerator door can be known to be opened.
S112: and responding to the trigger signal, triggering a camera in the refrigerator, and shooting an image of the food in the process of putting the food into the refrigerator.
When the refrigerator receives a trigger signal for opening the refrigerator door, a camera in the refrigerator (for example, the camera C at the top of the refrigerator shown in fig. 3) is triggered in response to the trigger signal, and the camera can shoot an image of food in the process of putting the food into the refrigerator, so that the image of the food put into the refrigerator can be obtained. Wherein, the camera can carry out a lot of in the refrigerator door opening time and shoot many images, perhaps directly shoots a section video.
When a plurality of cameras exist in the refrigerator, the cameras can be used for shooting images of food put into the refrigerator or taken out of the refrigerator from a plurality of different angles, so that the complete images of the food can be obtained by combining the plurality of images, and the subsequent food identification precision is improved.
S12: the image is identified to derive the maturity of the food.
Wherein, the maturity of the food is a judgment index corresponding to the preset food freshness. The maturity of the food may be divided into different levels so that the approximate freshness of the food may be determined according to the different levels. For example, a maturity of 1 represents that the food is freshest, e.g., a maturity of 1 when freshly picked, and 0 represents that the food has spoiled, the maturity can be divided into three grades, good, medium and poor. For example, a good rating of 0.7-1 indicates that the food is in the best serving time, a medium rating of 0.3-0.7 indicates that the food is within the upcoming expiration time, and a poor rating of 0-0.3 indicates that the food has expired.
The relationship between the maturity of different foods and the placing time can be stored in the refrigerator in advance, for example, the maturity of an apple after being placed for one day is 0.9, the maturity of the apple after being placed for two days is 0.8, and so on, so as to judge the freshness of the food according to the current maturity of the food.
Specifically, in an application example, the image recognition algorithm may be used to recognize the acquired image, and identify the characteristics of the type, the quantity, the maturity and the like of the food in the image. The characteristics corresponding to the food can be judged according to the characteristics of the shape, the color, the texture, the fullness and the like of the food in the image. Of course, the neural network model trained in the deep learning mode can be used for identifying the characteristics of the maturity and the like of food.
S13: and acquiring the eating boundary date corresponding to the maturity of the food.
Wherein the boundary date of consumption is at least one predetermined boundary date that the food can be consumed before expiration. For example, if there are two boundary dates corresponding to the good maturity levels of the food, i.e., boundary dates corresponding to maturity levels of 1 and 0.7, respectively, the boundary date for eating corresponding to the good maturity level is a boundary date corresponding to 0.7, i.e., the best eating date; and the boundary date of eating corresponding to the medium grade or the poor grade of the maturity of the food is the boundary date corresponding to the maturity of 0.3, namely the inedible date. The specific value of the edible boundary date can be the placing time from the freshest maturity, and if the placing time is reduced to 0.7 after 2 days, the optimal edible date is 2 days. Of course, the eating boundary date may also be a placing date at a time corresponding to the current maturity of the food, and if the placing date decreases from the current maturity of 0.5 to 0.3 for 3 days, the inedible date is 3 days.
Specifically, in one application example, after the maturity of the food is obtained through food image recognition, the maturity level to which the maturity belongs can be obtained according to the maturity of the food, and then the corresponding eating boundary date can be obtained from the maturity level. For example, when the maturity of the food is identified to be 0.5, the maturity level of the food is known to be medium, and then the edible boundary date corresponding to the medium level can be obtained as the inedible date (e.g., 5 days).
S14: and sending corresponding prompt information in the edible boundary date or a preset time range before the edible boundary date so as to remind a user of processing the food in time.
Wherein the preset time range is a time range preset according to the date of the food put into the refrigerator, and the preset time range is not greater than the time difference between the date of the food put into the refrigerator and the eating border date. The value of the preset time range may be fixed, for example, one or two days before the eating boundary date, or may be non-fixed, and the preset time range of each food is different, for example, the optimal prompt time range corresponding to a certain food is calculated according to the maturity curve of the food, so as to prompt the user to process the corresponding food in time for different foods.
The food is usually fresh when being placed in the refrigerator, after the maturity of the food is acquired through the image of the food, the prompt information corresponding to the eating boundary date can be sent within a preset time range before the eating boundary date or the eating boundary date of the food according to the state of the food so as to remind a user of timely processing the food, and therefore the user can be reminded of eating the food before the food is in a better state or is about to expire.
The prompt information can be displayed through a display device arranged on the refrigerator, can be prompted in a voice mode, and can be prompted in a mode of pushing information to other connected terminal equipment (such as a mobile phone of a user). The prompt message may include information on the type, placement location and quantity of the food, and the optimal food duration of the food, so that the user can know which food needs to be treated specifically. The prompt message can be sent periodically, for example, 9 o' clock per day, or a food report can be sent every week, or non-periodically.
Specifically, in an application example, after identifying that the maturity of a certain food is 0.5, it may be obtained that the eating boundary date corresponding to the maturity is 5 days, that is, the placing time from the freshest maturity is 5 days, and the time corresponding to the maturity of the current food is 2 days, that is, the placing time from the freshest maturity is 2 days, and it may be calculated that the current maturity is overdue after being placed for 3 days, that is, a prompt message needs to be sent within 3 days, and the prompt message may include the type of the food and the optimal eating period of the food (for example, within 3 days if the current date prompts), so as to prompt the user to process the food in time. For example, when the mobile phone of the user enters a preset range, the connection is automatically performed, and the prompt message of food is pushed to the mobile phone of the user to remind the user to process the food in time. The refrigerator can also periodically remind in the optimal eating time, and the optimal eating time limit can be synchronously updated by reminding each time (if the refrigerator is placed for one day, the reminding is updated to be within 2 days).
In addition, a database can be arranged in the refrigerator to store food data, including information such as types, quantities, placing time, placing positions and maturity of the food during placing, so that the maturity of the food and other information can be updated in time to obtain corresponding eating boundary dates.
In this embodiment, by acquiring an image of food placed in the refrigerator, identifying the image to obtain a maturity of the food, then acquiring a corresponding eating boundary date according to the maturity of the food, and sending corresponding prompt information before the eating boundary date or the eating boundary date to prompt a user to process the food in time, so that the user only needs to place the food in the refrigerator, the refrigerator can automatically judge the maturity of the food and the corresponding eating boundary date according to the acquired food image, and make a corresponding prompt, and timely prompt the user to process food materials without manually setting eating time and the like, thereby simplifying user operation, realizing intelligent monitoring of the refrigerator on the food, and improving the intelligent degree of the refrigerator. Meanwhile, the image is used for recognition, so that the food can be reminded according to the preservation time and preservation conditions of different food materials, the prompt is accurate, and the accuracy of the prompt is prevented from being influenced by the fact that the information set by the user is mistaken.
In other embodiments, the user is prompted to eat when the food is at a better freshness, which may allow the user to eat while the food maintains a higher nutritional value.
As shown in fig. 4 in detail, the second embodiment of the food monitoring method of the present application is further defined in that step S13 includes:
s131: judging whether the maturity of the food is in the maturity range corresponding to the optimal eating time.
Each kind of food has the optimal eating time, more nutritional value or water and the like of the food can be lost and the taste may not be added when the optimal eating time is exceeded, and the maximum nutritional efficacy of the food can be exerted if the food can be eaten within the optimal eating time. Therefore, in this embodiment, for the maturity of the food, one maturity range may be set as the maturity range corresponding to the optimal eating time of the food. Of course, the optimal eating time may correspond to different ranges of ripeness for different foods.
Specifically, in an application example, the database of the refrigerator stores maturity ranges corresponding to the optimal eating times of different foods, and after the maturity (e.g., 0.8) of a certain food is obtained, the maturity range (e.g., 0.7-1) corresponding to the optimal eating time of the food can be obtained in the database, so that whether the current maturity of the food is within the maturity range corresponding to the optimal eating time can be determined.
If the food ripeness is within the ripeness range corresponding to the optimal eating time, the following step S132 is executed, otherwise, the step S133 is executed.
S132: the optimal consumption boundary date of the food is obtained.
The optimal eating time of the food has two boundary dates, wherein one boundary date is corresponding to the latest freshness degree, and the other boundary date is corresponding to the boundary date close to expiration, and the optimal eating boundary date is the boundary date close to expiration.
When the food is in the optimal eating time, namely the freshness of the food is in a good state, the boundary date which is closest to the expiration in the optimal eating time of the food is obtained, and the reminding can be timely carried out when the food is aged to the maturity which is out of the maturity range corresponding to the optimal eating time, namely the food is not in the optimal eating time.
Further, after step S132, the method further includes:
s141: and sending out optimal eating prompt information in a first preset time range before the optimal eating boundary date or the optimal eating boundary date so as to remind a user to eat the food when the food is kept in a good state.
The first preset time range is a time range preset according to the date of the food put into the refrigerator and the optimal eating date of the food, and the time range is not more than the time difference between the date of the food put into the refrigerator and the optimal eating boundary date. The value of the first preset time range may be fixed or non-fixed.
When the maturity of the food is still in the maturity range corresponding to the optimal eating time, that is, when the food is still in the optimal eating time, after the optimal eating boundary date of the food is obtained, the optimal eating prompt message is sent within the first preset time range (for example, within one day before the optimal boundary date) at or before the optimal boundary date, so that the user can be prompted to eat the food in a good state in time. For example, after the optimal eating date is obtained, the optimal eating boundary date can be converted into the first time from the current date, and then when the optimal eating prompting information is sent, the user is prompted to eat within the first time (for example, 5 days) from the current date so as to eat when the edible nutritive value is high, and the efficacy of the food is exerted to the maximum extent.
S133: the inedible boundary date of the food is obtained.
The inedible boundary date for a food is the time corresponding to the upper limit of the food's outdated maturity range or a food maturity rating of a poor upper limit maturity (e.g., 0.3 out of 0-0.3) or a food maturity rating of a medium lower limit maturity (e.g., 0.7 out of 0.3-0.7).
Alternatively, if the food is not at the optimal consumption time, the food may be overdue and inedible, or may be still edible after the food is overdue, the specific prompting mode may be determined according to the maturity range of the food. As shown in fig. 4, after step S133, the method further includes:
s134: and judging whether the maturity of the food is out of the maturity range corresponding to the optimal eating date and the inedible boundary date.
Each food also has its corresponding expiration time (such as shelf life on the food package), i.e., inedible boundary date, and beyond the inedible boundary date, the food is out of date and goes bad and cannot be eaten. Therefore, in this embodiment, an edible time (or an upcoming expiration time) between the inedible boundary date and the optimal edible date of the food is set according to the maturity of the food, and the edible time corresponds to the range of maturity at which the food can be eaten but should be eaten as soon as possible.
Specifically, in an application example, the database of the refrigerator further stores a maturity range (e.g., 0.3-0.7) corresponding to the expiration time (or the upcoming expiration time) of the food, and after the maturity of the food is obtained, it can be determined whether the maturity of the food is within the range, and if so, the step S142 is executed in response to that the maturity of the food is within the maturity range corresponding to the optimal eating date and the inedible boundary date.
S142: and sending out an upcoming expiration prompt message within a second preset time range before the inedible boundary date or the inedible boundary date to remind a user to process the food before the food is immediately expired.
The second preset time range is a time range preset according to a date when the food is put into the refrigerator and a date when the food cannot be eaten, and is not greater than a time difference between the date when the food is put into the refrigerator and the inedible boundary date. The value of the second preset time range may be fixed or non-fixed. The second preset time range may be the same as or different from the first preset time range. For example, to avoid wasting food, the second time range may be set larger to prompt the user to treat food earlier in time.
When the maturity of the food is out of the maturity range corresponding to the optimal eating date and the inedible boundary date, step S143 is executed in response to the maturity of the food being out of the maturity range corresponding to the optimal eating date and the inedible boundary date.
S143: and sending out an expiration prompt message to remind the user that the food is expired.
Specifically, in the above application example, the inedible date of the food is obtained, and it is determined that the food is within the edible time, the inedible boundary date may be converted into a second time (e.g., 4 days) from the current date, and then when the upcoming expiration prompting message is sent, the user is prompted to eat within the second time (e.g., 4 days) from the current date, so as to process the food before the expiration of the food, thereby avoiding wasting the food. And when the food is out of the edible time, an expiration prompt message is directly sent out to remind the user that the food is expired.
In the embodiment, whether the food is in the optimal eating time can be judged according to the maturity of the food, and the user is reminded to eat the food in the optimal eating time so as to exert the effect of the food to the maximum extent, and the user is reminded to process the food in the edible time when the food is not in the optimal eating time, so that the food can be processed before the food is overdue, and the waste of food materials is avoided; meanwhile, expiration prompt can be carried out when the food is expired so as to remind a user that the food is expired and is processed in time, and the refrigerator is prevented from being polluted or other foods are prevented from being polluted.
In other embodiments, the refrigerator may further store a maturity curve of the food for different foods to automatically estimate the eating boundary date of the food so as to timely inform the user of the processing.
As shown in fig. 5, the third embodiment of the food monitoring method of the present application is based on the second embodiment of the food monitoring method of the present application, and further defines that step S132 or S133 includes:
s61: a maturity curve for the food was obtained.
Wherein, the maturity curve of the food is the relationship curve of the freshness of the food and the time, the maturity curves of different types of food are different, for example, the maturity of the food such as vegetables and fruits is changed faster than the maturity curve of the meat product, therefore, the maturity curves of the vegetables and fruits are steeper than the maturity curve of the meat product.
S62: acquiring an optimal eating boundary date corresponding to the optimal eating time from a maturity curve of the food; or acquiring the inedible boundary date corresponding to the food expiration time from the maturity curve of the food.
Specifically, in one application example, the ripeness curves of different types of food are stored in the refrigerator in advance, and after the types of the food are obtained, the ripeness curves can be obtained. Because the maturity range corresponding to the optimal eating time of the food or the maturity range corresponding to the food expiration time is preset, and the maturity of the food describes the corresponding relationship between the maturity of the food and the placing time, after the maturity range corresponding to the optimal eating time of the food or the maturity range corresponding to the food expiration time is obtained, the optimal eating boundary date corresponding to the optimal eating time of the food or the inedible boundary date corresponding to the food expiration time can be directly obtained from the maturity curve of the food. After the eating boundary date is obtained, the step of sending out corresponding prompt information before the eating boundary date or the eating boundary date to remind the user to process the food in time can be executed. For example, as shown in fig. 6, in the maturity curve of food a, the optimal eating time of food a corresponds to the maturity range of 0.7-1, and the optimal eating boundary date is the time corresponding to the maturity of 0.7 (e.g., 1.5 days in fig. 6). If the maturity of the current food a is 0.8, the time of the food a from the optimal boundary date can be converted into 0.5 day according to the current maturity of 0.8 and the obtained optimal boundary date of eating of 1.5 days, and then the optimal eating prompt message can be sent within 0.5 day. In other application examples, the refrigerator may also have a networking function, and the maturity curve corresponding to the food may be found from a cloud or a network server.
In addition, the ripeness curve of the food is also related to the preservation environment of the food, such as temperature, humidity, etc., and therefore, the ripeness curves of the same type of food are different for different temperature and humidity environments. Environmental conditions of different positions in the refrigerator are different, and maturity curves corresponding to food can be obtained by combining the placement positions of the food in the refrigerator.
Optionally, before the food is expired, a food intake suggestion may be given, as shown in fig. 5, and after step S62, the method further includes:
s63: outputting a feeding optimization suggestion of the food within a third preset time range before the boundary date cannot be eaten or within a fourth preset time range before the optimal boundary date is eaten.
Wherein the third preset time range is a time range preset according to the date when the food is put into the refrigerator and the date when the food cannot be eaten, and the time range is not greater than the time difference between the date when the food is put into the refrigerator and the boundary date when the food cannot be eaten. The value of the third preset time range may be fixed or non-fixed. The fourth preset time range is a time range preset according to the date when the food is put into the refrigerator and the optimal eating date of the food, and the time range is not more than the time difference between the date when the food is put into the refrigerator and the optimal eating border date. The value of the fourth preset time range may be fixed or non-fixed. The third preset time range may be the same as or different from the fourth preset time range.
Specifically, in the case where the food has not expired yet, that is, the food can be eaten yet, for example, the food is still in the optimal eating time, or in the eating time, the eating optimization suggestion of the food, for example, the optimal eating time of the food, the preferred eating manner of the food, the inedible date, or the time from expiration may be output to the user in two days before the optimal eating boundary date, or in three days before the inedible boundary date, and when the same kind of food has individuals with different maturity, the eating suggestion in batches may be given to maintain better freshness, or the suggestion that the fresh food or the stale food is eaten preferentially may be given for the user to select flexibly, and the like. Therefore, in the embodiment, the refrigerator can not only estimate the eating boundary date of the food according to the maturity curve and provide corresponding prompt information so that the user can process the food in time, but also provide an optimized eating suggestion of the food before the food is not expired so as to provide a more intelligent and optimized eating prompt for the user, and further improve the intelligent degree of the refrigerator.
In other embodiments, when a camera is provided inside the refrigerator, the camera may be used to periodically capture images of food placed in the refrigerator to monitor the state of the food in real time.
As shown in fig. 7 in detail, the fourth embodiment of the food monitoring method of the present application is based on the first embodiment of the food monitoring method of the present application, and after step S12 is further defined, the method may further include:
s21: images of food in the refrigerator are periodically taken.
In particular, when a camera is provided inside the refrigerator or a camera on the top of the refrigerator can take images of food inside the refrigerator when the refrigerator door is closed, the images of the food inside the refrigerator can be periodically taken by using the camera inside or on the top of the refrigerator so as to monitor the food in the refrigerator in real time.
S22: images acquired at different times are compared to record the change of food in the refrigerator.
Specifically, after each image of food in the refrigerator is obtained, the image can be compared with the image of food in the refrigerator obtained in the previous time, and through processing the images, information of food in each image, such as the number, type, placement position, maturity and the like of the food can be identified. The identification results of the images acquired at different times are compared, so that the change conditions of food in the refrigerator, such as the change conditions of the maturity of the food, can be obtained, the state of the food in the refrigerator can be monitored in real time, a user can be informed of processing in time, the maturity change of the food can be acquired, the maturity curve of the food can be obtained, the eating boundary date of the food can be estimated, and the user can be reminded of processing the food in time before or on the eating boundary date. In addition, the food currently existing in the refrigerator can be matched with the diet plan of the user or the default strategy of the refrigerator (for example, the eating sequence is the prior eating, namely the food which is about to expire) according to the change of the quantity, the type and the maturity of the food in the refrigerator, eating optimization suggestions or purchasing suggestions and the like are provided for the user, so that the real-time monitoring and intelligent management of the food in the refrigerator are realized, and the intelligent degree of the refrigerator is further improved.
In the food monitoring method, the food image can be processed by adopting a deep learning neural network model so as to identify the maturity of the food.
As shown in fig. 8 in detail, the fifth embodiment of the food monitoring method of the present application is based on the first embodiment of the food monitoring method of the present application, and the step S12 is further defined to include:
s121: and inputting the image into the image recognition model to obtain a recognition result output by the image recognition model.
The image recognition model is pre-stored in a refrigerator, is a neural network model which is trained by a deep learning method in advance, and can intelligently recognize information of food in the image and output a recognition result according to the input image. The recognition result includes information on the type, quantity and maturity of the food in the image. The training samples adopted by the image recognition model comprise images and the like which are put in and taken out of various foods in different light (including light in a refrigerator and ambient background light) environments, so that the image recognition model can be suitable for recognition in various different environments.
S122: and acquiring the maturity of the food in the identification result.
Specifically, after the food image is acquired, the image is input into an image recognition model, the image recognition model can automatically recognize the information of the food in the image and output a recognition result, and the maturity of the food in the image can be acquired from the recognition result.
Alternatively, a plurality of different storage spaces may be provided in the refrigerator, the environment in each storage space is different (e.g., temperature and humidity are different), the type of food suitable for the storage space is also different, and the image recognition model may recognize the type of food so as to instruct the user to put the food into a designated position. As shown in fig. 8, before step S122, the method further includes:
s123: and judging whether the image recognition model is successfully recognized.
In particular, since the image recognition model is a neural network model trained with limited sample data, there may be situations where recognition is unsuccessful, for example, for food held by opaque bags, which is often not possible. When the recognition fails, it will output the signal which can not be recognized or error, and when the recognition succeeds, it will output the recognition result, which is different from the signal which can not be recognized or error. Therefore, whether the recognition is successful or not can be judged according to the output result of the image recognition model.
If the identification is successful, the following step S124 is executed.
S124: and sending out a storage prompt according to the type of the food in the recognition result so as to guide the recommended placement position of the food in the refrigerator.
Specifically, the image recognition model can recognize the food type and the corresponding maturity in the image by using the input image, and the food type can be obtained from the output recognition result. Different storage spaces are arranged in the refrigerator, the storage environment of each storage space can be different, and the storage spaces are suitable for different types of food to be stored, so that after the types of the food are identified, a storage prompt (such as a light or voice prompt) can be sent out to guide a user to put the food into a recommended placement position in the refrigerator.
For example, as shown in fig. 3, the fresh-keeping area of the refrigerator is provided with a plurality of compartments, the storage environment of each compartment is different, the compartment L1 is suitable for placing fruits, the compartment L2 is suitable for placing vegetables, each compartment is provided with a controllable lamp X which is usually in a non-lighting state, and when the type of food which is identified is suitable for being placed into a certain compartment, the controllable lamp X corresponding to the compartment is lighted, so that the user can be guided to place the food into the lighted compartment.
Alternatively, if the image recognition model recognition fails, the following step S125 is performed.
S125: and sending out voice interactive information to prompt the user to confirm the type of the food put in or taken out so as to prompt the user to put the food into the suggested placing position or update the data of the food stored in the refrigerator.
Specifically, when the image recognition model fails to recognize, the refrigerator cannot know the type of food put in or taken out by the user, and at this time, the refrigerator may input voice interaction information to the user by using a voice interaction device, such as a speaker, prompt the user to input the type of food, or select several possible types for the user to select. Then, the type of food input or confirmation by the user is received, when the user puts the food into the refrigerator, the user can be continuously prompted to put the food into the suggested placing position, the data of the food stored in the refrigerator can be updated simultaneously, and when the user takes the food out of the refrigerator, the data (such as type, quantity and the like) of the food stored in the refrigerator can be updated. Wherein, the user can input or confirm the food type through voice or touch screen input, or terminal text input interaction, etc.
Further, after step S122, the method includes:
s126: and acquiring the environment information corresponding to the suggested placement position.
The environmental information may include temperature, humidity, remaining space, and the like.
S127: and obtaining a maturity curve of the food by combining the environmental information and the type of the food.
Specifically, different environmental conditions may affect the change speed of different food maturity, the refrigerator may pre-store maturity change trend information of different food types at different positions in the refrigerator, and after obtaining the food type, the environmental information, and the current food maturity, the corresponding maturity change trend information, that is, the maturity curve of the food, may be found.
S128: and combining the maturity curve, and continuing to execute the step of acquiring the eating boundary date corresponding to the maturity of the food.
Wherein, the food maturity curve is the corresponding relation between the maturity of the food and the placing time of the food, and the maturity curves of different foods may be different. By using the maturity curve of the food, step S13 may be executed to obtain the boundary date of consumption corresponding to the maturity of the food, so as to send out a corresponding prompt message before the boundary date of consumption or the boundary date of consumption of the food, so as to remind the user to process the food in time.
During the edible time of the food, the refrigerator can also give eating optimization suggestion or purchase suggestion of the food according to the data of the food in the refrigerator.
Alternatively, the refrigerator may also issue a prompt to guide the user to put the food to the designated location when the user does not put the food to the designated location. As shown in fig. 8, after step S124, the method further includes:
s129: an image of the proposed placement location in the refrigerator is obtained.
Wherein, after the user puts the food into the refrigerator, the refrigerator can continue to acquire the image of the food in the refrigerator, wherein the image comprises the suggested placement position.
S130: it is identified whether the food is placed in the suggested placement location.
Specifically, the refrigerator may recognize whether corresponding food is placed in the suggested placement position in the image through an image recognition algorithm or a pre-trained image recognition model, and if the corresponding food is not placed in the suggested placement position, the following step S131 is performed, otherwise, the step S122 is continuously performed.
S131: and sending out a voice prompt to prompt the user to place the food to the suggested placement position.
When the corresponding food is not placed in the suggested placing position, the refrigerator can send out a voice prompt to prompt a user to place the food in the suggested placing position, so that the freshness keeping time of the food is prolonged to the maximum extent.
In other embodiments, the refrigerator may also recognize whether the user puts food into the refrigerator or takes food out of the refrigerator in order to give different suggestions.
As shown in fig. 9, the sixth embodiment of the food monitoring method of the present application is based on the first embodiment of the food monitoring method of the present application, and further includes:
s31: images of the process of putting food into or taking food out of the refrigerator are acquired.
Specifically, an image capturing device is installed in the refrigerator, for example, a camera is installed on the top of fig. 3, and when the door of the refrigerator is opened, the camera is triggered to perform multiple times of shooting, so that a process image of food being put into or taken out of the refrigerator can be shot, and the process image includes multiple images. The camera can also directly shoot videos, and further process images of food placed in the refrigerator or food taken out of the refrigerator can be obtained from the obtained videos.
S32: the process image is processed to identify a food pick-up action in the process image.
Specifically, in an application example, a plurality of consecutive images at adjacent times in the process image may be compared, for example, two images taken at a previous time and two images taken at an adjacent subsequent time are processed to identify a position change of food or a position change of a hand for taking food, that is, whether a food taking action is to be taken in or taken out may be identified. For example, if it is recognized that the hand holds food in the image of the initial stage and it is recognized that the hand does not hold food when the hand leaves the visible area of the refrigerator camera in the image of a later period, it can be determined from the image frame taken that the taking action is put in; on the contrary, if the fact that the food is not taken when the hand enters the visible area of the camera in the image in the initial stage is recognized, and the food is taken when the hand leaves the visible area of the camera in the image after a period of time, the taking action can be judged to be taking.
In another application example, images of food taken from the refrigerator can be collected in advance, wherein the images comprise two types of images of food taking and placing, and a neural network model based on deep learning is trained to be used as a sample to identify whether the food taking action is the food taking or the food taking action. After the neural network model is trained, after a process image of food put into a refrigerator or taken out of the refrigerator is acquired, the process image is directly input into the neural network model, the neural network model can be identified, and then the identification result of taking action is directly output to judge whether the food is put into or taken out.
S33: and judging whether the food taking action in the process image is taking.
If the food taking operation is recognized as taking out, step S34 is executed.
S34: data of food stored in the refrigerator is updated to give eating optimization advice or purchasing advice of food in the refrigerator.
Specifically, when the process image is processed, the information of the food in the single frame image of the process image, including the type, quantity, maturity, etc. of the food, may also be recognized, for example, when the recognition is performed by using a neural network, the neural network model may be the same image recognition model as that in step S121, and the image recognition network may recognize not only the taking action of the food but also the information of the food. And then in the case that the taking action of the user is identified as taking, the data of the food stored in the refrigerator can be updated so as to give optimized eating suggestions or purchasing suggestions and the like of the food in the refrigerator, so that the food in the refrigerator can be intelligently managed.
Optionally, when the process image is processed, the type of the food in the process image may be identified, and further, when the food taking action is identified as putting in, a storage prompt may be given. Specifically, as shown in fig. 9, after step S33, if it is recognized that the food is taken as being put in, the following step S35 is executed.
S35: depending on the type of food, a storage reminder is issued to guide the food to a suggested placement location in the refrigerator.
Specifically, after the process image is processed, the information (including at least the type of the food) of the food can be obtained while recognizing the taking motion of the food as the putting in. Since the types of foods suitable for storage may be different in different storage locations in the refrigerator, the freshness keeping time of the foods can be maximally extended by placing the foods in the suitable storage locations. Thus, depending on the type of food, the refrigerator may issue a storage alert (e.g., a light or voice alert) to guide the recommended placement of the food in the refrigerator in order to maximize the freshness of the food.
Of course, in other embodiments, after the step S33 determines that the picking operation is putting in, the type and amount of food in the refrigerator may be changed, and the data of the food stored in the refrigerator may be updated.
When the food put in the refrigerator is a plurality of similar foods, in order to reduce the storage amount, a maturity closest to expiration may be selected as a representative to be monitored.
As shown in fig. 10, the seventh embodiment of the food monitoring method of the present application is based on the first embodiment of the food monitoring method of the present application, and the step S12 is further defined to include:
s41: and judging whether the image is identified to identify the maturity of different individuals of the same kind of food.
In particular, the food that a user places in the refrigerator may be multiple, for example, multiple apples, each having its own maturity. At this time, if the ripeness of different individuals of the same kind of food, for example, the ripeness of a plurality of apples, is recognized after the acquired food image is recognized, the following step S42 is performed, otherwise, the step S43 is performed.
S42: and acquiring the pre-overdue maturity range of the similar food, and taking the maturity of different individuals, which is closest to the pre-overdue maturity range, as the maturity of the similar food.
Wherein, the pre-expired maturity range is a preset maturity range of food to be expired. The values may be different for different food types. The refrigerator may pre-store pre-expiration maturity ranges corresponding to different types of food, or the refrigerator may search the pre-expiration maturity ranges of the same type of food from the cloud or the network.
S43: the identified maturity of a single individual or the same maturity of different individuals is taken as the maturity of the same type of food.
When the maturity of different individuals is the same or only one individual is available, the identified maturity of a single individual or the same maturity of different individuals can be used as the maturity of the same kind of food by only using the identified maturity as the maturity of the food, and step S13 is continuously executed to judge whether the food is expired or about to expire according to the maturity of the food. When the ripeness of a plurality of different individuals is different, a ripeness closest to the pre-overdue ripeness range inevitably exists, and at the moment, the ripeness closest to the pre-overdue ripeness range can be used as the ripeness of the same type of food, so that the condition that a certain individual is overdue due to improper ripeness of the same type of food and is not reminded is avoided. In addition, the maturity of the foods of the same type is represented by the maturity closest to the pre-expiration range, so that the maturity data of the foods needing to be stored in the refrigerator can be reduced, and the storage resources are saved.
For some foods, the gas generating device can generate some gas, and the concentration or the composition of the gas generated when the foods are about to expire can be changed, so that the gas in the refrigerator can be sensed to assist in identifying the maturity change of the foods.
As shown in fig. 11, the eighth embodiment of the food monitoring method of the present application is based on the first embodiment of the food monitoring method of the present application, and after step S12 is further defined, the method further includes:
s51: data of an odor sensor in a refrigerator is acquired.
Wherein, the number of the smell sensors can be a plurality of and are arranged in different interlayers in the refrigerator so as to identify the smell of food in the different interlayers. Of course, the odor sensor may be only one, sensing the odor in the entire refrigerator. The odor sensor may sense information of odor type, concentration, etc. in the refrigerator and generate data.
S52: and combining the data of the odor sensor and the recognition result of recognizing the food image to obtain the maturity of the food.
Specifically, in one application example, the refrigerator may obtain data of the odor sensors in the different partition layers, obtain information such as components and concentrations of odor emitted by food placed in the different partition layers in the refrigerator according to the data, estimate the approximate maturity of the food in the different partition layers, and combine the estimated maturity with the recognition result of the food image to comprehensively determine the maturity of the food.
In the embodiment, the food maturity is judged by combining the data of the odor sensor, the image recognition result can be supplemented, and the accuracy of judging the food maturity is improved.
The above embodiments of the food monitoring method of the present application can be combined with other non-conflicting embodiments.
As shown in fig. 12, in an embodiment of the refrigerator of the present application, the refrigerator 10 includes: an image acquisition device 101 and a processor 102 connected to each other.
The image capturing device 101 may be a camera, the number of which may be one or more, and it may be disposed on the top of the refrigerator 10, or may be disposed on the top and inside of the refrigerator 10 at the same time. The image taking apparatus 10 is used to take images of food put in the refrigerator 10.
The processor 102 is used to control the operation of the refrigerator 10, such as controlling the image capturing device 101 to capture images. The processor 102 may also be referred to as a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip having signal processing capabilities. The processor 102 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The processor 102 is specifically configured to execute instructions to implement the methods as provided in any one of the first to eighth embodiments of the food monitoring method of the present application or non-conflicting combinations thereof.
Optionally, the refrigerator 10 further includes an odor sensor 103 coupled to the processor 102. The smell sensor 103 is used to sense smell in the refrigerator 10 and generate data.
The processor 102 is configured to obtain data of the odor sensor 103, and combine the data with a recognition result of recognizing an image of food to obtain the maturity of the food.
Optionally, the refrigerator 10 further comprises a voice device 104 connected to the processor 101. The voice device 104 may include a microphone and a speaker for voice interaction with a user.
Optionally, the refrigerator 10 further comprises a memory 105 connected to the processor 102. The memory 105 is used to store instructions and quantities required for the processor 102 to execute, such as information on food placed in the bin. The memory 105 may also be connected to the image capturing device 101, the odor sensor 103, and the voice device 104 to store data captured by the image capturing device.
Of course, in other embodiments, the refrigerator 10 may further include other components such as a display (not shown), and is not limited herein.
In this embodiment, after the processor of the refrigerator acquires the image of the food placed in the refrigerator by using the image acquisition device, the image is identified to obtain the maturity of the food, and then whether the food is expired or is about to be determined according to the maturity of the food is determined, when the food is expired, expiration prompt information is sent out to remind a user that the food is expired, and when the food is about to be expired, expiration prompt information is sent out to remind the user that the food is about to be expired, so that the user can automatically determine whether the food is expired or about to be expired according to the acquired food image and prompt the user without manually setting expiration time and the like by the user only by placing the food into the refrigerator, thereby simplifying user operation, realizing intelligent monitoring of the food by the refrigerator, and improving the intelligent degree of the refrigerator.
As shown in fig. 13, in an embodiment of the apparatus with storage function of the present application, the apparatus with storage function 20 stores therein program instructions 210, and the program instructions 210 are executed to implement the method provided in any one of the first to eighth embodiments of the food monitoring method of the present application or their non-conflicting combinations.
The apparatus 20 with a storage function may be a portable storage medium such as a usb disk, an optical disk, a portable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or a magnetic disk, which can store program instructions, a mobile terminal, a home appliance controller, or an independent component that can be integrated in a refrigerator, such as a control chip, or a server that stores the program instructions, and the server can send the stored program instructions to other devices for operation, or can self-operate the stored program instructions.
In an embodiment, the device 20 with storage function may also be a memory as shown in fig. 11.
In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the purpose of illustrating embodiments of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application or are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.
Claims (20)
1. A food monitoring method, comprising:
acquiring an image of food put in a refrigerator;
identifying the image to obtain the maturity of the food;
acquiring an edible boundary date corresponding to the maturity of the food;
and sending corresponding prompt information in the edible boundary date or a preset time range before the edible boundary date so as to remind a user of processing the food in time.
2. The method of claim 1, wherein obtaining an eating boundary date corresponding to the maturity of the food comprises:
judging whether the maturity of the food is in a maturity range corresponding to the optimal eating time or not;
responding to the maturity of the food in the maturity range corresponding to the optimal eating time, and acquiring the optimal eating boundary date of the food;
and responding to the maturity of the food not being in the maturity range corresponding to the optimal eating time, and acquiring the inedible boundary date of the food.
3. The method of claim 2, wherein the issuing of the corresponding prompt message to remind the user to process the food in time within a preset time range before the boundary date of consumption or the boundary date of consumption comprises:
sending an optimal eating prompt message within a first preset time range before the optimal eating boundary date or the optimal eating boundary date to remind a user to eat when the food is kept in a good state; or,
and sending out an upcoming expiration prompt message within a second preset time range before the inedible boundary date or the inedible boundary date to remind a user to process the food before the food is immediately expired.
4. The method of claim 3, wherein said obtaining said food after a non-edible boundary date comprises:
judging whether the maturity of the food is out of the maturity range corresponding to the optimal eating date and the inedible boundary date;
responding to the maturity of the food out of the maturity range corresponding to the optimal eating date and the inedible boundary date, and sending out expiration prompt information to remind a user that the food is expired;
responding to the maturity of the food in the maturity range corresponding to the optimal eating date and the inedible boundary date, and continuing to execute the step of sending out an upcoming expiration prompt message in the preset time range before the inedible boundary date or the inedible boundary date so as to remind a user of processing the food before the food is about to expire.
5. The method of claim 2,
the obtaining of the optimal eating boundary date of the food comprises:
obtaining a maturity curve of the food;
acquiring the optimal eating boundary date corresponding to the optimal eating time from the maturity curve of the food;
the obtaining of the inedible boundary date for the food comprises:
obtaining a maturity curve of the food;
and acquiring the inedible boundary date corresponding to the food expiration time from the maturity curve of the food.
6. The method of claim 2, wherein the issuing of the corresponding prompt message to remind the user to process the food in time within a preset time range before the boundary date of consumption or the boundary date of consumption comprises:
outputting a feeding optimization suggestion of the food within a third preset time range before the inedible boundary date or within a fourth preset time range before the optimal eating boundary date.
7. The method of claim 1, wherein the acquiring an image of food placed in the refrigerator comprises:
receiving a trigger signal for opening a refrigerator door;
and responding to the trigger signal, triggering a camera in the refrigerator, and shooting an image of the food in the process of putting the food into the refrigerator.
8. The method of claim 1, wherein said identifying the image for maturity of the food comprises:
periodically taking images of food in the refrigerator;
and comparing the images acquired at different times to track and record the change of food in the refrigerator.
9. The method of claim 1, wherein the identifying the image to derive the maturity of the food comprises:
inputting the image into an image recognition model to obtain a recognition result output by the image recognition model;
acquiring the maturity of the food in the identification result;
wherein the image recognition model is a deep learning based neural network model pre-stored in the refrigerator.
10. The method of claim 9,
before the obtaining of the maturity of the food in the identification result, the method includes:
judging whether the image recognition model is successfully recognized or not;
in response to the successful recognition of the image recognition model, sending a storage prompt according to the type of the food in the recognition result so as to guide the food to a suggested placement position in a refrigerator;
after the obtaining of the maturity of the food in the identification result, the method includes:
acquiring environment information corresponding to the suggested placement position;
acquiring a maturity curve of the food by combining the environmental information and the type of the food;
and combining the maturity curve, and continuing to execute the step of acquiring the eating boundary date corresponding to the maturity of the food.
11. The method of claim 10, wherein the issuing a storage prompt to guide the food to a suggested placement location in a refrigerator according to the type of the food in the recognition result comprises:
acquiring an image of the suggested placement position in the refrigerator;
identifying whether the food item is placed in the suggested placement location;
in response to the food not being placed in the suggested placement location, issuing a voice prompt to prompt a user to place the food to the suggested placement location.
12. The method of claim 10, wherein after determining whether the image recognition model is successfully recognized, further comprising:
and responding to the failure of the image recognition model recognition, sending out voice interaction information, prompting the user to confirm the type of the put-in or taken-out food, prompting the user to put the food into the suggested placing position or updating data of the food stored in the refrigerator.
13. The method of claim 1, further comprising:
acquiring a process image of food put in or taken out of a refrigerator;
processing the process image to identify a food pick-up action in the process image;
judging whether the food taking action in the process image is taking out;
in response to the food taking action in the process image being a take, data of food stored in the refrigerator is updated to give a feeding optimization suggestion or a purchasing suggestion of food in the refrigerator.
14. The method of claim 1, wherein the identifying the image to derive the maturity of the food comprises:
judging whether the images are identified to identify the maturity of different individuals of the same kind of food or not;
in response to identifying the maturity of different individuals of a food of the same type, obtaining a pre-expiration maturity range for the food of the same type;
and taking the maturity of the different individuals, which is closest to the pre-expired maturity range, as the maturity of the same type of food.
15. The method of claim 1, wherein after identifying the image to obtain the maturity of the food, further comprising:
acquiring data of an odor sensor in the refrigerator;
and combining the data of the odor sensor and the recognition result of the image recognition to obtain the maturity of the food.
16. A refrigerator, characterized by comprising: an image acquisition device and a processor connected to each other;
the image acquisition device is used for acquiring images of food put in the refrigerator;
the processor is configured to execute instructions to implement the method of any one of claims 1-15.
17. The refrigerator of claim 16, further comprising: and the memory is connected with the processor and is used for storing instructions and data required by the processor to execute.
18. The refrigerator of claim 16, further comprising: an odor sensor connected to the processor for sensing odor in the refrigerator and generating data;
the processor is used for acquiring the data and obtaining the maturity of the food by combining the data and the recognition result for recognizing the image.
19. The refrigerator of claim 16, further comprising: and the voice equipment is connected with the processor and is used for carrying out voice interaction with a user.
20. An apparatus having a memory function, having stored therein program instructions, characterized in that the program instructions are executed to implement the method according to any one of claims 1-15.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811549015.8A CN109631486A (en) | 2018-12-18 | 2018-12-18 | A kind of food monitoring method, refrigerator and the device with store function |
| PCT/CN2019/123804 WO2020125456A1 (en) | 2018-12-18 | 2019-12-06 | Food monitoring method, refrigerator, and apparatus having storage function |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811549015.8A CN109631486A (en) | 2018-12-18 | 2018-12-18 | A kind of food monitoring method, refrigerator and the device with store function |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109631486A true CN109631486A (en) | 2019-04-16 |
Family
ID=66075104
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811549015.8A Pending CN109631486A (en) | 2018-12-18 | 2018-12-18 | A kind of food monitoring method, refrigerator and the device with store function |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109631486A (en) |
| WO (1) | WO2020125456A1 (en) |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110457004A (en) * | 2019-07-29 | 2019-11-15 | 上海秒针网络科技有限公司 | The playback method and device of prompt information |
| CN110598591A (en) * | 2019-08-28 | 2019-12-20 | 卓尔智联(武汉)研究院有限公司 | Information processing method and device, equipment and storage medium |
| CN110659676A (en) * | 2019-09-09 | 2020-01-07 | 卓尔智联(武汉)研究院有限公司 | Information processing method, device and storage medium |
| CN110942286A (en) * | 2019-11-27 | 2020-03-31 | 上海秒针网络科技有限公司 | Food management method and device and storage medium |
| CN111258858A (en) * | 2020-01-07 | 2020-06-09 | 青岛海尔电冰箱有限公司 | Refrigerator and control method thereof |
| CN111277671A (en) * | 2020-03-18 | 2020-06-12 | 天津商业大学 | Intelligent household refrigerator of Internet of things |
| WO2020125456A1 (en) * | 2018-12-18 | 2020-06-25 | 广东美的白色家电技术创新中心有限公司 | Food monitoring method, refrigerator, and apparatus having storage function |
| CN111486656A (en) * | 2020-04-08 | 2020-08-04 | 海信（山东）冰箱有限公司 | Refrigeration equipment and control method thereof |
| WO2021027655A1 (en) * | 2019-08-15 | 2021-02-18 | 青岛海尔电冰箱有限公司 | Method for viewing and tracking storage items |
| CN112393521A (en) * | 2020-12-04 | 2021-02-23 | 武汉碗上好科技有限公司 | Refrigerator remote food selling and purchasing control method and refrigerator |
| CN112394149A (en) * | 2019-08-13 | 2021-02-23 | 青岛海尔智能技术研发有限公司 | Food material maturity detection prompting method and device and kitchen electrical equipment |
| CN112432414A (en) * | 2020-12-11 | 2021-03-02 | 恩亿科(北京)数据科技有限公司 | Refrigerator freshness management method and system based on image recognition |
| CN112842062A (en) * | 2021-03-01 | 2021-05-28 | 广东美的厨房电器制造有限公司 | Control method and device of cooking equipment, cooking equipment and readable storage medium |
| WO2021110066A1 (en) * | 2019-12-06 | 2021-06-10 | 广东美的白色家电技术创新中心有限公司 | Food maturity level identification method and device, and computer storage medium |
| CN113154786A (en) * | 2020-01-22 | 2021-07-23 | 青岛海尔电冰箱有限公司 | Food reminding method, refrigerator and computer-readable storage medium |
| CN113154784A (en) * | 2020-01-22 | 2021-07-23 | 青岛海尔电冰箱有限公司 | Food reminding method, refrigerator and computer-readable storage medium |
| CN113218889A (en) * | 2020-01-21 | 2021-08-06 | 青岛海尔电冰箱有限公司 | Fruit detection method, refrigerator and storage medium |
| CN113633184A (en) * | 2020-04-27 | 2021-11-12 | 佛山市顺德区美的电热电器制造有限公司 | Cooking appliance, detection method, detection device and computer-readable storage medium |
| CN113701432A (en) * | 2021-08-11 | 2021-11-26 | 安徽康佳同创电器有限公司 | Multifunctional refrigerator with fruit ripening acceleration function and control method thereof |
| CN113915939A (en) * | 2021-03-25 | 2022-01-11 | 海信（山东）冰箱有限公司 | Refrigerator and food material management method |
| CN114877611A (en) * | 2021-03-31 | 2022-08-09 | 青岛海尔电冰箱有限公司 | Method and equipment for improving image recognition accuracy rate and refrigerator |
| WO2023154539A1 (en) * | 2022-02-14 | 2023-08-17 | Springhouse Technologies Inc. | Live inventory and internal environmental sensing method and system for an object storage structure |
| WO2023193635A1 (en) * | 2022-04-05 | 2023-10-12 | 海尔智家股份有限公司 | Refrigeration appliance for tracking stored item in freezer compartment, and method |
| US12063127B2 (en) | 2021-02-15 | 2024-08-13 | International Business Machines Corporation | Smart refrigerator for smart consumption of food |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202281453U (en) * | 2011-09-16 | 2012-06-20 | 海尔集团公司 | Refrigerator with freshness detection device |
| KR20160000931A (en) * | 2014-06-25 | 2016-01-06 | (주)한국크레딕라이프 | A SYSTEM FOR MANAGING FOOD KEEPING IN A REFRIGERATOR BY AN IoT AND THE REFRIGERATOR THEREOF |
| CN105444511A (en) * | 2014-08-29 | 2016-03-30 | 酷派软件技术（深圳）有限公司 | Intelligent temperature control method and device for refrigerator and refrigerator |
| CN106568294A (en) * | 2016-10-14 | 2017-04-19 | 青岛海尔特种电冰箱有限公司 | Control method for refrigerator, and refrigerator |
| CN108224894A (en) * | 2018-01-08 | 2018-06-29 | 合肥美的智能科技有限公司 | Food material freshness identification method and device based on deep learning, refrigerator and medium |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102257497B1 (en) * | 2014-07-21 | 2021-05-31 | 삼성전자주식회사 | Gas sensor, refrigerator having the same and control method for the refrigerator |
| CN106996676A (en) * | 2016-01-25 | 2017-08-01 | 希姆通信息技术（上海）有限公司 | Refrigerator management system |
| CN107024433A (en) * | 2016-02-02 | 2017-08-08 | 苏州三星电子有限公司 | The method and system of food freshness in a kind of detection refrigerator |
| CN107782059A (en) * | 2016-08-24 | 2018-03-09 | 南京乐朋电子科技有限公司 | Food fresh keeping system for prompting in a kind of refrigerator |
| CN108662842A (en) * | 2017-03-27 | 2018-10-16 | 青岛海尔智能技术研发有限公司 | The detecting system and refrigerator of food in refrigerator |
| CN108131879A (en) * | 2017-11-21 | 2018-06-08 | 珠海格力电器股份有限公司 | Refrigerator and cleaning method, device and system thereof |
| CN108759294A (en) * | 2018-07-14 | 2018-11-06 | 王长贵 | A kind of refrigerator intelligent recognition and management system |
| CN109631486A (en) * | 2018-12-18 | 2019-04-16 | 广东美的白色家电技术创新中心有限公司 | A kind of food monitoring method, refrigerator and the device with store function |
-
2018
- 2018-12-18 CN CN201811549015.8A patent/CN109631486A/en active Pending
-
2019
- 2019-12-06 WO PCT/CN2019/123804 patent/WO2020125456A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN202281453U (en) * | 2011-09-16 | 2012-06-20 | 海尔集团公司 | Refrigerator with freshness detection device |
| KR20160000931A (en) * | 2014-06-25 | 2016-01-06 | (주)한국크레딕라이프 | A SYSTEM FOR MANAGING FOOD KEEPING IN A REFRIGERATOR BY AN IoT AND THE REFRIGERATOR THEREOF |
| CN105444511A (en) * | 2014-08-29 | 2016-03-30 | 酷派软件技术（深圳）有限公司 | Intelligent temperature control method and device for refrigerator and refrigerator |
| CN106568294A (en) * | 2016-10-14 | 2017-04-19 | 青岛海尔特种电冰箱有限公司 | Control method for refrigerator, and refrigerator |
| CN108224894A (en) * | 2018-01-08 | 2018-06-29 | 合肥美的智能科技有限公司 | Food material freshness identification method and device based on deep learning, refrigerator and medium |
Cited By (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020125456A1 (en) * | 2018-12-18 | 2020-06-25 | 广东美的白色家电技术创新中心有限公司 | Food monitoring method, refrigerator, and apparatus having storage function |
| CN110457004A (en) * | 2019-07-29 | 2019-11-15 | 上海秒针网络科技有限公司 | The playback method and device of prompt information |
| CN112394149A (en) * | 2019-08-13 | 2021-02-23 | 青岛海尔智能技术研发有限公司 | Food material maturity detection prompting method and device and kitchen electrical equipment |
| CN112394149B (en) * | 2019-08-13 | 2023-12-22 | 青岛海尔智能技术研发有限公司 | Food material maturity detection prompting method and device and kitchen electric equipment |
| CN114270124B (en) * | 2019-08-15 | 2023-05-16 | 青岛海尔电冰箱有限公司 | Method for viewing and tracking stored items |
| WO2021027655A1 (en) * | 2019-08-15 | 2021-02-18 | 青岛海尔电冰箱有限公司 | Method for viewing and tracking storage items |
| CN114270124A (en) * | 2019-08-15 | 2022-04-01 | 青岛海尔电冰箱有限公司 | Method for viewing and tracking stored items |
| CN110598591A (en) * | 2019-08-28 | 2019-12-20 | 卓尔智联(武汉)研究院有限公司 | Information processing method and device, equipment and storage medium |
| CN110659676A (en) * | 2019-09-09 | 2020-01-07 | 卓尔智联(武汉)研究院有限公司 | Information processing method, device and storage medium |
| CN110942286A (en) * | 2019-11-27 | 2020-03-31 | 上海秒针网络科技有限公司 | Food management method and device and storage medium |
| US12094228B2 (en) | 2019-12-06 | 2024-09-17 | Guangdong Midea White Home Appliance Technology Innovation Center Co., Ltd | Method of identifying level of doneness of food, device, and computer storage medium |
| WO2021110066A1 (en) * | 2019-12-06 | 2021-06-10 | 广东美的白色家电技术创新中心有限公司 | Food maturity level identification method and device, and computer storage medium |
| CN111258858A (en) * | 2020-01-07 | 2020-06-09 | 青岛海尔电冰箱有限公司 | Refrigerator and control method thereof |
| CN113218889A (en) * | 2020-01-21 | 2021-08-06 | 青岛海尔电冰箱有限公司 | Fruit detection method, refrigerator and storage medium |
| CN113154786B (en) * | 2020-01-22 | 2022-10-04 | 青岛海尔电冰箱有限公司 | Food reminding method, refrigerator and computer-readable storage medium |
| CN113154784A (en) * | 2020-01-22 | 2021-07-23 | 青岛海尔电冰箱有限公司 | Food reminding method, refrigerator and computer-readable storage medium |
| CN113154786A (en) * | 2020-01-22 | 2021-07-23 | 青岛海尔电冰箱有限公司 | Food reminding method, refrigerator and computer-readable storage medium |
| CN111277671A (en) * | 2020-03-18 | 2020-06-12 | 天津商业大学 | Intelligent household refrigerator of Internet of things |
| CN111486656A (en) * | 2020-04-08 | 2020-08-04 | 海信（山东）冰箱有限公司 | Refrigeration equipment and control method thereof |
| CN113633184B (en) * | 2020-04-27 | 2022-10-11 | 佛山市顺德区美的电热电器制造有限公司 | Cooking appliance, detection method, detection device and computer-readable storage medium |
| CN113633184A (en) * | 2020-04-27 | 2021-11-12 | 佛山市顺德区美的电热电器制造有限公司 | Cooking appliance, detection method, detection device and computer-readable storage medium |
| CN112393521B (en) * | 2020-12-04 | 2022-12-20 | 合肥荣事达电子电器集团有限公司 | Refrigerator remote food selling and purchasing control method and refrigerator |
| CN112393521A (en) * | 2020-12-04 | 2021-02-23 | 武汉碗上好科技有限公司 | Refrigerator remote food selling and purchasing control method and refrigerator |
| CN112432414A (en) * | 2020-12-11 | 2021-03-02 | 恩亿科(北京)数据科技有限公司 | Refrigerator freshness management method and system based on image recognition |
| US12063127B2 (en) | 2021-02-15 | 2024-08-13 | International Business Machines Corporation | Smart refrigerator for smart consumption of food |
| CN112842062A (en) * | 2021-03-01 | 2021-05-28 | 广东美的厨房电器制造有限公司 | Control method and device of cooking equipment, cooking equipment and readable storage medium |
| CN113915939A (en) * | 2021-03-25 | 2022-01-11 | 海信（山东）冰箱有限公司 | Refrigerator and food material management method |
| CN114877611A (en) * | 2021-03-31 | 2022-08-09 | 青岛海尔电冰箱有限公司 | Method and equipment for improving image recognition accuracy rate and refrigerator |
| CN114877611B (en) * | 2021-03-31 | 2023-09-29 | 青岛海尔电冰箱有限公司 | Method, equipment and refrigerator for improving image recognition accuracy |
| CN113701432A (en) * | 2021-08-11 | 2021-11-26 | 安徽康佳同创电器有限公司 | Multifunctional refrigerator with fruit ripening acceleration function and control method thereof |
| WO2023154539A1 (en) * | 2022-02-14 | 2023-08-17 | Springhouse Technologies Inc. | Live inventory and internal environmental sensing method and system for an object storage structure |
| WO2023193635A1 (en) * | 2022-04-05 | 2023-10-12 | 海尔智家股份有限公司 | Refrigeration appliance for tracking stored item in freezer compartment, and method |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020125456A1 (en) | 2020-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109631486A (en) | A kind of food monitoring method, refrigerator and the device with store function | |
| CN106705564B (en) | The control method and refrigerator of refrigerator | |
| CN107423421A (en) | Menu recommends method, apparatus and refrigerator | |
| CN105184719A (en) | Foodstuff monitoring method and device | |
| CN106885447A (en) | A kind of food management method of refrigerator | |
| CN113551478B (en) | Method and device for updating residual shelf life of food and refrigerator | |
| EP3734205A1 (en) | Intelligent refrigerator and information reminder method based on intelligent refrigerator | |
| CN108766529A (en) | A kind of intelligence recipe recommendation method and system | |
| CN111382660A (en) | Augmented reality feedback for appliance inventory | |
| WO2022041070A1 (en) | System for monitoring food preservation duration and smart refrigerator | |
| CN108061415A (en) | A kind of intelligent refrigerator and the method based on intelligent refrigerator food control | |
| CN112329509A (en) | Food material expiration reminding method and device, intelligent refrigerator and storage medium | |
| CN109780813A (en) | Refrigerator control method and device, storage medium and refrigerator | |
| WO2021057769A1 (en) | Method for viewing and tracking stored items | |
| CN114119552A (en) | Closed space freezing and refrigerating environment control system and method based on computer vision | |
| CN113566487A (en) | Food material storage system, refrigerator, food material management method and intelligent refrigerator system | |
| CN114544658A (en) | Method and device for determining deteriorated food, storage medium and electronic device | |
| WO2018045839A1 (en) | Shopping trigger method and device for intelligent refrigerator, and computer-readable storage medium | |
| CN113239780A (en) | Food material determining method and device, electronic equipment, refrigerator and storage medium | |
| CN113483525A (en) | Preservation equipment and food material management method | |
| CN113154780A (en) | Refrigerator dish storage temperature management method, refrigerator and storage medium | |
| CN111209849A (en) | Method and device for detecting freshness of food materials, food storage equipment and storage medium | |
| CN110274421A (en) | A kind of intelligent refrigerator | |
| CN110274435A (en) | Food control method and system in a kind of refrigerator based on effective period of food quality | |
| KR102543862B1 (en) | Object Recognition for Storage Structures |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190416 |
|
| RJ01 | Rejection of invention patent application after publication |