WO2023062880A1

WO2023062880A1 - Area management apparatus and program

Info

Publication number: WO2023062880A1
Application number: PCT/JP2022/024657
Authority: WO
Inventors: 大介林; 圭介稲田; 泰久森; 功記加藤
Original assignee: 日立グローバルライフソリューションズ株式会社
Priority date: 2021-10-12
Filing date: 2022-06-21
Publication date: 2023-04-20
Also published as: CN117716436A; JP2023057695A

Abstract

Conventionally, suggestions regarding food in storage have entailed the possibility of causing excess inventory due to a lack of consideration of nutrients in the current inventory, or it has been unknown how long shopping shortfalls could be prevented due to a lack of consideration of past nutrient intake.　A refrigerator 1 which stores food, the refrigerator being one example of an area management apparatus according to the present invention for solving the aforementioned problem, comprises: an identification unit 124 which identifies usage status of management areas in the refrigerator 1; a health index calculation unit 126 which specifies nutrients which are recommended for intake by a prescribed user and calculates a health index corresponding to these nutrients on the basis of the aforementioned usage status; and a display unit 128 which displays the health index.

Description

Area management equipment and programs

The present invention relates to an area management device that performs information processing regarding accommodation or storage (hereinafter simply storage) of articles including foodstuffs. Among these, it particularly relates to a technology for providing information according to usage conditions including internal inventory. Note that the area management equipment includes not only the warehouse for storage, but also a computer, a mobile terminal, a web server, and an internal management system (area management system) including at least one of these for managing it. . Further, the storehouse may be any storehouse that can store articles, and includes refrigerators, pantries, food storage such as so-called floor storage, food storage rooms, and article shelves.

　Currently, there is a technology that proposes insufficient nutritional value and ingredients in response to the growing health consciousness. For example, there is a technique for obtaining an ingested nutrient from a menu ingested by a user for a certain period of time in the past, calculating a lack of nutrients from the difference from the nutrient to be ingested, and calculating an appropriate menu (Patent Document 1). Patent Literature 1 describes that "the total nutritional value of a menu used during an arbitrary period is calculated, and a menu that allows intake of the nutritional value that is lacking in terms of nutritional balance is presented."

In addition, there is a technology that considers the nutrients in the current inventory and proposes the purchase of low-cost ingredients that supplement the lack of nutrients (Patent Document 2). In Patent Document 2, "presentation of foodstuffs for priority consumption based on a comprehensive cost performance evaluation of ingredients in stock and presentation of recommended foodstuffs for purchase that optimizes cost performance, which is the balance between the price and the supply of insufficient nutrients for consumers, A refrigerator that can be realized at the same time.”

JP-A-2002-251518 JP-A-2006-90646

Here, when making proposals for nutritional deficiencies and food ingredients, it is possible to make proposals that are more suitable for the user by considering the usage situation such as storage inventory (inside inventory). However, US Pat. No. 5,700,001 does not consider the nutrients in the current inventory, which could be the cause of overstock. In addition, in Patent Document 2, the past intake of nutrients is not taken into consideration, and although it is possible that excess inventory can be suppressed, it is unclear how long the shortfall in purchases can be prevented. As described above, in

Patent Literatures

1 and 2, it was difficult to make proposals suited to the intake situation of the user.

In order to solve the above problems, the present invention uses a management barometer that indicates the user's nutritional status, taking into consideration the nutrients that the user is recommended to take in the future. More preferably, the health index corresponding to the nutrient is calculated based on the nutrient that is recommended to be ingested according to the usage status of the storage. In addition, proposal information on recipes for ingesting nutrients is created according to health management barometers including health indexes or nutritional parameters.

More specifically, the present invention includes the following configurations (1) and (2).
(1) In an area management device that performs information processing on the storage of foodstuffs, a recognition unit that recognizes the usage status of the management area related to the storage, and based on the usage status, recommends nutrients to be taken by a predetermined user. An area management device having a health index calculation unit that identifies and calculates a health index corresponding to the nutrient, and an output unit that outputs the health index.
(2) In an area management device that performs information processing on food storage, a recognition unit that recognizes the usage status of the management area related to storage, and a predetermined user based on the usage status of the management area related to storage a health management barometer calculation unit that identifies recommended nutrients and calculates a health management barometer that indicates the nutritional status of the user; a health index calculation unit; and intake of the specified nutrients according to the health management barometer. An area management device having a proposal information generation unit for generating proposal information about a recipe for cooking, and an output unit for outputting the proposal information.

In addition, the present invention includes a health management method and a computer program using the area management equipment of (1) and (2) above. Also, the area management devices of (1) and (2) include computers such as storages and mobile terminals. Furthermore, systems and subsystems including the repositories and computers described above are also included in the area management equipment of the present invention.

According to the present invention, it is possible to make proposals regarding nutrient intake in line with the situation including the user's intake.

Problems, configurations and effects other than the above will be clarified by the following description of the embodiment.

1 is an overall configuration diagram of an in-chamber management system in an embodiment; FIG. 4 is a flow chart showing an internal storage management process in Embodiment 1. FIG. FIG. 10 is a diagram showing an example of a fisheye camera image CG captured by a camera; FIG. 4 is a diagram showing an example of an image obtained by converting a fisheye camera image CG into a planar image; It is a figure which shows an example of the foodstuff nutritional component table used by an Example. It is a figure which shows the example of a display of proposal information. It is a figure which shows an example of the order screen displayed on a portable terminal. It is a flowchart which shows the in-fridge management process in an Example. FIG. 10 is a diagram showing an example of a healthy recipe displayed in Example 2; It is a figure which shows the example which provided the camera in the chamber|chamber interior of the refrigerator main body. FIG. 3 is a diagram showing a configuration of a mobile terminal 7 that performs in-fridge management processing;

Hereinafter, embodiments of the present invention will be described based on the drawings. In this embodiment, a repository is used as the area management device. In this embodiment, the usage status of the storage is recognized, and based on this, nutritional parameters and health indices are calculated as a management barometer indicating the nutritional status of the user according to the nutrients that the user should take. Based on the calculated management barometer, information on recipes for ingesting the nutrients that the user should ingest is proposed. The information about this recipe includes recipes that are cooking contents, additional ingredients that are recommended to be added, and recipe cooking methods.

In the examples described later, a refrigerator will be used as an example of storage. However, the storehouse of this embodiment is not limited to the refrigerator, and includes an article shelf and the like. In addition, the storage area is not limited to specific cabinets, shelves, furniture, or home electric appliances, and can be an area for storing foodstuffs consumed by the user (hereinafter sometimes referred to as a management area). For example, it can be the user's entire home. In the embodiment, the inside of the storage is described as an example of the management area. A food material that is stored in a management area and becomes a management target is sometimes referred to as a controlled food material.

In addition, the camera will be used as an example of the shooting unit. However, the present embodiment is not limited to cameras, and can be widely applied to sensor information such as weight sensors and mycotoxin detection, IC tag information, character recognition of package information, and the like.

In addition, in the present embodiment, the recognition unit manages the management area, for example, recognizes the usage status of the foodstuffs in the storehouse and the foodstuffs consumed by the user. In the embodiment below, the recognition unit performs recognition using the results of the imaging by the imaging unit, but may be configured to perform recognition according to input from the user to the mobile terminal 7 such as a smartphone or the storage itself. . In addition, in the present embodiment and the following examples, foods will be described as an example of articles. In addition, foodstuffs may be materials, and may be cooked foodstuffs or seasonings. In addition, the storage management system (area management system) in each of the following embodiments, each device described later, and combinations thereof are included in the area management equipment of the present invention.

First, Example 1 will be described with reference to FIGS. 1 to 7. FIG. FIG. 1 is an overall configuration diagram of an internal warehouse management system (area management system) in Example 1 and Example 2 described later. The refrigerator 1 is connected to a portable terminal 7, a web server 8, and a computer 9 via a network CN to configure the refrigerator management system. Each device will be described below.

First, the refrigerator 1 includes a control unit 10 and a refrigerator main body 20 as a "storage main body".

Also, the portable terminal 7 as an external device is a terminal used by the user (user) of the refrigerator 1 . The portable terminal 7 displays the food order list transmitted from the refrigerator 1, as will be described later. The mobile terminal 7 can be realized by an information processing device such as a tablet, a smart phone, or a PC. In addition, although the main processing in Examples 1 and 2 is executed by the refrigerator 1 (control unit 10), it may be executed by the portable terminal 7, or may be executed by the refrigerator 1 and the portable terminal 7. . This will be described later with reference to FIG.

Also, the web server 8 includes, for example, a net supermarket and a recipe site. By communicating with these devices, the refrigerator 1 can purchase foodstuffs, obtain cooking methods, recipes, and the like. A computer 9 is a computer for distributing machine learning models and the like for providing information in the first and second embodiments.

Next, the details of the refrigerator 1 will be explained. A camera 50 for photographing the interior of the refrigerator is attached to the upper portion of the refrigerator body 20 . The position of the camera may be inside the refrigerator as well as outside the refrigerator. FIG. 10 is a diagram showing an example in which cameras 50a to 50e are provided inside the refrigerator body 20. As shown in FIG. The camera 50a and the camera 50b are examples installed above the door. Also, the camera 50c is an example installed in the upper part inside the main body. As described above, each camera can be installed at a position other than the top inside the refrigerator body 20 . An example of this is the position of the

cameras

50d and 50e. A camera 50d is provided at the center of the door, and a camera 50e is provided at the bottom of the refrigerator. Note that the installation positions and the number of cameras are not limited to the example in FIG. In particular, the number of installations may be one or more.

The control unit 10 that controls the refrigerator 1 includes, for example, a processor 11, a storage device 12, a communication unit (I/F in the figure) 14 connected to the network CN, and an I/O interface 13 (I/F in the figure). /O). The storage device 12 includes a main storage device composed of volatile or nonvolatile memory, and an auxiliary storage device composed of flash memory, hard disk drive, or the like.

A part or all of the computer programs and data stored in the storage device 12 can also be transmitted to the outside via the communication network CN. Conversely, the computer program and data can also be transmitted from the external computer 9 or the like to the storage device 12 via the communication network CN and stored.

Also, a storage medium MM such as a flash memory or hard disk drive can be connected to the control unit 10, and part or all of the computer program and data can be transferred between the storage device 12 and the storage medium MM.

The storage device 12 includes an imaging unit 121, an image conversion unit 123, a recognition unit 124, a table control unit 125, a health index calculation unit 126, a proposal information generation unit 127, a display unit 128, and an ordering unit 129. , and a predetermined computer program that implements the in-fridge control unit 130 is stored. The storage device 12 also includes an image buffer 122 . However, the image buffer 122 may have an independent structure.

Then, the processor 11 executes each of these computer programs to implement each functional unit (out of the above 121 to 130, excluding the image buffer 122). That is, each functional unit of the storage device 12 in FIG. 1 corresponds to the program. Therefore, each of these units can be read as each computer program, and the processor 11 realizes the processing and function of each unit described later by each computer program. However, each of these units may be realized by dedicated hardware, FPGA (Field Programmable Gate Array), or the like. Furthermore, these computer programs may be configured in one or less than the number shown. In this case, each functional unit can be configured as a computer module (simply called a module).

As described above, the processor 11 operates as a functional unit that provides a predetermined function by executing processing according to a computer program. For example, the processor 11 functions as the image conversion unit 123 by executing processing according to an image conversion program. The same is true for other computer programs. Furthermore, the processor 11 also operates as a functional unit that provides each function of a plurality of processes executed by each computer program. In this embodiment, the computer program is executed by the processor 11 and one processor, but may be executed by a plurality of processors.

The imaging unit 121 acquires camera images from the camera 50 via the I/O interface 13 and stores the acquired camera images in the image buffer 122 . The camera 50 as a "camera section" is configured as a fisheye camera. Camera 50 has, for example, a fisheye or wide-angle lens.

The image conversion unit 123 converts a camera image taken with a fisheye or wide-angle lens into a planar image. Since a well-known technique can be used when an image captured by a fisheye or wide-angle lens is developed into a planar image, the description thereof is omitted.

The recognition unit 124 is composed of a learning-based image recognition unit and a rule-based image recognition unit. The learning-based image recognition unit and the rule-based image recognition unit each recognize ingredients in the refrigerator from the converted planar image. The learning-based image recognition unit includes, for example, a pre-learned machine learning model such as deep learning. When a plane image is input, the learning-based image recognition unit outputs a recognition result of ingredients included in the plane image.

Also, when a plane image is input, the rule-based image recognition unit recognizes ingredients in the refrigerator based on the rule. The rule-based image recognition unit divides the input planar image into regions, labels the article for each region, and outputs the contents of the recognized label as text. Known techniques can be applied to these learning-based image recognition unit and rule-based image recognition unit.

Although the recognition unit 124 of this embodiment recognizes the usage status of the refrigerator 1 using an image, it is not limited to this as described above. In this case, the imaging unit 121, the image buffer 122 and the image conversion unit 123 can be omitted. Then, instead of these, a configuration for use in recognition by the recognition unit 124 is provided. For example, when a weight sensor is used, a functional unit is provided to perform processing for associating weight with food.

In addition, the table control unit 125 controls the contents of the ingredient nutritional component table T1 (see FIG. 5) that defines the nutritional components for each ingredient. That is, the table control unit 125 accesses the ingredient nutritional component table T1, and uses this result in other functional units. The food nutrient component table T1 is provided inside the table control unit 125 . Also, the nutritional components of the ingredient nutritional component table T1 are nutritional standards (standard values) for calculating the health index by the health index calculator 126. FIG. As an example of these nutritional components, the values in the "Japanese Food Standard Ingredients Table 2020 Edition" by the Ministry of Education, Culture, Sports, Science and Technology are taken as an example.
However, the ingredient nutritional component table T1 is not limited to this example, and a ingredient manufacturer or the like may independently create nutritional component values for each ingredient.

In addition, the health index calculation unit 126 calculates the health index from the in-fridge health index (management area health index) and the consumption health index. This calculation includes preparation. This preparation consists of the following two stages. In the first step, the health index calculation unit 126 refers to the recognized nutrients of the ingredients, that is, the values of the nutritional components from the table control unit 125 . Take proteins, fats, and carbohydrates as examples of these nutrients. However, Examples 1 and 2 are also broadly applicable to nutrients such as vitamins and minerals.

In the second step, the health index calculation unit 126 sums up the recognized food ingredients within the categories of protein, fat, and carbohydrates, and calculates the ratio of total protein, fat, and carbohydrates. As this ratio, for example, it is possible to refer to the ratio to the total energy amount when protein, lipid, and carbohydrate of the foodstuff of interest are each converted into energy amount.

In calculating the in-fridge health index, the health index calculator 126 compares the total ratio of protein, fat, and carbohydrate with a reference value to calculate each in-fridge nutrition parameter. In this example, as an example of the standard value, the value of “Dietary Reference Intakes for Japanese People” (2020 edition) by the Ministry of Health, Labor and Welfare (Equation 1-1) is used.
p*: f*: c* = 13-20: 20-30: 50-65 (Formula 1-1)
however,
p* + f* + c* = 100 (equation 1-2)
Here, p* is the reference intake of protein, f* is the reference intake of fat, and c* is the reference intake of carbohydrate. However, the standard value is not limited to the value of the Ministry of Health, Labor and Welfare, and the standard value may be set independently by a food material manufacturer or the like. The user's race, sex, age, and other information may be input and set in consideration of such information. Also, setting of the reference value of each nutrient may be received from the user.

Here, the nutritional parameters are information indicating the nutrients of the ingredients to be managed, such as the ingredients in the refrigerator. Therefore, the higher the value, the more the user can be suggested to purchase a large amount of foodstuff containing the nutrient. Also, the nutrition parameters consist of in-chamber nutrition parameters and consumed nutrition parameters. A nutrient parameter of a certain nutrient indicates whether two or more kinds of nutrients as a whole is a ratio of a certain nutrient to the whole. In this example, as described above, three types of nutrients, proteins, lipids, and carbohydrates, will be described as a whole.

The in-fridge health index is calculated by subtracting from 100 the absolute values of the in-fridge nutrition parameters of protein, fat, and carbohydrate. (Equation 2-1) to (Equation 2-5) are used for this calculation. Note that the in-fridge health index is a value between 0 and 100 as expressed by (Equation 2-6). However, the formulas are not limited to (Formula 2-1) to (Formula 2-6), and other formulas may be used.
In-fridge health index = 100-|np _ip |-|np _if |-|np _ic | (Formula 2-1)
np _ip =x _t2 -p* ... (Equation 2-2)
np _if =y _t2 - f* ... (equation 2-3)
np _ic =z _t2 -c* (equation 2-4)
however,
_xt2 + _yt2 + _zt2 = 100 (Equation 2-5)
0≦internal health index≦100 (Formula 2-6)
Here, np _ip is the protein nutrient parameter, np _if is the lipid nutrient parameter, and np _ic is the carbohydrate nutrient parameter. Also, x _t2 is the currently recognized protein ratio of the inside food material, y _t2 is the currently recognized fat ratio of the inside food material, and z _t2 is the currently recognized carbohydrate ratio of the inside food material.

In other words, the absolute value of the in-fridge nutrition parameter for a certain nutrient suggests how far the nutrient ratio of the foodstuff stored in the management area deviates from the reference ratio. For this reason, the absolute value of the in-fridge nutrition parameter of a certain nutrient is the value of the user's certain nutrient from the present until the completion of the intake of these ingredients, assuming that the user has taken all the ingredients in the management area. It can be interpreted that it suggests how much the intake balance of
The in-fridge health index is obtained by subtracting the sum of the absolute values of the in-fridge nutrition parameters for each nutrient from the arbitrarily determined maximum value (100 in this embodiment). Therefore, the in-fridge health index is the nutrient balance of the ingredients stored in the area to be managed, that is, the nutritional value of the nutrients that the user would take if he lived with the currently managed ingredients. suggest a degree of balance.

Also, the consumption health index is calculated by subtracting from 100 the absolute value of each consumption nutritional parameter of protein, fat, and carbohydrate. (Formula 3-1) to (Formula 3-5) are used for this calculation.
Note that the consumption health index is a value of 0 or more and 100 or less as represented by (Equation 3-6). However, the formulas are not limited to (Formula 3-1) to (Formula 3-6), and other formulas may be used.

First, controlled food materials that have been consumed from an arbitrary timing in the past to the present are specified, and the total energy amount of each nutrient (protein, lipid, and carbohydrate in this embodiment) contained in these consumed controlled food materials is calculated. . That is, the total energy amount of each nutrient that the users would have ingested from any timing in the past to the present is calculated. Then divide by total energy for each nutrient. As a result, it is possible to calculate the ratio of each nutrient that the users would have ingested from any timing in the past to the present. Let us call these the protein, lipid and carbohydrate consumption nutritional parameters respectively and define them as np _cp , np _cf and np _cc respectively. The information on these consumed controlled foodstuffs can be recognized by a camera or the like provided in the storage. Details are described separately.

i.e.
Consumption health index = 100-|np _cp |-|np _cf |-|np _cc |
np _cp = (proportion of protein ingested by users from arbitrary timing in the past to the present) (Formula 3-2)
np _cf = (percentage of lipids ingested by users from arbitrary timing in the past to the present) (equation 3-3)
np _cc = (percentage of carbohydrates ingested by users from arbitrary timing in the past to the present) (Formula 3-4)
np _cp +np _cf +np _cc =100 (equation 3-5)
0≦consumed health index≦100 (equation 3-6)
In other words, the absolute value of the consumed nutritional parameter of a certain nutrient is the amount consumed by the person who consumes the food material in the management area, that is, the user or the person who lives with the user, from the past timing designated by the user to the present. Suggests the ratio of waxy nutrients to total energy intake.

The consumption health index is obtained by subtracting the sum of the absolute values of the consumption nutritional parameters of each nutrient from the arbitrarily determined maximum value (100 in this embodiment). For this reason, the consumption health index suggests the degree of balance of nutrients that would have been consumed by a person consuming the food material in the management area from the past timing designated by the user to the present.

Also, the health index is calculated as, for example, an arithmetic mean of the in-fridge health index and the consumption health index.
(Formula 4) is used for this calculation.
Health index = (inside health index + consumption health index)/2 (Equation 4)
The evaluation of this health index is A (80-100): Fairly good, B (60-79): Good, C (40-59): Normal, D (20-39): Needs improvement, E (0- 19): Considerable improvement is required. This is done in 5 steps, but it is not limited to this way and the manufacturer may set it independently. For this reason, the health index calculated using the indoor health index and the consumption health index is the nutritional balance expected to be ingested by the users from the past timing designated by the user to the completion of the current intake of the controlled ingredients. Suggest. In this way, it is possible to evaluate the nutritional balance by taking into account both the balance of nutrients ingested over a predetermined period of time in the past and the current nutritional balance of controlled foodstuffs.

The health index calculated using the indoor health index and the consumption health index is not limited to calculation by arithmetic mean, but can be calculated by a method that does not interfere, such as a geometric mean or some kind of averaging calculation.

In addition, the inside health index is a value calculated based on the nutrients of each ingredient recognized inside the refrigerator. According to the refrigerator health index, for example, it is possible to suggest additional ingredients to the user so as to maintain the health and nutrition balance in the refrigerator.

As an example of calculating the health index in the refrigerator, if the ratio of nutrients in the ingredients in the refrigerator recognized at a predetermined timing such as the current time is protein: fat: carbohydrate = 20: 35: 45, the reference value ratio of nutrient intake p* , f*, and c*, protein was within the reference range, fat was 5 above the upper limit, and carbohydrate was below the lower limit by 5. In this case, even if the standard value for fat is set to the upper limit and the standard value for carbohydrates to the lower limit, proposal information is created to purchase high-carbohydrate foodstuffs in order to maintain a healthy balance in the refrigerator. , suggest this. The in-fridge nutrition parameters for this example ratio are np _ip =0, np _if =−5, np _ic =5. Also, the in-fridge health index is 100-5-5=90.

In this way, by proposing ingredients that should be acquired so as to raise or maintain a high in-fridge health index, the nutritional balance of controlled ingredients is optimized. As a result, it becomes easier to cook a menu with a suitable nutritional balance using controlled ingredients without shortage, and it becomes easier to reduce the necessary number of times of shopping.

The consumption health index is a value calculated from the nutrients of each ingredient that is determined to have been ingested by the user within a certain period of time from the present to the past. According to the consumption health index, for example, it is possible to suggest to the user foods that should be consumed or a preferable menu based on the consumption nutritional balance of time-series differences.
The user can freely select N days as the time series difference from the previous day, 2 or 3 days before, or 1 week before. Therefore, it is possible to propose a purchase in which the inventory in the warehouse is kept until N days specified by the user.

As an example of calculating this consumption health index, the user sets the time-series difference to the previous day, and the ratio of the nutrients of the ingredients consumed on the previous day to the total energy intake is protein: lipid: carbohydrate = 23: 37: 40. Consider the case. When compared to the reference value ratios p*, f*, c* of nutrient intake, protein was 3 above the upper limit of reference, fat was 7 above the upper limit of reference, and carbohydrate was 10 below the lower limit of reference. It can be seen that the carbohydrate intake ratio is low. In this case, high-carbohydrate foodstuffs or menus can be proposed in order to adjust the consumption nutritional balance of time-series differences. The nutrient consumption parameters are np _cp =3, np _cf =7, np _cc =-10. Also, the consumption health index is 100-3-7-10=80.

In addition, the health index is calculated based on inventory food information (inside health index) and food consumption history (consumption health index). For this reason, the health index is an index corresponding to nutrients, and more preferably, information that is easier to understand based on nutritional parameters.

In addition, based on the health index and nutritional parameters, that is, the health management barometer, we will propose the amount of ingredients to be additionally purchased in consideration of the amount of inventory in the refrigerator and the nutrients that should be taken. The purpose of the health index is to make it easier to secure the necessary and sufficient amount of foodstuffs that should be ingested from a nutritional point of view in the future in the refrigerator. Thus, the health care barometer indicates the user's nutritional status, and more preferably includes health index and nutritional parameters.

Taking the values of the in-fridge health index and consumption health index given above, the health index is: in-fridge health index (=90)+consumption health index (=80)/2=85. The nutrition parameters are np _p =np _ip +np _cp =3, np _f =np _if +np _cf =2, np _c = _npic +np _cc =-5 by adding the nutrition parameters in the refrigerator and the consumption nutrition parameters. be.

Note that the health index calculation unit 126 can calculate nutrition parameters, which are a type of health management barometer similar to the health management index, and can be understood as a type of health management barometer calculation unit.

In addition, proposal information generating section 127 includes additional ingredients determining section 1271 and healthy recipe information generating section 1272 . In order to increase the health index, the additional foodstuff determination unit 1271 proposes the amount of foodstuffs to be additionally purchased in consideration of the stock amount in the refrigerator and the nutrients to be ingested based on the nutritional parameters. As a result, it becomes easier to secure the necessary and sufficient amount of food materials to be ingested in the future in terms of nutrition in the refrigerator.
In the case of the example given above, the health index is 85 and the nutritional parameters are np _p =3, np _f =2, np _c =-5. Suggest a purchase, with the intention of increasing the health index. An example of the proposed additionally purchased ingredients includes eggs and the like.

In addition, the healthy recipe information generation unit 1272 displays recipes that can be made from ingredients in the refrigerator, and proposes recipes that increase the health index. Regardless of the health index, the index used for additional purchase may be either the indoor health index or consumption health index, or may be used in combination. This makes it possible to propose a wide range of recipes that can effectively ingest preferable nutrients, suppressing food loss and making it easier to create recipes. The recipe information may be acquired from the Internet, or a manufacturer or the like may independently create a table. Recipe information can be obtained by a known technique. It is assumed that the recipe information will be displayed after being rearranged in order of health index, but this is not the only option, and the recipe information may be displayed in a list without prioritization.

It should be noted that the proposal information generation unit 127 may further include a cooking method identification unit that identifies the cooking method of the dish indicated in the recipe. Furthermore, the proposal information generation unit 127 may be configured to include at least one of the additional ingredient determination unit 1271, the healthy recipe information generation unit 1272, and the cooking method identification unit. It is desirable that the cooking method specifying unit obtains the cooking method from an external device, as with the healthy recipe information generating unit 1272 .

In addition, the display unit 128 displays the health index, the refrigerator health index, the consumption health index, the current nutritional balance in the refrigerator, the nutritional balance of ingredients consumed in the past, the ingredients in the refrigerator, additional purchased ingredients, healthy recipes, and the like. 1 is displayed on a display device such as a liquid crystal display. It should be noted that it is not necessary to display all of the items described above, and only one of them may be displayed. Here, the display unit 128 may be implemented as an output unit such as the I/O interface 13 or the I/F 14 .

In addition, the ordering unit 129 displays a list of foodstuffs that are proposed to be additionally purchased in order to improve the health index on a display device such as the mobile terminal 7 or the liquid crystal display of the refrigerator, and automatically orders the displayed foodstuffs. Alternatively, the user may press an order button to semi-automatically place an order for ingredients from an online supermarket. The index for placing an order may be not only the health index but also the in-fridge health index and consumption health index. Note that the ordering unit 129 may be omitted without being included in the processing.

In addition, the internal control unit 130 controls the temperature and humidity inside the refrigerator 1 by controlling a motor and a compressor (not shown).

Next, images captured by the camera 50 will be described with reference to FIGS. 3 and 4. FIG. FIG. 3 is a diagram showing an example of a fisheye camera image GC captured by the camera 50. As shown in FIG. The camera 50 of this embodiment is a fish-eye lens, and FIG. 4 shows a fish-eye camera image GC taken with this lens. Here, it is difficult to cause the recognition unit 124 to recognize the ingredients by using the fisheye camera image CG as it is. That is, in the learning phase of the machine learning model, since learning is performed using undistorted images, it is difficult to recognize the fisheye camera image GC that is distorted by the fisheye or wide-angle lens 52 as it is.

For this reason, the image conversion unit 123 converts an image containing distortion captured with a fisheye lens into a planar image. For this reason, learning is performed in cooperation with the computer 9 described above. Here, FIG. 4 is a diagram showing an example of an image obtained by converting the fisheye camera image GC into a planar image. For this purpose, the image conversion unit 123 generates the recognition image G30 by synthesizing the distortion-removed right door developed image G32, front developed image G33, left door developed image G34, and upper developed image G31.

As a distortion removal method, a known camera calibration technique may be used. As an example, using an image whose pattern after distortion correction is known in advance, such as a checkboard, feature points before and after correction are extracted. Parameters may be estimated.

This completes the description of the image captured by the camera 50, and next describes the processing of the present embodiment.

FIG. 2 is a flow chart showing an example of the in-fridge management process performed by the control unit 10. As shown in FIG. First, in step S<b>16 , the imaging unit 121 acquires a visible light image captured by the camera 50 .
In addition, it is desirable that the photographing unit 121 stores the acquired visible light image in the image buffer 122 as the camera image GC.

Next, in step S17, the image conversion unit 123 reads the camera image GC captured in step S16 from the image buffer 122 and develops the fisheye image into a planar image.

Next, in step S18, the learning-based image recognition unit of the recognition unit 124 receives the input of the recognition image G30 generated by the image conversion unit 123, and executes image recognition of the ingredients using the machine learning model obtained from the computer 9. Let The rule-based image recognition unit of the recognition unit 124 receives the input of the recognition image G30 generated by the image conversion unit 123, performs image recognition of ingredients on a rule basis, and outputs the content of the recognized label. Note that the processing order of learning-based image recognition and rule-based image recognition is not limited to the above, and they may be executed in parallel.

Next, in step S19, the health index calculator 126 calculates the health index using the ingredient nutritional component table T1. It is desirable that this health index includes the in-fridge health index and consumption health index. FIG. 5 is a diagram showing an example of the ingredient nutritional component table T1.
The nutrient ratio of the food in the refrigerator recognized by the recognition unit 124 is calculated based on the food nutrient component table T1. For this reason, in this step, the health index calculator 126 calculates the in-fridge health index based on the nutrients of each ingredient recognized from the inside of the refrigerator. The in-fridge health index is calculated based on the current user's food inventory information. The purpose of the in-fridge health index is to keep the in-fridge nutritional balance so that additional ingredients can be suggested to the user based on the consumption health index.

Next, in step S20, the health index calculation unit 126 calculates the consumption health index based on the consumption history of foodstuffs that are determined to have been consumed by the user within a certain period from the present to the past. For this reason, the health index calculator 126 can calculate the consumption health index based on the ingredients consumed by the user from the time-series difference (change) of the ingredients in the refrigerator. For this calculation, it is desirable that the recognizing unit 124 identifies the consumable foods that are consumed by the user from time-series changes in the foods in the refrigerator.

Here, the purpose of the consumption health index is to suggest to the user which foodstuffs are better to consume from the consumption balance of time-series differences in combination with the refrigerator health index. The user can freely select N days as the time-series difference from the previous day, 2 or 3 days before, or 1 week before. Therefore, it is possible to propose a purchase in which the inventory in the warehouse is kept until N days specified by the user.

Next, in step S21, the health index calculation unit 126 calculates a health index based on the user's current stock food information (in-fridge health index) and food consumption history (consumption health index). The health index is an index corresponding to the nutrients recommended for intake by the user, and more preferably information for facilitating the securing of necessary and sufficient amounts of ingredients to be nutritiously ingested in the future in the refrigerator. desirable.

Next, in step S22, the additional foodstuff determination unit 1271 of the proposal information generation unit 127 identifies additional foodstuffs. As an example of this, the additional foodstuff determination unit 1271 identifies foodstuffs that are recommended to be purchased and their amounts in consideration of the inventory amount in the refrigerator and the nutrients to be taken, based on nutritional parameters, in order to increase the health index. and output this. As a result, it becomes easier to secure the necessary and sufficient amount of food materials to be ingested in the future in terms of nutrition in the refrigerator.

Here, one specific example of step S22 is shown below. The additional foodstuff determination unit 1271 grasps the shortage of each nutrient. For example, protein: 30 deficiency, lipid: 70 deficiency, etc.). Next, the additional foodstuff determination unit 1271 searches foodstuffs that match the most deficient nutrient (lipid in the above description) from the foodstuff nutritional component table T1. For example, nuts (lipid: 47.6) are searched.

Then, the additional foodstuff determination unit 1271 adds the nutrients of the searched foodstuff to calculate a new deficiency value. In the above example, protein: 10.2 deficit, lipid: 17.6 deficit.

Next, the additional ingredient determination unit 1271 searches for a new ingredient with the most deficient nutrients. In this case, the additional foodstuff determination unit 1271 searches foodstuffs satisfying the fat: 17.6 from the foodstuff nutritional component table T1. In addition, when any nutrient is excessive in the search for the food ingredient with the most deficient nutrient, the additional food ingredient determination unit 1271 may be configured to cancel the searched food ingredient and search for the food ingredient with the next highest nutrient content. , may be used as a search result. Further, it may be configured such that the user decides whether to cancel or not.

In addition, the additional foodstuff determination unit 1271 may specify foodstuffs by performing calculations using a combination optimization algorithm based on the ratio of each nutrient deficiency. In this case, it is desirable to calculate so that the amount (number, weight) of the ingredients is minimized.

Next, in step S23, the display unit 128 displays the health index, the health index in the refrigerator, the consumption health index, the current nutritional balance in the refrigerator, the nutritional balance of ingredients consumed in the past, the ingredients stored in the refrigerator, additional ingredients, healthy recipes, The cooking method or the like is displayed on the display device or the mobile terminal 7 . It should be noted that it is not necessary to display all of the items described above, and only one of them may be displayed.

Here, an example of this display will be explained. First, FIG. 6 is a diagram showing a display example of proposal information. This display may be performed by either the display device or the mobile terminal 7, but the case where the display is performed by the mobile terminal 7 will be described. The mobile terminal 7 can be realized by a so-called smart phone as described above. The mobile terminal 7 is not limited to a smart phone, and may be a tablet terminal, a wearable terminal, a laptop terminal, or the like. Note that the refrigerator management application running on the mobile terminal 7 communicates with the refrigerator 1 via the network CN on a regular or irregular basis. This application (in-fridge management application 720) will be described later with reference to FIG.

Next, FIG. 6 will be explained. The display example shown in FIG. In the upper left part 61, the health index and overall evaluation are displayed. In the lower left part 62 on the left side, information on food ingredients in the refrigerator (interior health index) and nutritional balance in the refrigerator are displayed. Consumed foodstuff information (consumption health index) and consumption nutritional balance are displayed in the left lower right portion 63 . In the right part 64, in order to increase the health index, foodstuffs that are newly proposed to be additionally purchased are displayed in consideration of the inventory amount in the refrigerator and the nutrients to be ingested. Here, the suggested information to be displayed is not limited to that shown in FIG.

It should be noted that among the foodstuffs containing a lot of nutrients proposed for additional purchase, foodstuffs with a high past consumption rate of the user may be displayed preferentially or only displayed. As an example of this, a case where an additional ingredient different from that shown in FIG. 6 is proposed will be described. In this case, an additional purchase of protein is suggested, and natto and eggs containing a lot of protein are often eaten. In such a case, foods containing a lot of protein are displayed in the order of sausage, pork, ham, natto, and eggs. However, it is assumed that the user has a tendency to eat a lot of natto and eggs based on the food consumption history of the user. In this case, in the next display, natto, eggs, ham, pork, and sausage may be displayed in this order, or only natto and eggs may be displayed. Now, the explanation of the display example of the proposal information is finished, and returning to FIG. 2, the explanation of the in-fridge management process is continued.

Next, in step S24, the ordering unit 129 displays a list of foodstuffs to be additionally purchased on the mobile terminal 7 or the display device, and automatically orders the displayed foodstuffs, or the user presses an order button to place an online order. You can semi-automatically order ingredients from a supermarket. Note that the ordering unit 129 may be omitted without being included in the processing.

Here, a display example in this step will be explained. FIG. 7 is a diagram showing an example of an order screen displayed on the mobile terminal 7. As shown in FIG. This ordering screen is displayed on the touch panel 703 as an ingredient ordering list 73 . This foodstuff order list 73 includes foodstuffs that have been received from the refrigerator 1 and are proposed for additional purchase.

An order destination 74 and an order button 75 are displayed below the ingredients order list 73 . When the operation of the ordering button 75 by the user of the refrigerator 1 is detected, the contents of the ingredients ordering list 73 are sent to the ordering party 74 . Although FIG. 7 shows an example in which a single supplier 74 is displayed, a plurality of suppliers 74 may be displayed so that the user can select one when ordering. Alternatively, an orderer may be selected from a plurality of pre-registered orderers according to the type of ingredients.

Also, the timing at which the additional food material determination unit 1271 generates the food material order list 73 is not limited to immediately after the camera 50 captures the image of the inside of the refrigerator, but can be performed regularly or irregularly.

Although an example has been shown in which image recognition of foodstuffs and generation and transmission of the foodstuff order list 73 are performed after the photographing unit 121 has photographed the inside of the refrigerator 1, the present invention is not limited to this. For example, the control unit 10 holds the food ordering list 73 calculated from the recognition result of the camera image GC in the storage device 12, and when receiving a request for the food ordering list 73 from the mobile terminal 7, the latest food ingredients The order list 73 may be transmitted to the mobile terminal 7 . Thereby, the user of the refrigerator 1 can quickly grasp the ingredients to be additionally purchased by referring to the portable terminal 7 even when the user is out.

Furthermore, the machine learning model of the recognition unit 124 can be updated to the latest machine learning model according to changes or additions to food packages. For example, a machine learning model received from a server (not shown) may be updated as the machine learning model of the recognition unit 124 .

As described above, in this embodiment, the health index is calculated using the user's product consumption history (consumption health index) and the current user's product information in the storage (inside health index). It is possible to propose the amount of ingredients to be additionally purchased in consideration of the stock amount of food and the nutrients to be taken. As for the item consumption history (consumption health index), the user can freely select the number of days of the past time-series difference, so that it is possible to propose purchases in which the stock in the storehouse is kept until after the selected number of days.

　Example 2 that proposes a healthy recipe for improving the health index will be described with reference to Figures 8 and 9. In this embodiment, differences from the first embodiment will be mainly described. In the present embodiment, among recipes that can be made from ingredients in the refrigerator, recipes with a higher health index are proposed to the user.

FIG. 8 is a flow chart showing an example of the in-fridge management process executed by the control unit 10 in this embodiment. Steps S16 to S18 and S24 of this process are the same as steps S16 to S18 and S24 described with reference to FIG. 2, so description thereof will be omitted. In this process, new steps S25 and S26 are added. Moreover, the same configuration as that of the first embodiment can be adopted for the configuration of the present embodiment.

First, in step S25, the healthy recipe information generation unit 1272 rearranges and displays recipes that can be made from the recognized ingredients in the refrigerator in order of health index, and proposes a recipe with a higher health index. This will make it easier to create healthy recipes that reduce food loss by proposing a wealth of healthy recipes that allow for effective intake of favorable nutrients. In addition, instead of sorting in order of health index, you may sort in order of in-fridge health index or consumption health index.

Next, in step S26, the display unit 128 causes the mobile terminal 7, the display device, etc. to display the ingredients in the refrigerator and healthy recipes in addition to the health index, additional ingredients, and the like. FIG. 9 is a diagram showing an example of the healthy recipe list 65 displayed in this step. In FIG. 9 it is divided into a left side 66 and a right side 67 . A list of ingredients in the refrigerator is displayed on the left side 66 by the display unit 128 . In addition, the display unit 128 displays the recipes that can be made from the ingredients in the refrigerator on the right side 67 in descending order of the health index.

In addition, the user can freely specify the type of health recipe according to health-oriented, stamina-oriented, event, etc. Cooking type designation 1, cooking type designation 2, and cooking type designation 3 are given as examples. The food type designation 1 places emphasis on balance, and includes staple foods, main dishes, side dishes, soups, milk/dairy products, and fruits. For the cooking type designation 2, fish-centered, meat-centered, vegetable-centered, etc. can be selected. Cooking type designation 3: Health-focused: When couples want to be healthy every day, Health-focused: When you want to eat today without worrying about your health index, Emphasis on stamina: Children who came home from club activities You can also choose when you want to eat a lot, events: when you want to celebrate, such as birthdays or passing exams.

The present embodiment configured in this way also has the same effect as the first embodiment. Furthermore, in this embodiment, by proposing recipes that increase the health index of ingredients in the refrigerator, it is possible to propose abundant recipes that allow effective intake of desirable nutrients, thereby suppressing food loss and making it easier to create recipes.

Although the inside management processing of each embodiment described above is executed by the control unit 10 of the refrigerator 1, it can also be executed by another device. A modified example in which the in-fridge management process is performed by the mobile terminal 7 will be described below. FIG. 11 shows the configuration of the mobile terminal 7 that performs the in-fridge management process of this modified example. The mobile terminal 7 of this example has a processor 701 , a storage device 702 , a touch panel 703 and a communication section 704 . The mobile terminal 7 can be realized by an information processing device (computer) such as a smart phone, as described above.

The processor 701 and storage device 702 have the same functions as the control unit 10 shown in FIG. Also, the touch panel 703 functions as an input/output unit. Furthermore, the communication unit 704 is connected with the network CN. This connection may be wireless or prioritized.

Here, the internal storage management application 720 (internal storage management program) that is stored in the storage device 702 and additionally executes the processing of this modified example will be described. The refrigerator management application 720 includes a recognition module 721 , a table control module 722 , a health index calculation module 723 , a proposal information generation module 724 , an order module 725 and an interior control instruction module 726 . Also, the proposal information generation module 724 is composed of an additional ingredients determination module 7241 and a healthy recipe information generation module 7242 . These are configured as one computer program (application), but each module may be configured as an independent computer program, or some modules may be collectively implemented as a computer program.

Each of these modules also performs the same functions as the functional units shown in FIG. That is, it has the following correspondence.
Recognition module 721: Recognition unit 124
Table control module 722: table control unit 125
Health index calculation module 723: health index calculation unit 126
Proposal information generation module 724: Proposal information generation unit 127
Additional ingredient determination module 7241: Additional ingredient determination unit 1271
Healthy recipe information generation module 7242: Healthy recipe information generation unit 1272
Ordering Module 725: Ordering Unit 129
In-fridge control instruction module 726: In-fridge control section 130
As described above, each module executes the same processing as the corresponding functional unit, but it is desirable that the internal control instruction module 726 further manages the usage status of food materials in the refrigerator.
For example, the internal control instruction module 726 acquires information read from a user's input or food code, and the recognition unit 124 recognizes the corresponding food.

Also, it is desirable that the refrigerator management application 720 be delivered to the mobile terminal 7 via the network CN. Therefore, the network CN will be implemented on the Internet.

According to each of the above embodiments and modified examples, the health index is calculated using the user's product consumption history (consumption health index) and the current user's product information in the storage (in-store health index), Considering the amount of stock in the refrigerator and the nutrients to be ingested, the amount of ingredients to be purchased additionally can be proposed. As for the item consumption history (consumption health index), the user can freely select the number of days of the past time-series difference, so that it is possible to propose purchases in which the stock in the storehouse is kept until after the selected number of days.

Although the description of the present invention is finished above, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, each of the embodiments described above has been described in detail for better understanding of the present invention, and is not necessarily limited to those having all the configurations described above.

It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. A part of the configuration of each embodiment can be deleted, added with another configuration, or replaced with another configuration.

Some or all of the above configurations, functions, processing units, processing means, etc. may be realized by hardware, for example, by designing them as integrated circuits. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files for realizing each function is stored in the storage device. This storage device includes storage devices such as non-volatile semiconductor memories, hard disk drives, SSDs (Solid State Drives), or computer-readable non-temporary data storage media such as IC cards, SD cards, and DVDs.

In addition, the control lines and information lines indicate what is considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In fact, it may be considered that almost all configurations are interconnected.

Furthermore, the above-described embodiments can be combined as appropriate, and combinations of those embodiments are also included in the scope of the present invention.

1: refrigerator, 7: mobile terminal, 8: web server, 9: computer, 10: control unit, 12: storage device, 121: photographing unit, 20: refrigerator body, 50: camera, 123: image conversion unit, 124: Recognition unit 125: Table control unit 126: Health index calculation unit 127: Proposal information generation unit 128: Display unit 129: Ordering unit

Claims

In an area management device that performs information processing on food storage, a recognition unit that recognizes the usage status of the storage management area;
a health index calculation unit that identifies nutrients that are recommended to be taken by a predetermined user based on the usage status and calculates a health index according to the nutrients;
An area management device having an output for outputting the health index.
The area management device according to claim 1,
The recognition unit, as the usage status,
recognizing the consumed consumable foodstuffs among the managed foodstuffs stored in the management area and the foodstuffs stored in the management area;
The health index calculation unit is an area management device that calculates the health index using a management area health index corresponding to the nutrients of the controlled food and a consumption health index corresponding to the nutrients of the consumable food.
In the area management device according to claim 2,
The health index calculation unit
calculating a consumption nutrition parameter indicating nutrients of the consumable food material and a controlled area nutrition parameter indicating nutrients of the controlled food material;
calculating the consumption health index using the consumption nutrition parameter;
An area management device that calculates the managed area health index using the managed area nutrition parameter.
In the area management device according to claim 3,
Further, referring to a storage unit that stores a food nutritional component table showing nutritional components for each food, the health index calculation unit uses the food nutritional component table to specify nutrients of the controlled food and the consumable food. Area management equipment.
In the area management device according to claim 2,
The health index calculation unit calculates the consumption health index according to the nutrients of the foodstuffs that were consumed within a past period substantially retroactively from the present time among the consumption foodstuffs.
In the area management device according to claim 2,
The output unit is an area for outputting information related to any of foods that should be consumed, foods that should be newly obtained, or preferable menus, according to at least one of the management area health index and the consumption health index. management equipment.
In the area management device that performs information processing on food storage,
a recognition unit that recognizes the usage status of the storage-related management area;
a health management barometer calculation unit that identifies nutrients recommended for intake by a predetermined user based on the usage status of the management area related to storage and calculates a health management barometer that indicates the nutritional status of the user; a calculation unit;
a proposal information generation unit that generates proposal information regarding recipes for ingesting the identified nutrients according to the health management barometer;
An area management device having an output unit that outputs the proposal information.
In the area management device according to claim 7,
The proposal information generation unit is an area management device having at least one of an additional food material determination unit that determines additional food ingredients, a healthy recipe information generation unit that generates the recipe, and a cooking method identification unit that identifies the cooking method of the recipe. .
In the area management device according to claim 8,
The health management barometer calculation unit is an area management device that calculates, as the health management barometer, at least one of a nutritional parameter indicating the nutrients recommended for intake and a health index according to the nutrients recommended for intake.
In the area management device according to claim 7,
The output unit is an area management device that outputs the proposal information and the health management barometer.
A program that causes one or more processors to execute the processing of the area management device according to any one of claims 1 to 10.