WO2024057563A1 - Foodstuff management system - Google Patents

Abstract

Provided is a foodstuff management system capable of smoothly registering foodstuffs with combined use of voice input. The foodstuff management system causes one or more computers to operate as: a startup unit for starting acceptance of voice input; a display output unit for causing a voice-recognized character string to be displayed on a display screen; an editing processing unit for accepting editing of the character string displayed on the display screen in accordance with an operation by a user, and, if editing has been performed, causing the edited character string to be displayed on the display screen; a termination adding unit for imparting a termination to the voice-recognized character string, and then handling the voice-recognized character string as a new character string; and a registration processing unit for causing the character string displayed on the display screen to be added to a foodstuff list as a foodstuff name, in accordance with an operation by the user.

Description

Food management system

The present invention relates to a food management system.

In recent years, with the advancement of IT technology, it has become possible to manage and control home appliances by utilizing the voice input function of mobile information terminals. For example, regarding refrigerators, Patent Document 1 states, ``The information processing system is designed to provide an information processing system that can improve usability in the management of foodstuffs stored in the refrigerator.'' It has a processing unit that can register multiple types of ingredients before the end of the reception. It is possible to execute a second mode in which the activated state is continued even if an instruction is received, and the second mode is executed for registering ingredients.''

JP 2021-128559 Publication

According to Patent Document 1, continuous voice input is possible without having to use a wake-up word for a mobile information terminal each time, and manual input is required for unknown information such as quantities associated with food names. I'm planning on giving you some guidance.

However, with voice input, there is a possibility of unintended food name recognition due to voice recognition errors, but there is room for improvement in smoothly correcting this and registering food ingredients.

From the above, in the present invention, one or more computers include a start section that causes one or more computers to start accepting voice input, a display output section that causes a voice-recognized character string to be displayed on a display screen, and a display output section that causes one or more computers to start accepting voice input. an editing processing unit that accepts editing of a string in response to a user's operation and displays the edited character string on the display screen when the string has been edited; Operations as a termination adding unit that treats a character string as a new character string, and a registration processing unit that adds a character string displayed on the display screen to a food list as a food name in response to a user's operation. We created a food management system that allows you to do this.

1 is a diagram showing an example of the overall configuration of a foodstuff management device according to an embodiment of the present invention. The figure which shows the example of a screen structure of the portable information terminal 2 at the time of voice input. FIG. 3 is a diagram showing an example of screen transition in method 1. 6 is a diagram illustrating an example of screen transition when candidate information exists in method 1. FIG. 6 is a diagram illustrating an example of screen transition in a case where candidate information does not exist in method 1. FIG. FIG. 2 is a sequence diagram showing a series of processing procedures between a user, a mobile information terminal, and a server in method 1; FIG. 7 is a diagram showing an example of screen transition in method 2. FIG. 7 is a diagram showing an example of screen transition in method 2. The figure which shows the screen transition example when changing a food ingredient name from a temporary list display. The figure which shows the example of a screen transition when a food ingredient name is deleted from a temporary list display. FIG. 7 is a sequence diagram showing a series of processing procedures between a user, a mobile information terminal, and a server in method 2; The figure which shows the example of a structure of a master database and a concierge database. The figure which shows the outline flow when referring to a master database and a concierge database. The figure which shows the data structure example of a concierge database. A diagram showing assumed storage room positions for each model and setting. A diagram showing screen transitions when referring to a concierge database. FIG. 3 is a sequence diagram showing the interlocking processing procedure of the master database and the concierge database. The figure which shows the process when there are multiple ingredients which completely match a search word. A diagram showing how to handle the display of ingredients whose consumption date has passed. A diagram showing how to handle ingredients deleted from a list of ingredients being stored. The figure which shows an example of the top screen immediately after starting an ingredients management function. The figure which shows the screen transition regarding the reminder function of consumption guideline registration. The figure which shows screen transition when a user can select which method to add ingredients.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the embodiment, a food management device (foodstuff management system) will be described, but the present invention is applicable not only to food materials but also to any item.

FIG. 1 is a diagram showing an example of the overall configuration of a food management device according to an embodiment of the present invention. The food management device includes a portable information terminal 2 as a computer, and a server 3 as a computer that stores a master database 31, a food list 32, and a concierge database 33. Computers linked together via a cloud or a network (hereinafter simply referred to as cloud 5) perform arithmetic processing to generate the screen of the mobile information terminal 2 as appropriate. For example, the food list 32 may be stored in the mobile information terminal 2 or the like. The refrigerator 4, which is an example of a storage, can be connected to the cloud 5 at least when it is equipped with a camera, which will be described later, for example.

As shown in FIG. 12, the master database 31 includes, for each unique food ID 311 and food name 312, a search word 313, reference food information 310 (estimated consumption period 314, recommended storage location 315, food category 316, one food category). Alternatively, two or more standard image data 317, 318) are stored in a linked manner as a set.

The ingredient list 32 that can be displayed on the mobile information terminal 2 is a list in which one or more ingredients are registered, and for each ingredient, an ingredient name 322 and ingredient details 320 (estimated consumption date 324, storage location 325, ingredient category 326, ingredient image 327) are linked and stored in the ingredient list 32. Please refer to FIG. 12 which shows an example of the display screen of the mobile information terminal 2 displaying the ingredient list 32. As will be described later, a link button 72 with the concierge database 33 can be displayed on the display screen of the ingredient details 320 shown in FIG. 12.

The food management device creates and maintains a food list 32 of food items managed by the user, and also sets and stores food name 322 and detailed food information 320 for each food item registered in the food list 32. The food material name 322 and food detailed information 320 can be set by referring to the food material name 312 and reference food material information 310 in the master database 31, or can be set by the user 1 through the mobile information terminal 2.

The mobile information terminal 2 includes a calculation unit CPU, a RAM for temporarily storing data, a ROM for recording programs, a communication unit Tr, and the like, which are connected via a bus BUS. The arithmetic unit CPU is capable of various processes according to programs recorded in the ROM, but FIG. 1 mainly shows processing functions specialized for voice input. The portable information terminal 2 can include a microphone that accepts voice input.

Functions related to voice input include a start section 21a that starts accepting voice input as a function of the voice input section 21 regarding voice input, and a start section 21a that starts accepting voice input of a new character string without requiring any operation by the user 1. and a continuous section 21b for starting. In addition, as a function of the display output unit 22 that displays a voice input character string etc. on the display screen, and the character conversion unit 23 that converts the input voice into a character string, termination is added to give the end of one character string. part 23a, an editing processing part 24 that accepts editing of the character strings displayed on the display screen in accordance with the operation of the user 1, and a registration processing part that registers ingredients corresponding to each of the displayed character strings in the ingredients list 32. 25. In addition, the above-mentioned individual functions are organically combined and executed, and the overall control including user requests is managed by the central processing unit 26 via the cloud 5. The overall processing unit 26 is not essential.

FIG. 2 shows an example of the screen configuration of the mobile information terminal 2 during voice input. In the display screen example in this case, a main screen 91 is placed at the top position, a character input screen 92 is placed at the bottom position, and a voice input status display section 93 and a character string display section 94 are arranged at the middle position, and at least these display sections are arranged. It is better to consist of For example, various menu buttons 96 are arranged at the bottom.

Note that information is displayed on the main screen 91 as it is successively changed in response to voice input or operation instructions from the user 1. The character input screen 92 is a character input screen using a keyboard, and allows the user 1 to manually input text (character string). The audio input state display section 93 uses, for example, a microphone symbol to distinguish and display when audio input is valid and when audio input is invalid. The character string display section 94 displays character strings generated in response to character input from a keyboard using the character input screen 92 or voice input from a microphone, either as they are or after being converted into kanji or the like. The bottom menu button 96 includes, for example, a home button for selecting a home screen in order from the left, as illustrated in FIGS. 8, 12, 18, 19, 20, 21, 22, and 23. , a food management button to select the food management screen, a driving status button to select the driving status screen, a support screen to select the support screen, and a settings button to select the setting screen.

In the above-described food management device, in the first embodiment of the present invention, voice input using the portable information terminal 2 is performed according to the processing procedure shown in FIG. 3 or FIG. Note that these series of processes are executed based on instructions from the overall processing unit 26.

Here, the processing procedure shown in FIG. 3 or FIG. 7 is executed by the foodstuff management device shown in FIG. 1. For example, text editing, which will be described later, is realized when the editing processing unit 24 accepts editing of a character string displayed on the display screen of the mobile information terminal 2 in response to an operation by the user 1. Continuous voice input is realized by the continuous section 21b of the voice input section 21 starting accepting voice input of a new character string.

<Method 1 of registering one ingredient at a time>
The processing procedure for registering foodstuffs in the foodstuff list 32 using Method 1 will be explained. FIG. 3 is a diagram showing an example of screen transition in method 1. FIG. 4 is a diagram showing an example of screen transition when candidate information exists in method 1. FIG. 5 is a diagram showing an example of screen transition in the case where candidate information does not exist in method 1.

First, when voice input is started by operating the voice input status display section 93 or inputting a wake-up word (S10), voice input of characters is accepted, and the recognized character string is displayed on the character string display section 94. If the end of voice input is detected, for example, when a period of silence continues for one second or more or the voice input status display section 93 is tapped (S11, Y), voice input of one ingredient name (temporary input) is detected. ) ends, and it is determined that this is the end of one character string (ingredient name). A character string that is speech-recognized after adding a terminator can be treated as a new character string. If the below-mentioned registration instruction is not given after the end is added and before new voice recognition starts, if the candidate food name is the only one found by searching the master database 31, that candidate food name is added to the food list 32. You may register the character string, or you may discard the character string input just before that.

Next, the display on the voice input status display section 93 is set to voice reception OFF, and reception of voice input is temporarily stopped. Also, at least up to now, the editing processing section 24 is enabled and the keyboard is displayed on the character input screen 92. In FIGS. 4 and 5, the editing processing unit 24 is enabled before receiving voice input, but this may be done.

Next, the master database 31 is searched using the character string displayed in the character string display section 94 as the search word 313 (S12). If an ingredient (ingredient ID 311 or ingredient name 312) matching the search word 313 is found (S13, Y), the ingredient name 312 is displayed on the main screen 91 as a candidate ingredient name (S14).

If the voice recognition is not working properly, the user can edit the character string displayed on the character string display section 94 as the voice recognition result through the editing processing section 24. If this character string has been edited (S15, Y), the process returns to S12 again based on the edited character string to proceed with the process. With this, it is possible to easily correct problems in the voice recognition results.

When the candidate food material name displayed on the main screen 91 is selected by tapping or the like (S16, Y), it is determined that a registration instruction has been given using the selected food material name 312. That is, the food name 312 is added to the food list 32 as the food name 322, and the reference food information 310 stored in the master database 31 is automatically set to the food material (S17).
On the other hand, if a candidate food ingredient is not found in the master database 31 and a registration instruction is given, for example by operating the "OK" button on the keyboard, the reference food information 310 cannot be used, so the detailed food information 320 is automatically The name of the ingredient inputted in the character string display section 94 is added to the ingredient list 32 without setting it. After adding one food ingredient to the food list 32 in this way, voice input is started again (S10). This start may be performed automatically as an operation of the continuous section 21b, or may be performed by an operation of the audio input state display section 93.

Note that if there is no food item matching the search word 313 in the master database 31 in S13, the candidate food item (foodstuff name 312) is not displayed (S13N). In this case, although not shown in FIG. 3, the user may edit the character string (S15) and search the master database 31 again for candidate ingredients (S12). Alternatively, the name of a food ingredient that is not stored in the master database 31 may be added to the food list 32 as it is (as is the character string displayed on the character string display section 94). At this time, although the food is added to the food list 32 with this character string as the food name 322, the detailed food information 320 is not automatically set because there is no reference food information 310 that can be referred to.

Additionally, S12 may be performed before the end of voice input.

4 and 5 show examples of screen transitions in each state in the first method. FIG. 4 shows a case where candidate information exists (S13, Y), and FIG. 5 shows a case where candidate information does not exist (S13, Y). , N) is an example of screen transition. Both show screen transitions from S10.

First, it will be explained with reference to FIG. Screen display contents sequentially transition from screen 101 to screen 105.

The screen 101 can be displayed immediately before S10 (the "start" state in FIG. 3). On the main screen 91, a part of the ingredients list in which the ingredients registered at this point (carrots, broccoli, chives) are listed can be displayed on the screen and in text. When the voice input state display section 93 is tapped, the screen changes to the screen 102 (S10).

A keyboard is displayed as a character input screen 92 in a part of the screen 101, for example at the bottom position. It allows for the combination of voice input and text editing.

The screen 102 is a screen where voice input is in progress, and the voice input status display section 93 is highlighted to indicate that voice input is in progress. The character string confirmed by voice input is displayed on the character string display section 94. In this example, it is assumed that the user utters "tomato" and the speech is recognized as intended.

Screen 103 is the screen after voice input (S11, Y). The master database 31 is referred to based on the voice input, and "tomato" and "cherry tomato" are displayed on the main screen 91 as candidate displays corresponding to the voice input (S14). If the voice recognition result is not good and, for example, a character string such as "TOMETO" is displayed on the character string display section 94, it can be edited using the character input screen 92. Note that in this state, since the audio input from the microphone is stopped (S12), the audio input status display section 93 displays a display indicating that audio input is disabled. Further, while the candidate food names are being displayed, the food display of the food list 32 is hidden. This makes it easy for the user to recognize that candidate foods are being displayed.

The screen 104 is a screen for confirming the candidate, and can be designated (selected) by tapping one of the ingredients displayed on the main screen 91. In method 1, for example, a tap on a candidate food material name functions as the registration instruction section 95. Here, it is assumed that "tomato" is selected and confirmed.

Screen 105 is the screen after the registration instruction is given. The selected food ingredient "tomato" has been added to the food list 32 on the main screen 91. At this time, the ingredient information for "tomato" is not added to the ingredient list 32 with only the ingredient name 322 "tomato", but detailed ingredient information is added by referring to the reference ingredient information 310 set in the master database 31. 320 is automatically set.
For example, based on the addition date 329 (see FIG. 12) of "tomato" to the ingredients list 32, and referring to the number of days set in the estimated consumption period 314, the date after this number of days from the addition date 329. is automatically set as the estimated consumption date 324 of "tomato". Further, for example, from an item whose ingredient name 312 in the master database 31 is "tomato", the recommended storage location 315 "vegetable room" is referred to and set as the storage location 325.

Now, after adding "tomato" to the list of ingredients, voice input is automatically activated on the screen 105 (S10). The voice input status display section 93 is highlighted to indicate that voice input is in progress. In this way, in method 1 which accepts registration instructions for one food ingredient at a time, acceptance of voice input is temporarily stopped after the end of the character string is added (S12). After the editing processing unit 24 operates in accordance with the instruction from the user 1, and the registration instruction unit 95 operates to add one ingredient to the ingredient list 32, the continuation unit 21b operates to input a new character string by voice. Registration will begin. You can immediately enter the next ingredient.

Next, FIG. 5 will be explained. The screen display contents sequentially transition from screen 111 to screen 115. Note that the screens 111 and 112 are basically the same as the screens 101 and 102 in FIG. 4, so the description will not be repeated. However, on the screen 112, it is assumed that the search word 313 that matches the master database 31 is a character string "〇△□×" that is not registered.

The screen 113 is the main screen because, as a result of referring to the master database 31 using the character string in the character string display section 94 as the search word 313, no matching candidate ingredients were found (the ingredient name 312 linked to the search word 313 was not found). No candidates are displayed on 91 (S13N).

Screen 114 shows a scene in which the "Complete" button as the registration instruction section 95 is selected by keyboard operation from the character input screen 92 in a state where no candidate ingredients have been found.

The screen 115 is the screen after the registration instruction, and a food ingredient whose food name 322 is the input character string "〇△□×" has been added to the main screen 91. However, since no corresponding candidate food ingredients have been found in the master database 31, the detailed food information 320 such as the estimated consumption date 324 and the storage location 325 cannot be automatically set. That is, only the ingredients name 322 is registered, and the ingredients detailed information 320 is registered as unset (S17). Then, acceptance of voice input is started again (S10).

In this way, if candidate information does not exist, the image 327 and estimated consumption date 324 of the food ingredient are not automatically set. Regarding the ingredient "〇△□×", the user can select this ingredient from the ingredient list 32 and edit it, and manually set the ingredient detailed information 320.

FIG. 6 is a sequence diagram showing a series of processing procedures between the user 1, the mobile information terminal 2, and the server 3 in the first method. In processing step S201, user 1 taps the microphone button (corresponding to the audio input state display section 93 in FIG. 2), and in response to this, mobile information terminal 2 starts the audio input function in processing step S202. When user 1 inputs voice in processing step S203, the voice is converted into a character string and then the character string is displayed in processing step S204. In processing step S205, the character string after voice input can be edited by keyboard operation.

The character string edited by voice input and/or keyboard operation is sent as a search word 313 to the server 3 in processing step S206, and the master database 31 in the server 3 is referred to to extract candidates. In processing step S207, candidate food names are presented to the mobile information terminal 2 side. As is clear from the above example, the number of candidate food names can be 0, 1, or 2 or more.

In processing step S208, the user 1 confirms the candidate food material name 312 displayed on the mobile information terminal 2. If you find an appropriate candidate food name 312 displayed, select the candidate food name 312 and instruct it to register. If not, edit the character string to change the search word 313 or update the master database. It is possible to give a registration instruction to register the food as it is in the food list 32 as it is not in the food list 31. In this way, the food name 322 to be added to the food list 32 is determined.

Information on the food name for which registration has been instructed is sent to the server 3 in processing step S209, and added to the food ingredients list 32 in the server 3 in processing step S210. At this time, the detailed food information 320 can be automatically set by referring to the reference food information 310 of the master database 31. Note that the food list 32 may be stored in the mobile information terminal 2.

In processing step S211, the registered food list 32 and detailed food information 320 are displayed on the mobile information terminal 2. The process moves to the voice input standby state of processing step S202 and waits for the next voice input.

<Method 2 that allows multiple ingredients to be registered at once>
FIG. 7 is a diagram showing the processing procedure of the second method. First, when voice input is started by operating the voice input status display section 93 or inputting a wake-up word (S20), character input by voice is accepted. If the end of voice input is detected, for example, when a period of silence continues for one second or more or when the voice input status display section 93 is tapped (S21, Y), voice input of one ingredient name (temporary input) is detected. ) ends, and it is determined that this is the end of one character string (ingredient name). That is, one character string is displayed as a temporary list on the character string display section 94 (S22). In method 2, multiple ingredients can be kept in the temporary list. Therefore, the temporary list is displayed in the range of the main screen 91 where part of the ingredients list 32 was displayed in method 1.

Further, in method 2, acceptance of voice input is resumed (S22). Subsequent steps S23 and S25 are executed as interrupt processing. Resuming acceptance of audio input may be performed automatically as an operation of the continuous section 21b, or may be performed by operating the audio input status display section 93.

While accepting voice input (S22), if any character string (ingredient name) registered in the temporary list is selected by tapping (S23, Y), editing processing is performed for the selected character string. The unit 24 is enabled and character string editing can be executed (S24). Voice input is stopped while character string editing is being executed.

Further, when a registration instruction is given by tapping a registration button as the registration instruction section 95 (S25, Y), the master database 31 is searched using each character string registered in the temporary list as a search word 313. . For each character string, if a food matching the search word 313 is found, the corresponding food name 312 is added to the food list 32, and the reference food information 310 is automatically set as the detailed food information 320 for that food. If a matching ingredient is not found, the detailed ingredient information 320 is not automatically set and the character string is added to the ingredient list 32 as the ingredient name 322. On the other hand, if no registration instruction is given (S25, N), voice input reception continues (S20).

8, 9, and 10 show examples of screen transitions in each state in method 2. FIG. 8 is an example of screen transition in method 2. FIG. 9 is an example of screen transition when changing the name of ingredients from the temporary list display. FIG. 10 is an example of screen transition when deleting a food ingredient name from the temporary list display.

First, in FIG. 8, the screen display contents sequentially transition from screen 121 to screen 125. However, this screen shows an example in which only the main screen 91 and the audio input status display section 93 are displayed. First, the screen 121 can be displayed in a state immediately before S20. In this screen, for example, on the main screen 91, information for explaining to the user 1 how to perform the operations shown in FIGS. 9 and 10 may be displayed. For example, ``You can edit the ingredient name by tapping the ingredient name on the temporary list'' or ``You can delete the ingredient by swiping the ingredient name on the temporary list.'' By tapping the voice input status display section 93 on the screen 121, acceptance of voice input is started (S20).

The screen 122 is a screen where voice input is in progress, and the voice input status display section 93 is highlighted to indicate that voice input is in progress. The character string input by voice is displayed on the main screen 91 as a character string display section 94.

Screen 123 is the screen on which voice input of one character string has been completed (S21, Y). In this example, it is assumed that the user utters "carrot" and the speech is recognized as intended. For example, by continuing silence for one second after finishing saying "carrot" or by tapping the voice input status display section 93, the voice input of one character string ends and the end of one character string is assigned. be done. After the termination is given, if there is no interrupt processing such as character string editing or registration instruction, acceptance of voice input is started again (S20). Further, when one or more character strings are input into the temporary list of the character string display section 94, the registration instruction section 95 (registration button) is activated.

Screen 124 shows a case in which the user inputs "broccoli," "leek," and "sake" in the same way using voice input and appropriate character string editing (repeating steps S20 to S25 and N).

When a registration instruction is given by tapping the registration instruction section 95 (S25, Y), each ingredient name (character string) registered in the temporary list of the character string display section 94 is used as a search word 313, and the method is 1, a search of the master database 31 and automatic setting of the detailed food information 320 are performed.

Screen 125 shows a state in which S26 has been completed and the screen has transitioned to the list display screen of food list 32. A detailed display of each food ingredient on the list display screen of the food list 32 is shown on a screen 126. On the screen 126, the number of days remaining until the estimated consumption date 324 (remaining "1 day" is displayed) and image data 327 are displayed for a food ingredient with a food name 322 "carrots". Further, superimposed on the image data 327 are displayed a vegetable icon 3251 indicating the vegetable compartment, which is the storage location 325, and an "N" icon 3252 indicating that the food has been newly registered in the food list 32. In this manner, on the list display screen, various icons indicating information about the food, such as the storage location 325 and the fact that the food is newly registered, can be displayed, particularly over the image data 327.

Note that in the example of FIG. 8, the character string "sake" is a polysemous word that can be the search word 313 for the ingredient name "sake" or "salmon". In this example, one of them, "salmon", is automatically selected and registered in the food list 32. However, as will be described later, regarding polysemous words, an instruction from the registration instruction section 95 may be used as a trigger, for example, to cause a transition to a screen for selecting which food ingredient in the master data 31 is to be selected.

FIG. 9 is an example of screen transitions when editing character strings using method 2 (S23, S24). On the screen 131, a temporary list of the character string display section 94 of the main screen 91 (“carrot,” “broccoli,” “leek,” and “salmon” are temporarily displayed) and a registration instruction section 95 are displayed. A microphone icon is displayed as the audio input state display section 93. Here, it is assumed that the user taps to edit "carrot" in the character string display section 94.

On the next screen 132, a character input screen 92 is displayed, and the editing processing section 24 functions. Further, in the character string display section 94, parts other than the selected character string "carrot" are displayed darkly. Using the keyboard on the character input screen 92, the character string "carrot" can be modified to, for example, "garlic". The modified screen 133 reflects the character string editing in the character string display section 94.

FIG. 10 is an example of screen transition for deleting ingredients from the temporary list in the character string display section 94. On the screen 141, when a predetermined operation, for example, a swipe operation is performed on the character string "carrot" from the character string display section 94 of the main screen 91, a "delete" button is activated as displayed on the next screen 142.
When the "delete" button is tapped, the screen changes to screen 143. That is, "carrot" is deleted from the temporary list.

FIG. 11 is a sequence diagram showing a series of processing steps according to FIG. 7 between the user 1, the mobile information terminal 2, and the server 3 in the second method. In processing step S221, user 1 taps voice input status display section 93 to instruct acceptance of voice input, and mobile information terminal 2 starts accepting voice input in process step S222 (S20). When user 1 inputs voice in processing step S223, the character string is displayed after converting the voice into a character string in processing step S224, and the end of the character string is determined based on the silent period (S21, Y). Add it to the list (S22). The system enters the voice input waiting mode again (S20).

In processing step S225, when the user 1 instructs to modify a character string using the keyboard in processing step S224, the input character string can be modified or deleted by operating the keyboard. The voice input and character string modification input in processing step S223 may be repeated multiple times, and processing steps S224 and S225 are also executed each time accordingly. Note that in the illustration, symbols a, b, and c represent a plurality of processes. Note that the reception of voice input (S20) may be temporarily stopped while the character string is being edited (S23, S24).

Thereafter, in processing step S226, user 1 instructs registration (S25, Y). This registration instruction is sent to the server 3 together with the temporary list in processing step S227. In processing step S228, the master database 31 in the server 3 is searched for each character string in the temporary list as a search word 313, and it is confirmed whether there is a matching food ingredient. If there is a matching food item, in processing step S229, the food material name 312, estimated consumption period 314, etc. are acquired from the master database 31, linked as detailed food information 320, and registered in the food list 32. When used in conjunction with the concierge function described later, the food material ID 311 may be acquired.

<Linking with concierge database 33>
The concierge database 33 is a database that can be created for a separate function independent of the ingredients management function (ingredients list 32 management function). For example, the concierge database 33 includes detailed information for each model of refrigerators produced by a certain manufacturer, such as what kind of storage chambers they have and what temperature setting patterns they have. . It can also be created for a concierge function that can output advice appropriate for that type of refrigerator when storing certain foodstuffs.
More specifically, if a certain type of refrigerator has only a refrigerated compartment and a chilled compartment, a function that can advise whether it is appropriate to store each food item in the refrigerated compartment or the chilled compartment is provided. be. For example, if the name of the ingredient is "raw ham," it can be advised to go to the chilled room, and if the name of the ingredient is "milk," it can be advised to go to the refrigerator. In addition, it is possible to clearly indicate where the chilled compartment is located in that type of refrigerator by using an icon representing a refrigerator. Furthermore, if the type of refrigerator allows the temperature setting of a certain storage compartment to be changed, for example, if the refrigerating temperature setting or freezing temperature setting can be changed, different advice can be given for each setting.

The master database 31 uses a food ingredient name 312 (or a food food ID 311) as a unique key, and includes information such as an estimated consumption period 314 and a recommended storage location 315 at the level of a storage room (refrigerator room, freezer room, vegetable room, etc.) provided in a general refrigerator. It includes as a data item and is suitable for general management of ingredients. On the other hand, the concierge database 33 similarly uses the food product name 332 (or food product ID 331) as a unique key, but includes information by refrigerator model and setting as data items. Therefore, if the master database 31, which assumes general level information about refrigerators, and the concierge database 33, which contains detailed level information by model and setting, can be linked (cooperated), it will be possible to build convenient functions. I can contribute.

In this embodiment, when searching for ingredients from the concierge database 33 in the ingredients management function, a function is provided that automatically inputs the ingredients name input or set in the ingredients management function (a function that uses the ingredients list 32, etc.). This contributes to convenience.

FIG. 12 is a diagram showing an example of the configuration of the master database 31 and the usage concierge database 33.

The master database 31 includes a food ID 311 that uniquely corresponds to the food ID 331 stored in the usage concierge database 33, a food name 312 that uniquely corresponds to the food ID 311, a search word 313 that corresponds to the food name 312, an expiration date, etc. A consumption guideline period 314, which is a corresponding index and is a consumption guideline from the time of food registration, a recommended storage location 315 that has been assigned as being suitable for the food ingredient indicated by the food ingredient name, and a food ingredient indicated by the food ingredient name 312 has been allocated to correspond. It includes a food category 316, standard image data 317 representing the appearance of the food material indicated by the food name 312, standard image data 318 corresponding to the food material category 316, and the like.

As described above, the detailed food information 320 displayed on the mobile information terminal 2 can be automatically set with information obtained by referring to the master database 31 using the search word 313. The detailed food information 320 may include information that is not displayed on the mobile information terminal 2, and in this embodiment, it has the same food material ID as the food material ID 311 of the master database 31. Then, by tapping the link button 72 on the display screen of the detailed food information 320, the user can search for and refer to the food material corresponding to this food material ID from the concierge database 33.

FIG. 13 explains the above-mentioned processing. First, in processing step S31, the master database 31 is searched by inputting the character string of the search word 313 on the mobile information terminal 2, and the food ingredient name 312 "carrot" is discovered and a registration instruction is issued. In response to this, in processing step S32, the master database 31 is referred to, and the information of the estimated consumption period 314, recommended storage location 315, food category 316, and standard image data 317, 318 corresponding to the food ingredient name 312 "carrots" is read out. The detailed information 320 is automatically set in the ingredients list 32 (S33).

Furthermore, suppose that user 1 wants to use the concierge function to refer to information other than the detailed food information 320. For example, in order to obtain advice on the appropriate location to store food ingredient name 312 "carrots" in the type of refrigerator used by user 1, a message is displayed on the display screen of ingredient detailed information 320 for food ingredient name 312 "carrots". 7. Operate the linked button 72. As a result, the concierge database 33 is searched using the food material ID 311 of this food material (there is no particular restriction as long as it can be linked, and the food material name 312 may also be used). As mentioned above, since the food material ID 311 in the master database 31 and the food material ID 331 in the how-to concierge database 33 are unique, it is possible to search and refer to the corresponding food material. In this way, by specifying a food item using the food management function and operating the link button 72, the concierge database 33 can be searched for this food item. User 1 can reduce the effort of activating the concierge function and entering the name of the ingredients again.

FIG. 14 is a diagram showing an example of the data structure of the concierge database 33. As mentioned above, the food material ID 331 is unique to the food material ID 311 of the master database 31. The search word 333 is used when the user uses the concierge function in addition to the ingredients management function. Here, since the search is made from the food ingredient ID 311 in the master database 31, there is no need to worry too much about it.

The concierge database 33 includes storage location/advice patterns 334 for each food item, which are subdivided by refrigerator model and setting. For example, as illustrated in the storage location/advice pattern 334 of FIG. 14, it is assumed that a "model A" refrigerator has a so-called six-door layout common to illustrations A to M shown in FIG. 15. The "Model A" refrigerator is a model that can have four settings, settings 1-4. Here, settings 1 to 4 have different temperature settings for the two storage compartments (switching compartments) shown in illustrations D and K, respectively, and the temperature settings for the storage compartments shaded in illustrations (D and K) are different. , setting 1 = (vegetables, frozen), setting 2 = (frozen, vegetables), setting 3 = (vegetables, vegetables), and setting 4 = (frozen, frozen). Further, in any setting, the storage room shown in illustration A is assumed to have a refrigerating temperature setting.

Then, for example, since the food ingredient name "carrot" 332 in FIG. 14 is a vegetable, a switching room with a vegetable temperature setting is recommended as the storage location. If there is no switching room with vegetable temperature settings, a refrigerator room should be recommended. The locations of the switching chambers for the vegetable temperature range in settings 1-4 correspond to illustrations D, K, L, and M, respectively. As the storage location/advice pattern 334, in settings 1, 2, and 3, illustrations D, K, and L corresponding to the locations of the switching chambers for the vegetable temperature range are set. On the other hand, in setting 4, since there is no switching room for the vegetable temperature range, illustration A is set, which shows the storage room for the refrigeration temperature range.

FIG. 16 is a diagram showing screen transitions when referring to the concierge database 33. Here, "tomato" is used instead of "carrot" mentioned above as the ingredient to be searched, but of course "carrot" can also be used. First, the screen 141 shows a display example of the main screen 91 of the mobile information terminal 2 after the reference food information 310 is reflected in processing step S33 of FIG. By tapping the interlock button 72 ("See advice on storage location") on this screen, the screen changes to a screen 142.

The screen 142 is an example of a concierge function, and is a screen for searching for recommended storage locations for food items by model. By tapping the interlocking button 72, the ingredient name 322 set by referring to the ingredient name 312 in the master database 31 is set as a search keyword in the concierge function. Here, "tomato" is searched from the concierge database 33. Although food names are used here for the convenience of explanation, if the food material ID 311 of the master database 31 is set when registering the food material in the food list 32, that may be used.

Then, the concierge function separately refers to the model information of the refrigerator used by user 1, and displays the recommended storage location for this model of refrigerator on the main screen 91 as the storage location for the selected "tomato". The location can be displayed. Here, if it is "setting 1" in the above format, illustration D will be displayed.

Additionally, as part of the concierge function, you can transition to a page with further advice on food preservation. Screen 143 displays further advice for this type, and indicates that it is recommended as a storage location "when the switching room is set to 'vegetables'".

FIG. 17 is a sequence diagram showing the interlocking processing procedure of the master database 31 and the concierge database 33. Continuing to the process shown in FIG. 6, in process step S212, the user 1 taps a food item of which the user wants to check the details from the food list 32 displayed on the mobile information terminal 2 to display detailed food information 320. In response to this, in processing step S213, detailed food information 320 and "View storage location advice" (linked button 72) are displayed on the mobile information terminal 2.

In processing step S214, the interlock button 72 is tapped. In processing step S215, the app moves to the "concierge function" screen, and in processing step S216, the model of the refrigerator 4 used by user 1, which has been obtained in advance, and the settings at the time of processing step S215 are obtained from the refrigerator 4. . In processing step S217, the concierge database 33 is searched using the ingredient name 322 (or ingredient ID 321) corresponding to the ingredient detailed information 320 as a key. Regarding the matching ingredients, the storage location/advice pattern 334 in the concierge database 33 is further obtained using the information already obtained in S216. In processing step S218, the obtained results are displayed on the mobile information terminal 2.

[Other features]
Next, various related functions will be explained.

<Pre-setting options for storage location 325>
In the above embodiment, the recommended storage location 315 is defined in the master database 31 for each food material ID 311. However, depending on the model of the refrigerator used by user 1, the recommended storage location 315 may not be provided. For this reason, a function may be provided in which a storage room among the recommended storage locations 315 can be automatically set as the storage location 325 in advance.

Specifically, a function is added to register the storage room included in the refrigerator used by user 1. Assume that, through this function, it is registered that the only storage rooms that can be used by user 1 are a "refrigeration room" and a "freezer room." In this case, for foodstuffs for which the "vegetable room" is specified as the recommended storage location 315, for example, "refrigeration room", which has a similar temperature range, may be automatically set as the storage location 325, or It is possible to prevent the storage location 325 from being automatically set.

<Selection and adjustment of automatically input detailed food information 320>
In the above embodiment, the estimated consumption period 314, recommended storage location 315, food category 316, and standard image data 317, 318 are defined as items that are automatically set through a search of the master database 31. However, for example, there may be users who do not want to manage the estimated consumption period 314. For this reason, a function may be provided that allows selection of items to be automatically set.

Furthermore, in the above embodiment, one kind of period is determined for each food ingredient in the master database 31 as the estimated consumption period 314 (for example, 5 days for "pork"). However, it may be assumed that there are users who are sensitive to freshness, such as those who use a refrigerator with an excellent freshness preservation function, or who are able to keep the food fresh for a longer period of time than usual. For this reason, for example, a function may be provided that allows the period automatically set as the consumption guideline period 314 to be adjusted to multiple levels such as "short", "standard", and "long".

Additionally, food stores may sell food at a discount because the expiration date is approaching. Since it may not be practical to set a normal consumption period 314 for such foods, a check box or the like may be provided to register them as so-called "service items." If the food is registered in the food list 32 with this checkbox checked, a consumption guideline period 314 that is shorter than usual may be set. The foods for which this check has been registered are preferably displayed on the display screen of the foodstuff list 32 so that they can be distinguished.

<Selection of which image data 317 or 318 of the master database 31 to reflect>
In the above embodiment, two types of standard image data 317 and 318 are prepared as image data to be automatically set as the detailed food information 320. For example, by preparing two types of data with completely different atmospheres, photo data and illustration data, each user can say, ``If it is photo data, Family A will buy it and use it'' or ``If it is illustration data, it is something that family A will buy and use.'' It can be envisaged that they will be used after establishing their own tacit agreement, such as, ``If Family B buys and uses this,''
For this reason, which image data to use as the detailed food information 320 may be selected manually each time instead of automatically.

<Database structure that differentiates food ingredient names 312 for each storage state>
In the above embodiment, the master database 31 has food names 312 such as "carrots" and "broccoli" that do not distinguish between food states. However, for example, regarding "shrimp", there may be different storage conditions, such as raw shrimp that is suitable for storage in a chilled room, and frozen shrimp that is sold in a frozen state. For this reason, food names 312 may be prepared in advance in the master database 31 or the concierge database 33, such as "shrimp (raw)", "shrimp (frozen)", etc., which distinguish the storage state. In this case, the recommended storage location 315 can be a database in which the former is set as a chilled room or a refrigerator room, and the latter is set as a freezing room.

<Processing when there are multiple ingredients that completely match the search word 313>
If, for example, a homophonic and ambiguous character string is input as the search word 313, two food ingredients names 312 may correspond to each other as a complete match. For example, the search word 313 ``salmon'' corresponds to the two ingredients names 312 ``sake'' and ``salmon,'' making it impossible to determine which one. In this case, the detailed food information 320 may be registered once without being set, and determined later.

FIG. 18 is a diagram illustrating processing when there is a plurality of ingredients that completely match the search word 313. First, the screen 151 is the display content when a temporary list is created in method 2, for example.
The screen 152 is the display content when the food names determined by using the character strings in the temporary list are reflected in the food list 32 according to the registration instruction. Since there is an exact match "sake" and "salmon" from the search word 313 "sake", we did not decide on the ingredients and treated it as "sake" as is, and added it to the ingredients list 32 in the same way as ingredients that are not in the master database 31. do. By doing so, it is possible to suppress setting of incorrect ingredients names 322 and ingredients detailed information 320.

The user 1 can add an appropriate food ingredient by transitioning from the screen 152 to the ingredient detailed information 320 screen for "salmon", text-editing the food ingredient name 322, and editing it again, for example, as "salmon". In response to having "salmon" set as the food ingredient name 322, the master database 31 is searched again and the detailed food information 320 is automatically set.

<Food management using refrigerator camera>
In the above example, ingredients are added to the ingredients list using voice input or character string editing using the keyboard, but for example, by attaching a camera to the refrigerator, the camera image is used to add ingredients to the ingredients list. You may also do so. The camera is included in the foodstuff management device illustrated in FIG.

First, using the method described in the above embodiment, the name of the food ingredient and the actual storage location are registered in the detailed food information 320 to make it known. The actual storage location is not a recommended storage location obtained by referring to the master database 31, but is a storage location in a refrigerator or the like used by the user 1 that can be modified as appropriate by the user 1. By doing so, information that matches actual storage can be obtained in advance.

Next, the camera captures an image of the food in the storage compartment, and this image is used to perform image recognition of the food. Publicly known methods can be used for image recognition processing. First, it is recognized whether an item is a food item or not, and the image is cut out for each food item. For example, when an image of the vegetable compartment is taken, the image is cut out for each food item, such as carrots or broccoli. At this point, image recognition can recognize the presence of food items, but does not recognize what each of the cut-out ingredients is.

Next, each cut-out food item is processed to recognize the item. At this time, it is preferable to configure the system so that information about the storage room imaged by the camera (refrigeration room, freezer room, vegetable room, etc.) can be acquired. This can be realized, for example, by requesting input from the user 1 before imaging. Further, the detailed information 320 of each food item in the food list 32 is referred to to see what is stored in each storage room.

By using the storage room information of the location imaged by these cameras and the storage location information of the food list 32, it is possible to improve the accuracy of image recognition. That is, for example, if information indicating that carrots are stored in the vegetable drawer is registered in the food list 32, the image recognition function will weight the food to indicate that there should be carrots in the vegetable drawer. You can advance your recognition.

Then, for each food item imaged by the camera, it is determined whether or not it has already been registered in the food list 32. For items determined to have already been registered, the image data captured by the camera can be registered as the image data 327 of the detailed food information 320 for a certain food item in the food list 32. By doing so, it is possible to employ image data for actual use instead of the standard image data 317 and 318, and therefore it is possible to provide an image close to what the user 1 remembers.

On the other hand, if the food is determined to be a food that is not registered in the food list 32, processing is started as a new food. That is, the master database 31 is searched using the ingredients name obtained through image recognition as the search word 313, and detailed information 320 of the discovered ingredients is set. However, this does not apply to the recommended storage location. Furthermore, a camera image is set as the image data 327.

Note that, at the current state of the art, it is not necessarily easy to perform image recognition with high accuracy.
For this reason, even if the image data 327 is set and presented to the user 1 for each food imaged by the camera, the user 1 visually determines whether or not it is necessary to add the food to the food list 32. good. In this case, you can temporarily add all the ingredients to the ingredient list 32 and have the user 1 delete unnecessary ingredients, or you can create a new list as an addition candidate list and present it. You may.

<Processing in the ingredients list 32 of ingredients whose estimated consumption date 324 has passed>
FIG. 19 is a diagram showing how to display foodstuffs whose estimated consumption date has passed. As illustrated in the screen 161 of FIG. 19, multiple types of lists can be defined as the food list 32. For convenience, the food list 32 described in the above embodiment is referred to as a "stored food" list. This is the list shown on screen 161.

When one day has passed since the state of the screen 161, the two foods at the top of the list on the screen 161 (carrots, broccoli) have passed the estimated consumption date 324 of the detailed food information 320 for each. When you display the "Ingredients in Storage" list again in this state, a pop-up will appear indicating that the ingredients whose estimated consumption date of 324 has passed have been deleted from the "Ingredients in Storage" list. is displayed (screen 162). These foods have been deleted from the "Foods in storage" list (screen 163).

Figure 20 is a diagram showing how ingredients that have been deleted from the list of stored ingredients are handled. The ingredient management device defines, for example, a "recently deleted ingredients" list in addition to the "stored ingredients" list. These lists are displayed as tabs on the same display screen, and it is possible to move between them with one tap. An ingredient registered in the "stored ingredients" list can be moved to the "recently deleted ingredients" list when the estimated consumption date 324 in the ingredient details information 320 has passed (screens 171, 172). An ingredient moved to the "recently deleted ingredients" list (screen 172) can be completely deleted, for example, 48 hours after being moved.

By doing this, it is possible to prevent ingredients from accumulating in the "Ingredients in Storage" list, and even if User 1 neglects to update the "Ingredients in Storage" list, it is possible to prevent the ingredients from accumulating in the "Ingredients in Storage" list to some extent. A corresponding list can be maintained. On the other hand, by moving them to the "Recently Deleted Ingredients" list, there are some ingredients that must be consumed relatively soon because the estimated consumption date of 324 has passed and they are still in storage. You can find this in the "Recently Deleted Ingredients" list.

The above-mentioned alarm function may be deactivated from the ingredient detailed information 320 of the ingredient moved to the "Recently Deleted Ingredients" list (screen 173). Further, the estimated consumption date 324 may be maintained as is from those registered in the "foodstuffs in storage" list.

<Top screen of ingredients management function (home screen)>
The top screen displayed when the food management device starts the food management function (food management application) on the mobile information terminal 2 will be described. FIG. 21 is a diagram showing an example of the top screen immediately after starting the ingredients management function. The top screen 181 has a function of displaying recommended ingredients, and in particular, a function is provided in which the recommended ingredients change depending on whether or not there is an ingredient whose estimated consumption date 324 is close to the ingredient list 32 (list of "ingredients in storage"). It is being Two screens 182 and 183 included in the top screen 181 will be described next.

If there are no ingredients in the ingredients list 32 whose estimated consumption date 324 is close (for example, only after 3 days), ingredients that are in season today are displayed (recommendation display, screen 182). In the recommendation display, if there are seasonal ingredients in the ingredients list 32, the ingredient with the least number of days remaining until the estimated consumption date 324 may be displayed.

If there is a food item in the food list 32 whose estimated consumption date is today, a message is displayed to suggest consumption of the food item (reminder display, screen 183). If the food list 32 includes a plurality of food items with the same estimated consumption date, they may be displayed randomly from among them, for example. For the reminder display, a push notification may be sent to the mobile information terminal 2, and in response to this, a screen transition such as displaying the ingredients list 32 may be made possible.

<Registration reminder for estimated consumption date 324>
FIG. 22 is a diagram showing screen transitions regarding the reminder function of consumption guideline registration. The ingredients management function of the ingredients management device can be configured as an application that integrates multiple functions such as the above-mentioned concierge function. For example, in such a case or not, a menu for transitioning to the ingredients management function can be displayed on the top screen 191, and by selecting this menu, it is possible to transition to the ingredients management function screen (screens 192 and 193). ).

Screen 192 shows a case where there are no registered ingredients in the ingredients list 32. In this case, when the estimated consumption date 324 is set in the detailed ingredients information 320 when ingredients are registered, a message is displayed accordingly to the effect that notifications (reminders) such as recommendations of ingredients to be used can be made.

Screen 193 shows a case where there are registered ingredients in ingredient list 32. If the estimated consumption date 324 is not set in the detailed food information 320 for any of the registered food materials, a message to that effect is displayed as illustrated on the screen 193. Whether or not the estimated consumption date 324 is not set for a food ingredient is determined by whether or not the number of days remaining until the estimated consumption date is not displayed for the food item in the list display of the food list 32 as shown in the screen 193. can. On the screen 183, for foodstuffs for which the estimated consumption date 324 is set, "D days remaining in the estimated consumption date" (D is the actual number of remaining days) is displayed.

A message indicating that some foods have not been set for consumption date 324 may indicate that there are foods whose consumption date 324 has not been registered for several days, or that the number of unregistered ingredients has exceeded a certain threshold. , etc. may be displayed. Further, the user 1 may be able to select whether or not to issue a message.

<Sort of food list 32 when user 1 owns multiple refrigerators>
The detailed food information 320 may also be configured such that it is possible to distinguish "which refrigerator" is the storage location. In the above embodiment, with the tacit understanding that the user 1 uses only one refrigerator, the storage locations that can be set as the storage location 325 are the "refrigerator compartment" and "vegetable compartment" in a certain refrigerator compartment. I was preparing a storage room as an option. However, two types of information may be set in more detail as the storage location 325. Specifically, the first type of information allows you to set storage facilities such as "refrigerator 1,""refrigerator2," and "warehouse," and the second type of information allows you to set storage facilities such as "refrigerator room" and "vegetable room." The storage location within the facility may be settable. The name of the first type of information may be defined by the user 1. The name of the second type of information may have a hierarchical structure so that it changes depending on the settings of the first type of information. For example, if "Refrigerator 1" is set, options such as "refrigerator" and "vegetable compartment" will be displayed, and if "warehouse" is set, "1st tier from the bottom" and "2nd tier from the bottom" will be displayed. It may be possible to define different options to be displayed.

The ingredients management device can be equipped with a filtering function and a sorting function so that the ingredients list 32 can be displayed for each of these storage locations 325. For example, the filtering function may include a function that allows only the foods stored in "refrigerator 1" to be displayed as a list of ingredients. As for the filtering function and sorting function, in addition to the function of sequentially displaying each storage location 325, for example, the function of filtering or sorting the ingredients recently registered in the ingredients list 32 (``stored ingredients'' list) is provided. You can leave it there. Note that for recently registered ingredients, an icon indicating this fact is displayed on the ingredient list 32 display screen (for example, screen 125 in FIG. 8) or the ingredient detailed information 320 display screen (for example, screen 320 in FIG. 12). May be displayed. As a specific example, the "N" icon 3252 illustrated in FIG. 8 may be shown.

<Variations of additional registration to ingredients list 32>
FIG. 23 is a diagram showing screen transitions when the user 1 can select which method to add ingredients. When the user instructs to start the "ingredients management" function on the top screen 201, the ingredients management top screen 202 is displayed. On the food management top screen 192, a food registration method selection button 75 is displayed together with a food list 32 that includes food items that have already been registered, and when this button is operated, the screen changes to a screen 203. On the screen 203, selection buttons for camera input, voice input, and text input are displayed as food registration methods. By selecting one of these, one of food input screens 203A, 203B, and 203C is displayed. Voice input is as in the above embodiment. In addition, text input using only the keyboard without using audio, and camera input that starts with setting image data 327 are also available.

<Detection of food turnover rate>
Although it differs depending on the user 1, for example, it is assumed that there are foods such as eggs, dairy products, and beer that are consumed relatively daily and have a high turnover rate and are purchased frequently. If such foods can be recognized, it is expected that it will be possible to provide notifications to prevent out-of-stock items, so we will explain one example. Here, it is assumed that the above-mentioned camera is used.

For example, food items that were added to the food list 32 and consumed within a short period of time (for example, within 24 hours), or food items that have changed location or orientation in the storage location using the image recognition function of the camera described above, can be checked at the rotation rate. There may be a way to estimate it as a candidate for a food with a high value. Another possible method is to display on the mobile information terminal 2 an image of a storage room containing foodstuffs captured by a camera, and have the user 1 select the foodstuff whose turnover rate is desired to be tracked. For food items that have a high turnover rate or are designated by the user 1, it is preferable to perform number recognition using image recognition by the camera. In order to facilitate number recognition, when turning on the number recognition function, the food management device displays, for example, the display screen of the mobile information terminal 2, so that the food items to be recognized are stored in areas close to each other in the storage room. A request message may also be output. Alternatively, a plurality of areas for storing food items to be recognized may be prepared in advance on the display screen of the mobile information terminal 2 for each food item, and the number of food items may be recognized for each area selected by the user 1. That is, the camera is configured in advance so that the interior of a certain storage room can be virtually divided into a plurality of areas and recognized. For example, if there are multiple shelves in the refrigerator, each shelf should be recognized as a different area. Then, the user specifies that a specific food item, for example yogurt, is to be placed in a predetermined area, for example, on a certain shelf. This makes it easier for the camera to recognize the yogurt placed in the area, thereby making it easier to recognize the number of yogurts and the rotation rate.

The user can specify the yogurt storage shelf through the mobile information terminal 2 or the like.

On the other hand, foods that are not consumed for a long time, or foods that have not changed their storage location or orientation are estimated to be candidates for foods with a low turnover rate. Since there is a concern that the user 1 may forget about the existence of foodstuffs with a low turnover rate, the notification may be made at a timing that does not depend on the estimated consumption date.

1: User, 2: Mobile information terminal, 3: Server, 4: Refrigerator (storage, warehouse), 5: Cloud (network), 31: Master database, 32: Ingredients list, CPU: Computing unit, Tr: Communication unit , Bus: Bus, 21: Audio input section, 21a: Start section, 21b: Continuous section, 22: Display output section, 23: Character conversion section, 23a: Ending section, 24: Edit processing section, 25: Registration processing section , 26: General processing section, 91: Main screen, 92: Character input screen, 93: Voice input status display section, 94: Character string display section

Claims (18)

1. on one or more computers,
   a start part that starts accepting voice input;
   a display output unit that displays the voice-recognized character string on a display screen;
   an editing processing unit that accepts editing of the character string displayed on the display screen according to a user's operation, and displays the edited character string on the display screen if the character string has been edited;
   a termination adding unit that adds a terminator to a voice-recognized character string and then treats the voice-recognized character string as a new character string;
   A food management system that operates as a registration processing unit that adds a character string displayed on the display screen as a food name to a food list in response to a user's operation.
2. The food management system according to claim 1,
   The display output section displays two or more character strings,
   The editing processing unit is capable of editing each character string displayed by the display output unit,
   The registration processing unit is a food management system that adds each character string displayed by the display output unit to a food list as a food name.
3. The food management system according to claim 2,
   It is possible to refer to a master database that stores food names and detailed food information corresponding to the food names,
   When the registration processing unit operates, the display output unit searches the master database for the ingredient name corresponding to each of the character strings displayed, and for character strings that have a corresponding ingredient name, the ingredient name and the corresponding A food management system that allows detailed food information to be added to the food list as a set.
4. The food management system according to claim 1,
   The display output unit displays a character string of 1,
   A foodstuff management system that automatically operates the start unit after the registration processing unit operates.
5. The food management system according to claim 4,
   It is possible to refer to a master database that stores food names and detailed food information corresponding to the food names,
   Searching the master database for a food ingredient name corresponding to the character string displayed by the display output unit, and if there is a corresponding food name, displaying it on the display output unit as a candidate food item;
   When the selection of the candidate food item displayed on the display screen is accepted from the user, the registration processing unit adds the food name of the candidate food material and the corresponding food detailed information as a set to a food list. .
6. The food management system according to claim 3 or 5,
   It is possible to refer to a concierge database that is prepared separately from the master database and can refer to ingredients corresponding to ingredients in the master database by ingredient name or ingredient ID,
   A food management system in which, by specifying a food in the food list, the food name or food ID of the food can be automatically entered as a search key for the concierge database.
7. The food management system according to claim 3 or 5,
   The detailed food information includes the storage location of the storage,
   A foodstuff management system in which a user can specify in advance the type of storage room that can be set as the storage location.
8. The food management system according to claim 3 or 5,
   The detailed information on the ingredients includes the estimated consumption period of the ingredients;
   A food ingredients management system that allows a user to specify in advance the length of the period set as the consumption guideline period.
9. The food management system according to claim 3 or 5,
   The detailed food information includes two or more types of standard image data of the food material,
   A food management system in which a user can specify the type of standard image data to be set as the standard image data when adding each food material to a food list.
10. The food management system according to claim 3 or 5,
    In the food management system, two or more food names indicating the same food but indicating different storage states are prepared in the master database.
11. The food management system according to claim 3 or claim 5,
    As a result of searching the master database using the character string displayed by the display output unit as a search word, if a plurality of candidate ingredients with exact matches are obtained, even if the registration processing unit is operated, the name of the ingredient will not be displayed. This is a food management system that does not set detailed food information.
12. The food management system according to claim 3 or 5,
    The food management system is equipped with a camera,
    Performing image recognition processing on the image captured by the camera and cutting out each food item included in the image,
    As a result of image recognition, for ingredients whose names have been recognized,
    setting an image of the food material cut out from the image as the detailed food information of the food material;
    acquiring at least a part of detailed food information corresponding to the food name from the master database and setting it as detailed food information of the food material;
    A food management system that adds the food to the food list.
13. The food management system according to claim 3 or 5,
    The food management system is equipped with a camera,
    Performing image recognition processing on the image captured by the camera and cutting out each food item included in the image,
    A food management system that displays food images cut out for each food material as a result of image recognition, and inquires of the user whether or not each food material needs to be added to the food list.
14. The food management system according to claim 1 or claim 4,
    The food list includes a list of stored food items and a list of deleted food items,
    In the food management system, food items that are included in the list of stored food items and whose estimated consumption date has passed are moved to the deleted food list.
15. The food management system according to any one of claims 1 to 5,
    The food management system displays a reminder to recommend the consumption of the food when the food list includes food whose estimated consumption date is near, and otherwise displays a recommendation to display food that is in season.
16. The food management system according to any one of claims 1 to 5,
    A food item that outputs a prompt to set a consumption date according to the presence of food items included in the food list for which a time has elapsed without a consumption date set or according to the number of ingredients for which a consumption date has not been set. management system.
17. The food management system according to any one of claims 1 to 5,
    An ingredient management system that allows you to set two or more storage locations for a single ingredient as detailed ingredient information.
18. The food management system according to any one of claims 1 to 5,
    The food management system is equipped with a camera that is configured to virtually divide the storage room into multiple areas and recognize them.
    Upon receiving a user's designation to designate one of the plurality of areas to store a predetermined food item, the number of pieces of the predetermined food item stored in the area is recognized and/or the turnover rate is tracked. , ingredients management system.

Images