JP2024036515A - Information processing system and information processing method - Google Patents

Abstract

An object of the present invention is to provide an information processing system that can improve usability. An information processing system has a processing section. The processing unit is capable of registering a plurality of types of foodstuffs by the end of one reception of the voice interaction function. The inventory status of ingredients managed by the voice interaction function can also be changed by inputting the contents of the display screen displayed on a predetermined device, and the inventory status of ingredients managed by the voice interaction function can be changed by inputting the contents of the display screen displayed on the predetermined device. If the ownership status of the ingredients is changed, the ownership status of the ingredients will be managed using the voice interaction function based on the ownership status after the change, and the management of the ownership status of the ingredients will include at least the name of the ingredients, the quantity of the ingredients, and In the voice interaction function, if the name of the food ingredient is accepted based on the user's utterance, but the quantity of the food ingredient is not included in the utterance content, a predetermined rule is It is treated as if a predetermined quantity based on the specified amount has been specified, and the registration of the food material is accepted. [Selection diagram] Figure 23

Description

Embodiments of the present invention relate to an information processing system and an information processing method.

An information processing system for managing food stored in a refrigerator has been proposed. It is expected that such a food management system will be easier to use.

Japanese Patent Application Publication No. 2017-204770

The problem to be solved by the present invention is to provide an information processing system and an information processing method that can improve usability.

The information processing system of the embodiment has a processing section. The processing unit is capable of registering a plurality of types of foodstuffs by the end of one reception of the voice interaction function. The inventory status of ingredients managed by the voice interaction function can also be changed by inputting the contents of the display screen displayed on a predetermined device, and the inventory status of ingredients managed by the voice interaction function can be changed by inputting the contents of the display screen. If the ownership status of the ingredients is changed, the ownership status of the ingredients will be managed using the voice interaction function based on the ownership status after the change, and the management of the ownership status of the ingredients will include at least the name of the ingredients, the quantity of the ingredients, and In the voice interaction function, if the name of the food ingredient is accepted based on the user's utterance, but the quantity of the food ingredient is not included in the utterance content, a predetermined rule is It is treated as if a predetermined quantity based on has been specified, and the registration of the food is accepted.

FIG. 1 is a diagram showing the overall configuration of an information processing system according to an embodiment. FIG. 1 is a front view showing a refrigerator according to an embodiment. FIG. 1 is a block diagram showing the functional configuration of a smart speaker according to an embodiment. FIG. 1 is a block diagram showing the functional configuration of a terminal device according to an embodiment. FIG. 1 is a block diagram showing the functional configuration of a server system according to an embodiment. The figure which shows an example of the content of inventory list DB of embodiment. The figure which shows an example of the content of shopping list DB of embodiment. The figure which shows an example of the content of the trash DB of embodiment. The figure which shows an example of the content of candidate list DB of embodiment. The figure which shows an example of the content of food ingredients DB of embodiment. FIG. 3 is a diagram showing an image displayed on a display screen according to an embodiment. The figure which shows the image of the top page of ingredients management of embodiment. The figure which shows the image of the modification of the top page of ingredients management of embodiment. The figure which shows the image which displayed all the ingredients list of embodiment. The figure which shows the image which the food ingredients list for every storage room of embodiment was displayed. The figure which shows the image of the modification in which the ingredients list of embodiment was displayed. The figure which shows the image when newly registering a food ingredient to the food list of embodiment. The figure which shows the image of the modification which newly registers a food ingredient to the food list of embodiment. The figure which shows the image when changing the ingredients list of embodiment. The figure which shows the image containing the trash can list of embodiment. The figure which shows the image containing the shopping list and candidate list of embodiment. FIG. 7 is a diagram illustrating another image including a shopping list according to an embodiment. 5 is a flowchart showing the flow of processing related to the voice interaction function of the embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. FIG. 7 is a diagram illustrating a modified example of an image displayed on the display screen of the embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment. The figure which shows the modification of the image displayed on the display screen of embodiment.

Hereinafter, an information processing system according to an embodiment will be described with reference to the drawings. In the following description, components having the same or similar functions are denoted by the same reference numerals. Further, redundant explanations of these configurations may be omitted. "Based on XX" means "based on at least XX" and may include cases where it is based on another element in addition to XX. "Based on XX" is not limited to cases in which XX is used directly, but may also include cases in which calculations and processing have been performed on XX. "XX or YY" is not limited to either XX or YY, but may include both XX and YY. "XX" and "YY" are arbitrary elements (for example, arbitrary information). In the following, "database" will be referred to as "DB". In the following, a case will be described in which the voice dialogue function is implemented, but speech from the device side is not essential. Therefore, "voice dialogue" in the following description may be replaced with "speech recognition." Each of the "best before date" and "expiration date" described below is an example of the "intake date" or the "first deadline".

<1. Overall configuration>
FIG. 1 is a diagram showing the overall configuration of an information processing system 1 according to an embodiment. The information processing system 1 is an example of a "foodstuff management system" and is a system for managing the inventory amount of food materials held by a user. The information processing system 1 includes, for example, a refrigerator 100, a smart speaker 200, a home router 300, a terminal device 400, and a server system 500. The network NW to be described later includes, for example, one or more of the Internet, a cellular network, a Wi-Fi network, a WAN (Wide Area Network), a LAN (Local Area Network), a public line, a telephone line, a wireless base station, and the like. The router 300 is, for example, a wireless router, but is not limited thereto.

Refrigerator 100 is placed in a user's home. Refrigerator 100 is connected to network NW via router 300, for example, and can communicate with server system 500. In this embodiment, the refrigerator 100 includes a display device 130, which will be described later. In this case, refrigerator 100 corresponds to an example of an "information display device." The refrigerator 100 is equipped with a microphone and a speaker, and may have the same voice interaction function as the smart speaker 200. In this case, the refrigerator 100 corresponds to an example of a "voice dialogue device (or voice recognition device)."

Smart speaker 200 is placed in the user's home. The smart speaker 200 is connected to the network NW via a router 300, for example, and can communicate with the server system 500. Smart speaker 200 may be able to communicate with refrigerator 100 via router 300, for example. The smart speaker 200 is an example of a "voice dialogue device (or voice recognition device)."

The terminal device 400 is, for example, a mobile terminal device owned by a user, such as a smartphone or a tablet terminal device. Terminal device 400 is connected to network NW and can communicate with server system 500. An application program (hereinafter sometimes referred to as an “app”) for managing ingredients and remotely controlling the refrigerator 100 is installed on the terminal device 400. Terminal device 400 is an example of an "information display device." However, the terminal device 400 is not limited to the above example, and may be a personal computer, a television receiver with an input function, or the like. The terminal device 400 is equipped with a microphone and a speaker, and may have the same voice interaction function as the smart speaker 200. In this case, the terminal device 400 corresponds to an example of a "voice dialogue device (or voice recognition device)."

The server system 500 is a server system that records various data for food management, and is configured by, for example, a plurality of server devices 501 and 502. The server devices 501 and 502 are cloud servers connected to the network NW. For example, the server system 500 includes a first server device 501 for voice interaction that controls the smart speaker 200 and a second server device 502 for device control that controls the refrigerator 100. However, the server system 500 may be configured with one server device, or may be configured with three or more server devices. The server system 500 may include an information processing unit that performs edge computing or fog computing, such as an information processing unit included in the router 300 and/or an information processing unit included in a router in the network NW. Various functional units included in the server system 500, which will be described later, may exist separately in two or more devices.

Refrigerator 100, smart speaker 200, terminal device 400, and server system 500 will be described in detail below.

<2. Refrigerator>
FIG. 2 is a front view showing the refrigerator 100. Refrigerator 100 includes, for example, a housing 110, a plurality of doors 130, a display device 140, an actuator 150, and a control unit 160. A plurality of storage rooms (storage rooms) 120 are provided inside the casing 110. The plurality of storage compartments 120 include, for example, a refrigerator compartment 121, a chilled compartment 122, a vegetable compartment 123, an ice making compartment 124, a small freezing compartment 125, and a main freezing compartment 126. In this embodiment, a refrigerator compartment 121 is arranged at the top, a vegetable compartment 123 is arranged below the refrigerator compartment 121, an ice-making compartment 124 and a small freezer compartment 125 are arranged below the vegetable compartment 123, and the ice-making compartment 124 and A main freezer compartment 126 is arranged below the small freezer compartment 125. However, the arrangement of the storage chamber 120 is not limited to the above example. These storage rooms 120 are each opened and closed by a plurality of doors 130.

The display device 140 is a liquid crystal display, an organic EL (Electro Luminescence) display, or the like, and includes a display screen 141 that can display images and/or videos. Display device 140 includes a touch sensor 142 provided overlapping display screen 141 . The actuator 150 includes a compressor that compresses a cooling refrigerant and sends it to the cooler, a blower that sends cold air cooled by the cooler to each storage room 120, and the like.

Control unit 160 controls display device 140 and actuator 150. For example, the control unit 160 can receive a user's operation on the display screen 141 via the touch sensor 142. By controlling the actuator 150, the control unit 160 changes the set temperature range of each storage compartment 120 of the refrigerator 100, changes the control mode of the chilled compartment 122, and cools the freezers 126, 126 more rapidly than in normal operation. It is possible to perform functions of the refrigerator 100 such as "quick freezing" and "quick ice making" which makes ice faster than in normal operation.

However, the configuration of refrigerator 100 is not limited to the above example. Refrigerator 100 may include a camera 170 provided inside housing 110 or on door 130. In this case, the image of the food taken by the camera 170 may be transmitted to the server system 500 and used for food management. This will be discussed later.

<3. Smart speaker>
FIG. 3 is a block diagram showing the functional configuration of smart speaker 200. The smart speaker 200 includes, for example, a microphone 210, a speaker 220, a wireless communication module 230, a storage section 240, and an information processing section 250.

Microphone 210 acquires the user's voice spoken to smart speaker 200 . Speaker 220 outputs audio to the outside of smart speaker 200. The wireless communication module 230 includes a high frequency circuit, an antenna, and the like, and is capable of wireless communication with the router 300, for example.

The storage unit 240 stores an audio DB 241. The voice DB 241 includes, for example, a simple voice recognition DB 241a. Trigger words, which are voice commands for activating the smart speaker 200, are registered in the simple voice recognition DB 241a. "Startup" refers to a state in which power consumption is kept below a specified level and a dormant state (a state in which it does not respond to user utterances other than the trigger word), to a standby state in which it is possible to respond to user utterances other than the trigger word (interaction possible). It means changing the state (a state in which voice commands can be received). The "standby state" may also be referred to as the "starting state." The trigger word is a predetermined specific word or phrase such as "Hello, XX".

The information processing section 250 includes, for example, an audio receiving section 251, an activation determining section 252, a data transmitting/receiving section 253, and an audio output section 254. The voice receiving unit 251 receives the voice (user's utterance) input into the microphone 210 as voice data. The activation determination unit 252 determines whether or not it is necessary to activate the voice interaction function. If the voice data received by the voice receiving unit 251 includes the trigger word registered in the simple voice recognition DB 241a, the activation determination unit 252 determines that activation is necessary, and activates the voice interaction function.

The data transmitting and receiving unit 253 transmits and receives control signals and data to and from the server system 500 by controlling the wireless communication module 230. The data transmitting/receiving unit 253 transmits audio data other than the trigger word acquired by the audio receiving unit 251 to the server system 500. The data transmitting/receiving unit 253 receives response voice data prepared by the server system 500 from the server system 500 . The data transmitting/receiving section 253 outputs the received response voice data to the voice output section 254 . The audio output unit 254 outputs the audio data received from the data transmitting/receiving unit 253 to the outside through the speaker 220 as audio (speech from the smart speaker 200).

<4. Terminal equipment>
FIG. 4 is a block diagram showing the functional configuration of the terminal device 400. Terminal device 400 includes, for example, a display device 410, a wireless communication module 420, a storage section 430, and an information processing section 440. The display device 410 is a liquid crystal display, an organic EL display, or the like, and includes a display screen 411 that can display images and/or videos. Display device 410 includes a touch sensor 412 provided overlapping display screen 411 . The wireless communication module 420 includes a high frequency circuit, an antenna, and the like, and can be connected to the network NW.

The storage unit 430 stores application programs for managing ingredients, remotely controlling the refrigerator 100, and the like. All or part of the information processing unit 440 is realized by executing the above application program by a hardware processor such as a CPU (Central Processing Unit) installed in the terminal device 400.

The information processing section 440 includes, for example, an operation reception section 441, a display device control section 442, and a data transmission/reception section 443. The information processing unit 440 is an example of a “processing unit”.

The operation reception unit 441 receives a user's operation on the display screen 411 via the touch sensor 412. The data transmitting and receiving unit 443 transmits and receives control signals and data to and from the server system 500 by controlling the wireless communication module 420. The data transmitting/receiving unit 443 transmits information indicating the user's operation content accepted by the operation accepting unit 441 to the server system 500. The data transmitting/receiving unit 443 outputs display-related control instructions and display data received from the server system 500 to the display device control unit 442. The display device control unit 442 displays an image or video included in the display data received from the server system 500 on the display screen 411 based on the control instruction received from the server system 500.

However, the configuration of the terminal device 400 is not limited to the above example. For example, it may have a function as a voice dialogue device similar to the smart speaker 200 described above. Further, the terminal device 400 may include a camera 450. In this case, the image of the food taken by the camera 450 may be transmitted to the server system 500 and used for food management. This will be discussed later.

<5. Server system>
FIG. 5 is a block diagram showing the functional configuration of the server system 500. An example in which the server system 500 controls a terminal device 400 that is an "information display device" and a smart speaker 200 that is an "audio dialogue device" will be described below. However, the "information display device" controlled by the server system 500 may be the refrigerator 100. In this case, "terminal device 400" in the following description will be read as "refrigerator 100". The “voice dialogue device” controlled by the server system 500 may be the terminal device 400 or the refrigerator 100. In this case, "smart speaker 200" in the following description can be read as "terminal device 400" or "refrigerator 100."

Server system 500 includes a communication module 510, an information processing section 520, and a storage section 530. Here, for convenience, functional units other than voice dialogue will be explained first, and then functional units related to voice dialogue will be explained. The communication module 510 includes a communication circuit and the like, and is connectable to the network NW.

<5.1 Configuration of information processing unit>
The information processing section 520 includes, for example, an instruction reception section 521, a registered information management section 522, a display control section 523, an audio processing section 524, a control mode change section 525, a control instruction generation section 526, and a response generation section 527. However, some or all of these functional units may be provided in the terminal device 400 or the refrigerator 100 instead of the server system 500. The information processing unit 520 is an example of a “processing unit”.

The instruction receiving unit 521 receives information indicating the user's operation (user's operation on the display screen 411) accepted by the operation receiving unit 441 of the terminal device 400 from the terminal device 400 via the communication module 510. The instruction reception unit 521 outputs information received from the terminal device 400 to the registration information management unit 522.

Based on the information received from the instruction receiving unit 521, the registration information management unit 522 stores various DBs (inventory list DB 601, shopping list DB 602, candidate list DB 604, trash can DB 603, stock supplies DB 607, user settings DB 606, remote control registration DB 609, etc.) to be described later. ) to update. In addition, in this embodiment, the registration information management unit 522 receives from the voice processing unit 524 information indicating the content (user's uttered instructions) analyzed by the voice processing unit 524, which will be described later. The registration information management unit 522 updates the various DBs described above based on the information received from the audio processing unit 524. The registration information management unit 522 will be described in detail later. In the following explanation, "user operation" or "user input" refers to a user operation accepted by the instruction receiving unit 521 (user operation on the terminal device 400), or a user's operation received by the instruction receiving unit 521 (user operation on the terminal device 400), or a user input that is analyzed by the audio processing unit 524, which will be described later. It means a user's operation (operation by speaking to the smart speaker 200).

The display control unit 523 generates display data including images or videos to be displayed on the display screen 411 of the terminal device 400 and control signals based on the contents of various DBs. The display control unit 523 transmits the generated display data and control signal to the terminal device 400 via the communication module 510. The control signal generated by the display control section 523 is a control signal that operates the display device control section 442 of the terminal device 400. Here, "displaying on the display screen" is not limited to the meaning of displaying an image or video on the display screen 411 by the display device control unit 442 of the terminal device 400, and does not mean displaying display data or video from outside to the terminal device 400. It may also include displaying an image or video on the display screen 411 by providing a control signal.

The audio processing unit 524 receives audio data acquired by the smart speaker 200 from the smart speaker 200 via the communication module 510. For example, the audio processing unit 524 receives audio data other than the trigger word from among the audio data acquired by the smart speaker 200. The audio processing unit 524 analyzes audio data received from the smart speaker 200. For example, the voice processing unit 524 converts voice data into text by performing voice recognition processing, and performs semantic interpretation by performing natural language processing on the voice data converted to text. The audio processing section 524 outputs the analysis result of the audio data to the control mode changing section 525, the control instruction generation section 526, and the response generation section 527.

The control mode changing unit 525 changes the operating state of the smart speaker 200 based on the analysis result of the audio data by the audio processing unit 524. The control mode changing unit 525 will be described in detail later.

Based on the analysis result of the audio data by the audio processing unit 524, the control instruction generation unit 526 generates an instruction to the refrigerator 100 (for example, changing the set temperature range of each storage compartment 120, changing the control mode of the chilled compartment 122) based on the user's utterance. , "quick freezing," or "quick ice making"), those instructions will be accepted. Control instruction generation unit 526 generates a control instruction corresponding to the received instruction to refrigerator 100. Control instruction generation section 526 transmits the generated control instruction to refrigerator 100 via communication module 510. Control unit 160 of refrigerator 100 drives actuator 150 based on the control instruction generated by control instruction generation unit 526 to realize control desired by the user.

If there is content to respond to the user's utterance analyzed by the voice processing unit 524, or if there is content to be notified to the user, the response generation unit 527 refers to the voice DB 610 (described later) and generates a smart speaker. 200 to generate audio data to be output. The response generation unit 527 transmits the generated audio data to the smart speaker 200 via the communication module 510. The smart speaker 200 drives the speaker 220 based on the audio data generated by the response generation unit 527 to provide a voice response or notification to the user.

<5.2 Various DBs>
The storage unit 530 stores an inventory list DB 601, a shopping list DB 602, a trash can DB 603, a candidate list DB 604, a food ingredients DB 605, a user setting DB 606, a regular supplies DB 607, a history DB 608, a remote control registration DB 609, and an audio DB 610. An inventory list DB 601, a shopping list DB 602, a trash can DB 603, a candidate list DB 604, a user setting DB 606, a regular supplies DB 607, and a history DB 608 are prepared for each user.

For convenience of explanation, an example will be shown below in which the above-mentioned plurality of DBs exist as DBs for ingredient management, but these DBs may be provided as a single unit and displayed in a specific list (shopping list or candidate list) to be described later. It may be managed by adding identification information (for example, a tag) indicating the food to the registration of each food ingredient. Note that the processes for various DBs (for example, registration, deletion, and confirmation of ingredients) described below are executed by the registration information management unit 522.

<5.2.1 Inventory list DB>
FIG. 6 is a diagram showing an example of the contents of the inventory list DB 601. The "inventory list" is a list in which ingredients owned by the user are managed. The "inventory list" is an example of the "first list." In the following explanation, the "inventory list" may be referred to as the "ingredients list". In the inventory list DB 601, ingredients that are in stock at the user's home are registered. In the inventory list DB 601, not only foods stored inside the refrigerator 100 but also foods stored outside the refrigerator 100 (for example, stored at room temperature) are registered. The inventory list DB 601 is an example of "holding information" indicating the quantity of food materials held by the user.

In the inventory list DB 601, for each ingredient, "management ID", "ingredient name", "ingredient classification", "storage location", "purchase date", "best before date" (or "expiration date"), "quantity", “Unit” and “history” are registered in association with each other. "Each ingredient" means, for example, the same ingredient purchased at the same time. In other words, even if they are of the same type, ingredients purchased on a certain date and time and ingredients purchased on a different date and time are registered as different ingredients. Note that the "purchase date" here refers to the registration date when a specific food ingredient was newly registered, i.e., the date when the stock amount increased, and is used to include cases where the food material was not acquired through purchase. .

“Foodstuff classification” means classification of foodstuffs. In this embodiment, the "foodstuff classification" includes "vegetables", "fruits", "meat", "seafood/seaweed", "eggs/dairy products", "beans/bean products", "processed foods", " There are 16 food categories: rice/bread, noodles, beverages, frozen foods, sweets, seasonings, spices, side dishes, and other foods. It is set. However, the above classification is an example, and the food classification classification can be set as appropriate.

"Storage location" means a location where food ingredients are stored. In this embodiment, four types of storage locations are set as the "storage location": "refrigerator room", "vegetable room", "freezer room", and "other". The “refrigerating room” corresponds to an area that includes the refrigerating room 121 and the chilled room 122. “Vegetable room” corresponds to the vegetable room 123. The "freezer compartment" corresponds to an area that includes the small freezer compartment 125 and the main freezer compartment 126. “Other” corresponds to a storage location outside the refrigerator 100.

However, the above classification is an example, and the storage location classification can be set as appropriate. For example, the refrigerator compartment 121 and the chilled compartment 122 may be separated as separate storage locations. The small freezer compartment 125 and the main freezer compartment 126 may be divided into separate storage locations. Furthermore, the storage location may be divided into smaller sections such as "top shelf of the refrigerator compartment" and "right door pocket of the refrigerator compartment." A "storage location" may also be referred to as a "storage room" or a "storage area." For convenience of explanation, the term "refrigerating compartment 121" may include the chilled compartment 122, and the term freezing compartment 126 may include the small freezing compartment 125.

The latest quantity of ingredients is registered in "Quantity". In the "history" field, when the amount of ingredients decreases, information indicating the quantity after the decrease, the date and time of the decrease, etc. is registered.

The registration information management unit 522 registers, deletes, and confirms ingredients in the inventory list DB 601 based on, for example, user operations. In the present embodiment, the registration information management unit 522 detects foodstuffs with close expiration dates (or expiration dates) a predetermined number of days (for example, one day before) before the expiration dates (or expiration dates) of the foodstuffs registered in the inventory list DB 601. The terminal device 400 is notified of this fact. This notification is displayed on the display screen 411 of the terminal device 400 as a push notification. When the expiration date (or expiration date) of the food ingredients registered in the inventory list DB 601 has passed, the registration information management unit 522 automatically deletes the ingredients whose expiration date (or expiration date) has passed from the inventory list DB 601 and performs the process described later. The food registration is moved to the trash can DB 603 and candidate list DB 604. The "inventory list" is displayed as a "foodstuff list" on the display screen 411 of the terminal device 400 under the control of the server system 500. A display example of the "inventory list (ingredients list)" will be described in detail later.

<5.2.2 Shopping list DB>
FIG. 7 is a diagram showing an example of the contents of the shopping list DB 602. The "shopping list" is a list that the user can refer to when shopping in order to prevent forgetting to buy something, and is a list of foodstuffs for which the user's purchase plan (for example, explicit purchase intention) has been confirmed. The "shopping list" is an example of a "third list" and is a list with a higher possibility of purchase than the candidate list described later. However, the "shopping list" is not limited to the user's explicit purchase intention, but may include foodstuffs registered based on an estimate based on the user's constant stocking intention (intention to keep a certain food item as a regular item). For example, ingredients registered in the "shopping list" are once registered in the "candidate list" described later, and then are moved from the "candidate list" to the "shopping list" based on the user's intention (user operation). This will add it to your shopping list. However, the ingredients to be registered in the "shopping list" may be directly registered in the "shopping list" based on the user's operation, without going through the candidate list. The “shopping list” is displayed on the display screen 411 of the terminal device 400 under the control of the server system 500. A display example of the "shopping list" will be described in detail later.

In the shopping list DB 602, ingredients to be displayed as a "shopping list" are registered. In the shopping list DB 602, a "management ID", "ingredient name", "ingredient classification", "quantity", and "unit" are associated and registered for each food ingredient. The registration information management unit 522 registers, deletes, and confirms ingredients in the shopping list DB 602 based on user operations.

<5.2.3 Trash DB>
FIG. 8 is a diagram showing an example of the contents of the trash can DB 603. Food items deleted from the inventory list DB 601 are temporarily registered in the trash DB 603. For example, food items registered in the inventory list DB 601 that have been consumed and deleted from the inventory list DB based on a user's operation are registered in the trash DB 603. In the present embodiment, the trash DB 603 stores food items that have been automatically deleted from the inventory list DB by the registration management unit 522 due to their expiry date (or expiration date) having passed among the food items registered in the inventory list DB. will also be registered. Food items registered in the trash can DB 603 can be re-registered in the inventory list DB by a user's operation. Food items registered in the trash can DB 603 are automatically deleted from the trash can DB 603 when a predetermined period (for example, three days) has elapsed from the date of registration. Note that even if foods whose expiration date (or expiration date) has passed are registered in the inventory list DB 601, the ingredients may not be automatically deleted from the inventory list DB 601, but may be deleted after waiting for a user's operation. Similarly, foodstuffs registered in the trash can DB 603 may be deleted after waiting for a user's operation, without providing a period for automatic deletion. In the trash can DB 603, for each food item, "management ID", "food name", "food classification", and registration date in the trash can DB 603 are registered in association with each other.

<5.2.4 Candidate list DB>
FIG. 9 is a diagram showing an example of the contents of the candidate list DB 604. The "candidate list" is a list that the user can refer to in order to prevent forgetting to buy something, and is a list of potential purchase ingredients for which the user's purchase plan (for example, explicit purchase intention) has not been confirmed. . The "candidate list" is an example of a "second list." The "candidate list" is an example of information that prompts the user to purchase ingredients based on the fact that the inventory of ingredients has decreased. The "candidate list" is purchase proposal information, and is generated using, for example, a deletion operation of ingredients registered in the inventory list DB 601. The “candidate list” is displayed on the display screen 411 of the terminal device 400 under the control of the server system 500. A display example of the "candidate list" will be described in detail later. Foodstuffs to be displayed as a "candidate list" are registered in the candidate list DB 604. In the candidate list DB 604, "management ID", "foodstuff name", "foodstuff classification", and registration date in the candidate list DB 604 are associated and registered for each food ingredient. As shown below, if the quantity of a specific ingredient registered in the inventory list DB 601 is to be registered in the candidate list DB 604 when it is reduced to a predetermined quantity before the quantity of the ingredient is reduced to zero, the ingredient The "quantity" (quantity of stock) may also be registered in the candidate list DB 604 in association with the food ingredient.

In this embodiment, when the quantity of ingredients registered in the inventory list DB 601 decreases and a predetermined condition is met, the decreased ingredients are automatically registered in the candidate list DB 604. The "predetermined condition" is, for example, that the number of foodstuffs after reduction becomes less than a predetermined threshold. In this embodiment, the "predetermined condition" is that the number of foodstuffs after reduction becomes zero (that is, the inventory of foodstuffs runs out). The "predetermined condition" is an example of the "first condition."

For example, when food items registered in the inventory list DB 601 are deleted and registered in the trash can DB 603, the same food items as those registered in the trash can DB 603 are automatically registered in the candidate list DB 604. "Deletion of registration" means that the stock of a specific food ingredient is gone (changed from being in stock to being out of stock). That is, the candidate list DB 604 includes ingredients that were deleted from the inventory list DB based on the user's operation after the ingredients registered in the inventory list DB 601 were consumed, and ingredients that were registered in the inventory list DB. Foodstuffs that are automatically deleted from the inventory list DB by the registration information management unit 522 when the expiration date (or expiration date) has passed are automatically registered. In both the shopping list DB 602 and the candidate list DB 604, ingredients that are out of stock (or have decreased in stock) are registered. The shopping list DB 602 and the candidate list DB 604 have different update timings. In the server system 500 of this embodiment, when a user deletes the registration of a certain food item from the inventory list DB 601 in response to eating/throwing it away, the display of that food item should not be necessary after that. , which is used as data to recommend shopping.

The registration information management unit 522 can accept a user's operation to move ingredients registered in the candidate list DB 604 to the shopping list DB 602. For example, the user can tap (single press) the display of the food item to be moved in the candidate list to be described later to select it, and move the selected food item to the shopping list. When the selected food item is registered in the shopping list DB 602, the registration of the food item is deleted from the candidate list DB 604, which is the source of movement.

Here, when a certain ingredient is already registered in the candidate list DB 604 and the same ingredient is directly registered in the shopping list DB 602 by user operation (in other words, the ingredient is added to the shopping list DB 602 without going through the candidate list DB 604). ) may be registered. In this case, the registration information management unit 522 registers the food item in the shopping list DB 602 and deletes the registration of the food item from the candidate list DB 604. That is, food ingredients that have already been registered in the shopping list DB 602 will not be registered in the candidate list DB 604 even if the quantity of inventory in the inventory list DB 601 satisfies the above-mentioned "predetermined condition".

When the number of ingredients registered in the stock list DB 601 decreases and the above predetermined conditions are met, the registration information management unit 522 registers the ingredients as a special ingredient registered in advance by the user (for example, registered in the regular supplies DB 607). In the case where the food item is a food item (food item), the food item may not be registered in the candidate list DB 604, but may be directly registered in the shopping list DB 602.

In this embodiment, the maximum number of ingredients (for example, 20) that can be registered in the candidate list DB 604 is determined. If a predetermined number (for example, the maximum number) of ingredients are already registered in the candidate list DB 604, the registration information management unit 522 deletes old registrations in the candidate list DB 604 in response to adding a new ingredient registration to the candidate list DB 604. Delete sequentially.

The registration information management unit 522 deletes old registrations from the candidate list DB 604 according to a priority order based on one or more of the registration date in the candidate list DB 604, the purchase date of the ingredients, the expiration date of the ingredients, or the expiration date of the ingredients. conduct. For example, among the ingredients registered in the candidate list DB 604, the registration information management unit 522 selects the oldest registered ingredients in order of purchase date, oldest expiration date, or oldest expiration date. Perform deletion. Note that when the quantity of a specific ingredient registered in the inventory list DB 601 is reduced to a predetermined quantity before it becomes zero as described above, that ingredient is to be registered in the candidate list DB 604, the inventory list DB 601 Among the foodstuffs still in stock, the foodstuffs may be deleted first in order of their expiration dates or in the order of their expiration dates.

Instead of/in addition to the above priority order, the registration information management unit 522 deletes old registrations in the candidate list DB 604 according to the priority order based on the food classification to which the food item belongs. For example, when a food item belongs to a specific food category, the registration information management unit 522 lowers the priority for deleting the food item (or removes it from deletion targets) to make it difficult for the food item to be deleted from the candidate list DB 604. The "specific food classification" is a food classification that has characteristics similar to regular supplies, such as seasonings and spices.

In place of/in addition to the above priority order, the registration information management unit 522 may delete old registrations in the candidate list DB 604 based on the past inventory frequency of the food ingredient and/or whether the food ingredient is included in the food group specified by the user. Do it according to priority based on whether or not. If the past inventory frequency of the food ingredient is equal to or higher than the threshold, the registration information management unit 522 lowers the priority for deleting the food item (or excludes it from deletion targets), thereby making it difficult for the food item to be deleted from the candidate list DB 604. The past inventory frequency can be obtained based on the history DB 608, which will be described later. When a food item is included in the food group designated by the user, the registration information management unit 522 lowers the priority for deleting the food item (or removes it from deletion targets), making it difficult for the food item to be deleted from the candidate list DB 604. The “designated food group” is, for example, a food group registered in the regular supplies DB 607.

By providing a candidate list, it is possible to reduce the possibility of forgetting to register items on the shopping list. Furthermore, since the candidate list serves as an assisting function for registration to the shopping list, it is possible to reduce the workload of registration to the shopping list.

<5.2.4 Ingredients DB>
FIG. 10 is a diagram showing an example of the contents of the food ingredients DB 605. The food ingredients DB 605 is a default DB that is commonly set for multiple users. The ingredients DB 605 is used for automatic input of ingredients information, which will be described later. A large number of commonly distributed foodstuffs are registered in the foodstuff DB. In the ingredients DB 605, for each ingredient, "ingredient name", "accepted ingredient name candidate", "ingredient classification", "ingredient classification icon", "default storage location (recommended storage location)", "best before date" (or expiration date)" are registered in association with each other. Each of the "ingredients classification", "ingredients classification icon", "default storage location", and "best-before date (or expiration date)" is an example of "associated information." The foodstuff DB 605 is an example of "related information" in which a foodstuff name and accompanying information are associated, and is an example of "first information." However, the "first information" mentioned here is a different concept from the "first information I1" described later. In this embodiment, an icon of a food classification is registered as one of the accompanying information, but an image indicating a food classification may be used instead. The food ingredients DB 605 has a general correspondence relationship between food items and storage locations for the food items (which is set in advance to be common to all users unless there is a special designation by the user, rather than making it different for each user). This is the registered information. This correspondence relationship is an example of a "first correspondence relationship."

As the "ingredient name acceptance candidates," one or more acceptance candidates are registered that take into account input blurring of ingredient names. For example, if the food ingredient is cherry tomatoes, in addition to "cherry tomatoes (written in katakana)," "petit tomato (written in hiragana)" and "tomato" are registered. The "default storage location" is a storage location inside the refrigerator 100 or outside the refrigerator 100, and a storage location suitable for storing foodstuffs is registered. As described above, when the refrigerator compartment 121 has shelves, the "storage location" refers to a part of the storage compartment 120 that is distinguished from other locations, such as the "top shelf of the refrigerator compartment 121." It may also be a classification that indicates. Note that the "Ingredient name acceptance candidate" is only used for the purpose of association with "ancillary information" (that is, the use of "related information"), and the "Ingredient name acceptance candidate" is used as the name of the ingredient shown to the user. The name itself registered by the user may be used instead of being converted using "candidate". For example, if the user inputs "Tomato from ○○", "Tomato" is registered as "Ingredient Name Acceptance Candidates", and the food category "Vegetables" has been prepared in association with "Tomato". At the same time, the name of the ingredients shown to the user on the display screen 411 of the terminal device 400, such as "Tomato from XXX", is registered in the inventory list DB 601. The input name may be used as is without any processing. Alternatively, only the part that is considered to be incorrect may be changed and used in the display, such as "Tomatoes from ○○".

Here, in the refrigerator 100, the arrangement of the storage chamber 120 differs depending on the model. For this reason, the recommended storage location for food items may differ depending on the model. Therefore, in this embodiment, the foodstuff DB 605 is prepared for each model of the refrigerator 100, for example. In other words, the food DB 605 differs depending on the model of the refrigerator 100. The registration information management unit 522 specifies the model of the user's refrigerator 100 based on the model information of the refrigerator 100 obtained by referring to the remote operation registration information DB 609 described later, and selects the food DB 605 corresponding to the specified model. Note that different information for each model of the refrigerator 100 may be prepared not for the entire foodstuff DB 605, but only for a portion of the information such as the information indicating the storage location in the foodstuffs DB 605.

Returning to FIG. 2, the explanation will be continued. In addition to the above-mentioned DB, the storage unit 530 stores a user setting DB 606, a regular supplies DB 607, a history DB 608, a remote operation registration DB 609, and an audio DB 610.

The user setting DB 606 is information registered by users. In the user setting DB 606, user-specific change settings for the ingredients DB are registered in order to provide flexibility in automatic input of ingredients information, which will be described later. For example, in the user setting DB 606, user-specific correspondence relationships between ingredients and storage locations are registered. The correspondence relationship is an example of a "second correspondence relationship." Specifically, in the food ingredients DB 605, "refrigerating room" is registered as the default storage location for "chicken". When the user purchases chicken meat and prefers to store it in the freezer compartment rather than in the refrigerator compartment, the user can register "chicken" and "freezer compartment" in association with each other in the user setting DB 606. The user setting DB 606 is another example of "related information" in which a food ingredient name and accompanying information are associated, and is an example of "second information." However, the "second information" referred to here is a different concept from the "second information I2" described later. The registration information management unit 522 performs registration, deletion, and confirmation in the user setting DB 606 based on user operations.

In the regular supplies DB 607, foodstuffs that the user often uses (for example, foodstuffs that the user does not want to run out of) are registered. In the regular supplies DB 607, foodstuffs (special foodstuffs) that the user has "declared" to keep on hand are registered. The registration information management unit 522 registers, deletes, and confirms the regular supplies DB 607 based on user operations.

The history DB 608 can be used instead of the user setting DB 606 or in combination with the user setting DB 606. In the history DB 608, for example, a user's history regarding preservation of foodstuffs (usage history of the refrigerator 100) is registered without being explicitly registered by the user. By referring to the history DB 608, the registration information management unit 522 can obtain information on which storage room 120 is frequently used for a certain food ingredient. The history DB 608 is generated based on the user's registration results in the inventory list DB 601 and/or the photographic results of the camera 170 of the refrigerator 100.

In the remote control registration DB 607, for each refrigerator 100, a "management ID", a "model", and a "management ID of a terminal device registered for remote control" are registered in association with each other. The display control unit 523 specifies the model of the user's refrigerator 100 from the remote control registration DB 607 based on the management ID of the terminal device 400 received from the terminal device 400, and controls the display screen of the terminal device 400 based on the specified model of the refrigerator 100. The picture of the refrigerator 100 displayed in 411 is changed. This will be described in detail later.

In the voice DB 610, a dictionary for interpreting the meaning of the user's utterances, voice information for generating response voice data, and the like are registered. Furthermore, the storage unit 530 stores image data of icons for food classifications.

<6. Screen display＞
Next, a screen display displayed on the display screen 411 of the terminal device 400 when managing food ingredients in the refrigerator 100 using the terminal device 400 will be described. The screen display described below is realized by the display device control unit 442 of the terminal device 400 executing control based on display data and control instructions transmitted from the server system 500.

<6.1 Top page of the refrigerator management screen>
FIG. 11 is a diagram showing an image displayed on the display screen 411. The left side of FIG. 11 is an image S1 showing a top page displayed when the application program is started on the terminal device 400. The image S1 includes a pattern R0 schematically showing the refrigerator 100 and a plurality of displays D11 to D16. The pattern R0 includes the schematically illustrated patterns R0a to R0f of each storage compartment 120 of the refrigerator 100. The pictures R0a to R0f are pictures showing a refrigerator compartment 121, a chilled compartment 122, a vegetable compartment 123, an ice making compartment 124, a small freezing compartment 125, and a main freezing compartment 126, respectively. Displays D11 to D16 show the setting contents (operating state) regarding each storage chamber 120.

When the user presses the display D17 indicating "settings" located at the upper right of the image S1, the display control unit 523 displays an image S2 indicating a menu superimposed on the image S1. The display control unit 523 causes the display screen 411 to display the image S3 (FIG. 13) of the top page of food management when "food management" included in the image S2 is operated (for example, tapped) by the user.

<6.2 Ingredients management top page>
FIG. 12 shows an image S3 of the food management top page. The image S3 includes a first picture R1, a second picture R2, first to fourth display boxes D1 to D4, and first to fourth icons K1 to K4.

In the present embodiment, in the image S3, patterns R1a to R1e indicating each storage room (storage location) 120 included in the first pattern R1 schematically showing the refrigerator 100, and a pattern related to food stored in the storage room 120 are included in the image S3. 1 information I1 are displayed in association with each other. Image S3 is an example of a "first display." The information included in each of the first to fourth display boxes D1 to D4 is an example of "first information I1." The first information I1 includes information regarding all foods stored in each storage room 120. For example, the first information I1 is the number of foods stored in each storage room 120.

However, the first information I1 may be information indicating the degree of filling (degree of use) of each storage chamber 120. The "degree of filling" is information based on the calculation result of dividing the number of foods stored in each storage chamber 120 by the number of foods when each storage chamber 120 is fully filled, and is the filling rate (usage rate) or It is expressed in terms of filling amount (“high”, “normal”, “low”), etc. The filling rate of each storage chamber 120 can be estimated more accurately by determining the volume and dimensions occupied by the food for each type of food, based on the type and quantity of food currently registered as being stored. It is possible. Instead of the volumes and dimensions of the foodstuffs, each combination of the type and quantity of foodstuffs that can be stored in each storage chamber 120 may be determined in advance. The first information I1 may include the names of the foods stored in each storage chamber 120 instead of/in addition to the number of foods and the degree of filling.

An example related to this embodiment will be described in detail below. The first pattern R1 is a pattern schematically showing the refrigerator 100. In addition to the outer shape of the refrigerator 100, the first pattern R1 includes patterns R1a to R1e that schematically show each storage compartment 120 of the refrigerator 100. The first pattern R1 includes, for example, a line indicating the outer shape of the refrigerator 100 and a line indicating the boundaries between the plurality of storage chambers 120. The first pattern R1 includes a pattern R1a showing the refrigerator compartment 121, a pattern R1b showing the vegetable compartment 123, a pattern R1c showing the ice making compartment 124, a pattern R1d showing the small freezing compartment 125, and a pattern R1e showing the main freezing compartment 126. .

In the present embodiment, the first pattern R1 does not include a pattern indicating the chilled compartment 122, but includes one pattern R1a corresponding to the refrigerator compartment 121 and the chilled compartment 122. In the present embodiment, even if the first pattern R1 is a double-door refrigerator 100, the boundaries between the left and right refrigerator compartment doors are omitted. Due to these, the first pattern R1 has a neat pattern. However, the first pattern R1 may include a pattern indicating the chilled compartment 122, or may include a boundary line between the left and right refrigerator compartment doors 130 in the double door refrigerator 100. Modifications of the first pattern R1 will be described in detail later. The first pattern R1 does not need to accurately represent the dimensions of each door of the actual refrigerator 100, and the dimensional ratio may be adjusted as appropriate to make it easier to see or to display information.

The display control unit 523 identifies the model of the user's refrigerator 100 based on the remote control registration DB 609, and includes the first picture R1 of the arrangement of the storage compartment 120 corresponding to the model of the user's refrigerator 100 in the image S3. In this case, the display control unit 523 may use the same first pattern R1 even if there is a slight difference in size ratio, as long as the arrangement of the doors 130 of the refrigerator 100 is the same.

The second pattern R2 is arranged outside the first pattern R1. For example, the second pattern R2 is placed to the side of the first pattern R1. In this embodiment, the second pattern R2 is arranged to the side of the first pattern R1 at the same height as the bottom of the refrigerator included in the first pattern R1. The second pattern R2 schematically shows a storage location outside the refrigerator 100. The second pattern R2 includes, for example, a pattern indicating a box-shaped storage section.

The first to fourth display boxes D1 to D4 include, for example, a picture representing a speech bubble. The first display box D1 includes first information I1 regarding the foods stored in the refrigerator compartment 121 and the chilled compartment 122. For example, information indicating the number of foods stored in the refrigerator compartment 121 and the chilled compartment 122 is displayed in the first display box D1. The first display box D1 is arranged on the side of the picture R1a of the refrigerator compartment included in the first picture R1. Viewed from another perspective, the first display box D1 is arranged at the same height as the pattern R1a of the refrigerator compartment included in the first pattern R1.

The second display box D2 includes first information I1 regarding the foods stored in the vegetable compartment 123. For example, information indicating the number of foods stored in the vegetable compartment 123 is displayed in the second display box D2. The second display box D2 is arranged on the side of the pattern R1b of the vegetable compartment 123 included in the first pattern R1. Viewed from another perspective, the second display box D2 is arranged at the same height as the pattern R1b of the vegetable compartment 123 included in the first pattern R1.

The third display box D3 includes first information I1 regarding the foods stored in the freezer compartments 125 and 126. For example, information indicating the number of foods stored in the freezer compartments 125 and 126 is displayed in the third display box D3. The third display box D3 is arranged on the side of the patterns R1d and R1e of the freezer compartments 125 and 126 included in the first pattern R1. From another perspective, the third display box D3 is arranged at the same height as the patterns R1d and R1e of the freezer compartments 125 and 126 included in the first pattern R1.

The fourth display box D4 includes first information I1 regarding foods stored outside the refrigerator 100 (foods not stored in the refrigerator 100). For example, information indicating the number of foods stored outside the refrigerator 100 is displayed in the fourth display box D4. The fourth display box D4 is placed directly above the second symbol R2. The first information I1 included in the fourth display box D4 is an example of "outside warehouse information."

The information displayed in the first to fourth display boxes D1 to D4 is generated, for example, by the display control unit 523 based on the inventory list DB 601. According to such a configuration, the usability of the inventory management system can be improved by visually linking each storage compartment 120 of the refrigerator 100 and food information.

In the present embodiment, when a user operation for specifying a first storage location (first storage room) included in the first pattern R1 is received for the first pattern R1, the display control unit 523 controls the first storage location Second information I2 (see FIG. 15), which is more detailed than the first information I1, regarding the food items stored at the location is displayed on the display screen 411. The “user operation to specify the first storage location” is, for example, tapping the first pattern R1 (the pattern indicating the first storage location), but is not limited to this, and corresponds to each storage room 120. This may be done by tapping one of the display boxes D1 to D4, or by speaking through the smart speaker 200.

The display control unit 523 may change the design, color, or size of the storage location corresponding to the storage room 120 tapped in the first design R1. According to such a configuration, it becomes easy to understand that the user has tapped.

In the present embodiment, when a user operation for specifying a first storage location included in the first pattern R1 is accepted for the first pattern R1, a list of foodstuffs stored in the first storage location is displayed on the display screen. 411. In the present embodiment, the display control unit 523 accepts a user operation for specifying a first storage location included in the first pattern R1 for the first pattern R1 while the image S3 is displayed on the display screen 411. If so, the display on the display screen 411 is switched to an image S5 that includes second information I2 (see FIG. 14) that is more detailed than the first information I1 regarding the food ingredients stored in the first storage room. Image S5 is an example of a "second display."

To be more specific, when the user's operation to designate the refrigerating compartment 121 for the first pattern R1 is accepted (when the user taps the pattern R1a or the first display box D1), the display control unit 523 selects the refrigerating compartment. A list of ingredients stored in the room 121 is displayed. Similarly, when the user's operation to designate the vegetable compartment 123 for the first pattern R1 is received (when the user taps the pattern R1b or the second display box D2), the display control unit 523 controls the vegetable compartment 123 displays the list of stored ingredients. When the user's operation to designate the freezing compartments 125 and 126 for the first picture R1 is received (when the picture R1d, R1e or the third display box D3 is tapped by the user), the display control unit 523 controls the freezing A list of ingredients stored in the chambers 125 and 126 is displayed. When the user's operation to designate an external storage location for the first pattern R1 is accepted (when the second pattern R2 or the fourth display box D4 is tapped by the user), the display control unit 523 controls the display of the refrigerator 100. Display the ingredient list for externally stored ingredients. Each of these food ingredient lists is an example of "second information I2." The ingredients list will be explained in detail later.

In the present embodiment, the registration information management unit 522 and the display control unit 523 are configured to allow the user to When the operation in the first mode is accepted, and the user's second mode for each storage room 120 (pictures R1a to R1e or first to fourth display boxes D1 to D4) included in the first pattern R1. A different control instruction may be received depending on the case where the operation is accepted. For example, when the user's operation in the first mode is accepted for each storage room 120 included in the first pattern R1, the display control unit 523 may cause the display to be saved in the selected storage room 120 by, for example, screen transition. The second information I2 regarding the food ingredients is displayed. On the other hand, when the user's operation in the second mode is accepted for each storage room 120 included in the first pattern R1, the registration information management unit 522 and the display control unit 523 display information in the selected storage room 120. A predetermined process is performed on all the stored ingredients at once.

The “first mode of operation” is, for example, a tap (single press) on the display screen 411. The "second mode of operation" is, for example, a long press, double tap, strong press (in the case of a pressure-sensitive display), swipe, flick, etc. on the display screen 411. The "predetermined process" may be any process that is different from the process according to the "operation of the first aspect," but it is possible to set a collective operation common to the food ingredients stored in the selected storage room 120. More suitable. For example, this may be a process of collectively deleting the registration of all foodstuffs stored in the selected storage room 120, or a process of opening a reception window for collective deletion. By providing a function for batch deletion, the contents of the refrigerator 100 can be easily organized. Moreover, if the batch deletion function is associated with the pattern or display box of each storage room 120 included in the first pattern R1, the operability will be improved.

From another perspective, when the user's operation in the first mode is received for each storage room 120 included in the first pattern R1, the registered information management unit 522 and the display control unit 523 The ingredients management screen (registration screen/display screen) will be opened, and even if the operation in the second mode is for the same location, if it is a long press, the process related to bulk deletion of ingredients registration in that storage location will be performed. Good too.

The first icon K1 includes, for example, an arrow-shaped pattern, and is arranged on the side of the first pattern R1. When the first icon K1 is operated (for example, tapped) by the user, the screen switches to a screen displaying a list of all ingredients in stock. This will be described in detail later.

The second to fourth icons K2 to K4 are arranged at the lower end of the image S3. When the second icon K2 is operated (for example, tapped) by the user, the screen switches to a search screen that allows searching for ingredients in stock. When the third icon K3 is operated (for example, tapped) by the user, the screen switches to a screen displaying a shopping list. The third icon K3 is an example of "display of an operation unit for switching to a screen on which foodstuffs that the user is encouraged to purchase" are displayed. This will be discussed later. When the fourth icon K4 is operated (for example, tapped) by the user, the screen switches to a screen displaying a trash can list. The trash can list is a list of food items registered in the trash can DB 603. The trash can list is generated, for example, by the display control unit 523 referring to the trash can DB 603.

In this embodiment, the second to fourth icons K2 to K4 are arranged below the first picture R1. Alternatively, the second to fourth icons K2 to K4 may be arranged above the first picture R1. For example, the third icon K3 may be placed at the upper right of the first pattern R1. The third icon K3 is an example of a "second operation section." The screen including the shopping list that is displayed when the third icon K3 is operated is an example of a "fourth display."

The arrangement of the door of the refrigerator 100 in the first pattern R1 and the patterns of the first to fourth display boxes D1 to D4 may be displayed in appropriate colors to make them easier to distinguish.

FIG. 13 shows an image S3A of a modification of the top page for food management. In this modification, the display control unit 523 displays the first pattern R1A instead of the first pattern R1. The first pattern R1A is a pattern that schematically shows the refrigerator 100 in a state where the left and right refrigerator compartment doors are opened and the inside of the refrigerator compartment 121 is visible. The first pattern R1A includes a pattern R1aa indicating the chilled chamber 122.

For example, the display control unit 523 displays the image S3A as the top page image for food management (that is, in place of the above-mentioned image S3). In another example, the display control unit 523 displays the above-mentioned image S3 as the image of the food management top page, and in the image S3, the pattern R1a indicating the refrigerator compartment 121 or the first display box D1 is operated by the user. The screen may be switched to the image S3A of this modification by being tapped (for example, tapped). In this case, the first pattern R1A is not limited to a pattern corresponding to the entire refrigerator 100, but a pattern that displays the refrigerator compartment 121 and the chilled compartment 122 in an enlarged manner (a pattern in which the other storage compartments 120 are omitted). You may let them.

<6.3 Ingredients list>
FIG. 14 is an image S4 in which the list of all ingredients L0 is displayed. The all foods list L0 is a list in which all foods registered in the inventory list DB 601 (all foods stored inside and outside the refrigerator 100) are displayed. In the list of all ingredients L0, for each ingredient, an icon indicating the ingredient classification, ingredient name, purchase date, expiration date (or expiration date), quantity, and unit are displayed. Food items are arranged in a predetermined order, regardless of where they are stored. The "predetermined order" is, for example, the order of meal name (original order), the order of oldest purchase date (or newest order), or the order of nearest (or farthest) expiration date (expiration date). The user can change the predetermined order (rearrange the display order of the ingredients) by operating (for example, tapping) the operation unit G0 included in the image S4. The present invention is not limited to this, and a function for sorting by storage location may also be provided in accordance with the user's setting operation.

The display control unit 523 displays the image S4 when the first icon K1 of the image S3 (see FIG. 12) is operated by the user. That is, the display control unit 523 generates display data including the list of all ingredients L0 based on the inventory list DB 601 and transmits it to the terminal device 400.

FIG. 15 is an image S5 in which a food list for each storage room 120 is displayed. For example, FIG. 15 is an image in which the food list L2 of the vegetable compartment 123 is displayed. The list of ingredients for the vegetable compartment 123 L2 is a list in which ingredients stored in the vegetable compartment 123 are selected from among the ingredients registered in the inventory list DB 601 and displayed. In the food list L2, for each food item, an icon indicating the food classification, food name, purchase date, expiration date (or expiration date), quantity, and unit are displayed. The ingredients are arranged in a predetermined order, similar to the list of all ingredients L0. The user can change the predetermined order (rearrange the display order of the ingredients) by operating (for example, tapping) the operation unit G0 included in the image S5.

When the picture R1b of the vegetable compartment 123 or the second display box D2 included in the first picture R1 of the image S3 (see FIG. 12) is operated (for example, tapped) by the user, the display control unit 523 displays the picture S5. Display. That is, the display control unit 523 generates display data including the food list L2 for the vegetable compartment 123 based on the inventory list DB 601 and transmits it to the terminal device 400. This also applies to the food list for the refrigerator compartment 121 and the food list for the freezer compartments 125 and 126.

FIG. 16 is a modified image S5A in which a food list is displayed. In this modification, image S5A includes displays of operation units O1, O3, and O4 for switching to another display. Another display to be switched to includes second information I2 regarding foods stored in another storage location instead of second information I2 regarding foods stored in the vegetable compartment 123.

For example, image S5A is for switching between the food list L1 for the refrigerator compartment 121, the food list L2 for the vegetable compartment 123, the food list L3 for the freezer compartments 125 and 126, and the food list L4 for other (stored outside the refrigerator 100). It includes operation parts O1, O2, O3, and O4. When the food list L2 for the vegetable compartment 123 is displayed, each of the food list L1 for the refrigerator compartment 121, the food list L3 for the freezer compartments 125 and 126, and the other food list L4 is an example of a "third display". It is. The four food ingredient lists L1, L2, L3, and L4 are switched by the user operating the operating units O1, O2, O3, and O4. The displays of the operation units O1, O2, O3, and O4 are set in common in the images in which the food list of any storage room 120 is displayed.

<6.4 New registration to ingredients list>
FIG. 17 shows an image S6 that is displayed when newly registering ingredients to the ingredients list (inventory list DB 601). In this embodiment, for example, when a user inputs a food name, the registration information management unit 522 displays the food material classification, storage location, purchase date, expiration date (or expiration date), food classification icon, and unit. Autofill. Each of the food classification, storage location, purchase date, expiration date, food classification icon, and unit is an example of "associated information."

To be more specific, based on the input food name, the registration information management unit 522 sets the food classification, default storage location, and unit that are registered in association with the food name in the food DB 605 as "food classification" and "food classification," respectively. "Save location" and "Unit" are automatically entered. When the registration information management unit 522 receives an input from a user who wants to register accompanying information along with the input of the name of a food ingredient, the registered information management unit 522 gives priority to the input content of the user over the food ingredients DB 605, and stores the above-mentioned food material and the accompanying information set by the user. Register by matching.

In this embodiment, if there is information registered in the user setting DB 606 corresponding to the input ingredient name (for example, a storage location whose settings have been changed), the registered information management unit 522 selects the user setting from the ingredient DB 605. Priority is given to inputting information in the DB 606. That is, the registration information management unit 522 can refer to the food ingredients DB 605 that shows the general correspondence between foods and storage locations, and the user setting DB 606 that shows the user-specific correspondence between foods and storage locations. If the user setting DB 606 exists, the registration information management unit 522 inputs the storage location based on the contents of the user setting DB 606, and if the user setting DB 606 does not exist, the storage location is input based on the food ingredients DB 605. state. Based on the food classification obtained by referring to the food food DB, the registration information management unit 522 acquires information indicating the icon of the corresponding food classification from the storage unit 530 and automatically registers the information. The registration information management unit 522 automatically registers the date when the food ingredient name is input as the "purchase date and time" and automatically registers the date and time of the expiration date (or expiration date) estimated based on the purchase date and time. According to such a configuration, it is possible to reduce the time and effort required when registering foodstuffs.

The registration information management unit 522 may automatically register the accompanying information based on the history DB 608 instead of or in combination with the user setting DB 606. That is, if the history DB 608 of the target food ingredient exists, the registration information management unit 522 sets the storage location to be input based on the history DB 608, and if the history DB 608 of the target food ingredient does not exist, the registration information management unit 522 inputs the storage location to the food ingredient DB 605. The storage location may be input based on the information. In this way, when registering a specific food material, if the accompanying information is input according to the history DB 608 instead of the food material DB 605, the condition may be that the frequency of registration of the food material is equal to or higher than a predetermined value. Of course, the frequency of registration of a specific food ingredient can be obtained by referring to the history DB 608.

The registration information management unit 522 inputs a first storage location as accompanying information when a predetermined condition is met regarding a certain food ingredient, and inputs a second storage location as accompanying information when the predetermined condition is met. It will be in the input state. The "predetermined condition" is a condition regarding the degree of filling of the first storage location or the second storage location, or the amount of foodstuffs to be registered as inventory. For example, in the case where "refrigerating compartment" is associated with "pork" in the food ingredients DB 605, if the filling rate of the refrigerating compartment 121 is higher than the threshold value, the user stops filling the refrigerator compartment 121 and stores it in the freezer. 126, and registers "freezing room" as the accompanying information for the pork. In another example, when "refrigerating room" is associated with "pork" in the food ingredients DB 605, if the amount of pork is higher than a threshold value, the accompanying information for the pork is " "Freezer room" is registered.

The registration information management unit 522 changes the automatically input expiration date (or expiration date) depending on whether the food is stored in the refrigerator compartment 121 or in the freezer compartments 125 and 126, even if the food is the same. You may. For example, the registration information management unit 522 sets the number of days until the expiration date (or expiration date) as the first number (for example, 3 days) when the same food is stored in the refrigerator compartment 121, and the freezer compartment 125, 126, the number of days until the expiration date (or expiration date) may be set to a second number of days (for example, one week) that is longer than the first number of days.

The registered information management unit 522 can accept a user's operation to change the accompanying information after the accompanying information is input based on the food ingredients DB 605. When the registration information management unit 522 receives such a user operation, it replaces the accompanying information automatically registered in the inventory list DB 601 with content based on the user operation.

In the example described above, when the user inputs a food ingredient name into the terminal device 400, the registration information management unit 522 controls the food ingredient DB 605 and/or The accompanying information is automatically input based on the user setting DB 606. The user's input of a food ingredient name into the terminal device 400 is an example of "input of information indicating the name of a food ingredient from the outside."

However, the "input of information indicating the names of ingredients from outside" is not limited to manual input by the user. For example, the registered information management unit 522 receives an image taken by the camera 170 of the refrigerator 100 or the camera 450 of the terminal device 400, identifies the food based on the received image, and identifies the food name and food classification corresponding to the identified food. , storage room, date of purchase, expiration date (or expiration date), food classification icon, unit, etc. may be automatically input. In this case, the registration information management unit 522 may identify the food material from the received image, for example, based on a learned model that has been machine learned for food image recognition. Furthermore, the registered information management unit 522 accepts the input of the ingredient name by the user's utterance on the smart speaker 200 instead of the user's manual input of the ingredient name, and the registered information management unit 522 receives the input of the ingredient name by the user's utterance on the smart speaker 200, and selects the ingredient classification, storage room, and purchase date corresponding to the accepted ingredient name. , expiration date (or expiration date), food classification icon, unit, etc. may be automatically input.

In one aspect, the food DB 605 may include a first DB in which food names and food classifications are associated, and a second DB in which food classifications and storage locations are associated. In this case, the registered information management unit 522 identifies the food classification corresponding to the food material based on the first DB and information indicating the food name inputted from the outside, and uses the specified food classification and the second DB to determine the storage location. The specified storage location may be automatically input as accompanying information.

FIG. 18 shows an image S6A of a modification example in which food is newly registered in the food list. In this modification, the characters indicating the content input by the user and the characters indicating the content automatically input by the registration information management unit 522 are displayed in different manners. For example, characters indicating the content input by the user are shown in black, and characters indicating the content automatically input by the registration information management unit 522 are displayed in a different form (such as blue or gray) as the temporary registration state. (indicated by a broken line in FIG. 18). In this case, the registration information management unit 522 may set the contents displayed in the different manners to the officially registered state based on the user's confirmation operation for the contents displayed in the different manners. According to this modification, it is possible to easily distinguish and confirm the content input by the user himself and the content input automatically.

<6.4 Changing the ingredients list>
FIG. 19 shows an image S7 when changing (deleting/moving) the food list. FIG. 19 illustrates the food list L2 for the vegetable compartment 123. The user can select a food ingredient by manipulating (for example, tapping) the display of the food material to be selected in the food list L2. When a food ingredient is selected, first to third icons K5 to K7 are displayed at the bottom of the image S7.

The first icon K5 is an icon for moving the registration of the selected food ingredient to another storage room. When the first icon K5 is operated (for example, tapped) by the user, the display control unit 523 displays a display S8 superimposed on the image S7 to prompt the user to select a destination room. The registered information management unit 522 changes the storage location associated with the food ingredient in the inventory list DB 601 based on the user's selection result on the display S8. As a result, the selected food item is deleted from the list of food items in the storage location before movement.

The second icon K6 is an icon for adding ingredients to a shopping list (shopping list DB 602), which will be described later. When the second icon K6 is operated (for example, tapped) by the user, the display control unit 523 adds the selected food item to the shopping list DB 602. In this case, the registration of the ingredients in the ingredients list remains unchanged.

The third icon K7 is an icon for moving the registration of the selected food ingredient to the trash can. When the registration information management unit 522 and the third icon K7 are operated (for example, tapped) by the user, the selected food item is deleted from the inventory list DB 601 and the deleted food item is registered in the trash can DB 603. As described above, in this embodiment, when a food item deleted from the inventory list DB 601 is registered in the trash can DB 603, the same food item is also registered in the candidate list DB 604.

<6.5 Trash list>
FIG. 20 shows an image S8 including a trash can list L5. The trash can list L5 is a list in which food items registered in the trash can DB 603 are displayed. In the trash can list L5, for each food item, an icon indicating the food classification, food name, purchase date, expiration date (or expiration date), quantity, and unit are displayed. In the image S8, a display D21 is displayed that informs the user that the food items moved to the trash can list L5 will be automatically deleted after a predetermined number of days (for example, three days) have passed since the food items were moved.

<6.6 Shopping list and candidate list>
FIG. 21 shows an image S9 including a shopping list L6 and a candidate list L7. The shopping list L6 is a list in which ingredients registered in the shopping list DB 602 are displayed. In the shopping list L6, for each ingredient, an icon indicating the ingredient classification, ingredient name, planned purchase quantity, and unit are displayed. The display control unit 523 generates display data including the shopping list L6 based on the shopping list DB 602 and transmits it to the terminal device 400.

On the other hand, the candidate list L7 is a list in which foods registered in the candidate list DB 604 are displayed. In the candidate list L7, an icon indicating a food classification and a food name are displayed for each food material. The display control unit 523 generates display data including the candidate list L7 based on the candidate list DB 604 and transmits it to the terminal device 400.

In this embodiment, the shopping list L6 and the candidate list L7 are displayed on the same screen on the display screen 411. For example, the shopping list L6 and candidate list L7 are displayed vertically side by side. The term "same screen" refers to a continuous screen that has continuous display content and can be displayed without switching the entire display content in response to an external operation. "Same screen" is not limited to cases where the display is always displayed at the same time, but is an operation in which the displayed content changes part by part in response to the user's sliding operation on the screen without changing the entire display content ( A screen displayed in response to a so-called scroll operation also corresponds to the "same screen" herein. For example, if there are many ingredients registered in the shopping list K6 and the display screen 411 is occupied by the shopping list L6, by scrolling down the shopping list L6, the candidate list L7 will be displayed following the shopping list L6. Including cases where In other words, "the same screen" includes a case where only a part (for example, the lower end) of the shopping list L6 and the candidate list L7 are displayed at the same time.
In this embodiment, the shopping list L6 and the candidate list L7 are displayed on the same screen on the display screen 411. For example, the shopping list L6 and candidate list L7 are displayed vertically side by side. "Same screen" means screens that can be viewed by users at the same time. However, "the same screen" is not limited to the case where they are always displayed at the same time. "Same screen" means, for example, if there are many ingredients registered in the shopping list K6 and the display screen 411 is occupied by the shopping list L6, by scrolling the shopping list L6 downward, you can continue the shopping list L6. This also includes the case where the candidate list L7 is displayed. In other words, "the same screen" includes a case where only a part (for example, the lower end) of the shopping list L6 and the candidate list L7 are displayed at the same time.

For example, with the shopping list L6 and the candidate list L7 displayed simultaneously, the registration information management unit 522 accepts a user's operation to move the registration of ingredients included in the candidate list L7 to the shopping list L6. That is, the user can move the ingredients displayed on the candidate list L7 to the shopping list L6.

FIG. 22 is a diagram showing another image S10 including the shopping list L6. The display control unit 523 hides the candidate list L7 on a screen that can display the shopping list L6 and the candidate list L7 side by side, or makes the candidate list L7 less noticeable compared to the shopping list L6 (for example, by making the color lighter). ) It is possible to accept user operations. In this embodiment, the candidate list L7 can be closed based on a user's operation. For example, when the display of an item indicating the candidate list L7 in the image S9 is operated (for example, tapped) by the user, the display control unit 523 displays the image S0 including both the shopping list L6 and the candidate list L7 as a candidate list. Switching to image S10 in which shopping list L6 is displayed with L7 closed. Thereby, the user can shop while checking only the shopping list L6, and it is possible to suppress purchasing mistakes compared to the case where the candidate list L7 is displayed.

<7. Voice dialogue function>
<7.1 Overall configuration of control>
Next, the voice interaction function of the server system 500 of this embodiment will be explained. In the present embodiment, the information processing unit 520 of the server system 500 can register or delete a plurality of foodstuffs by the end of one reception of the voice interaction function. In other words, the audio processing unit 524 can successively receive a plurality of instructions included in a group of user's utterances (a series of utterances, a series of dialogues). Note that activation of the voice interaction function (as described above, sometimes expressed as "voice recognition function") means transitioning to a state in which utterances of arbitrary content from the user can be accepted. When the voice interaction function is activated, the device executes instructions included in the user's utterances and responds to the user's utterances. When the voice dialogue function ends, the user's utterances are not accepted except for the trigger word that is a signal to start the voice dialogue function. The period from when the voice dialogue function is activated once until the voice dialogue function is terminated is called "one reception of the voice dialogue function."

Specifically, after the trigger word is spoken to the smart speaker 200 and the voice interaction function is activated, the voice processing unit 524 ends the voice interaction function when a predetermined termination condition is met. The control mode changing unit 525 selects a first mode in which the voice interaction function is terminated under a first termination condition, and a second mode in which the voice interaction function is terminated under a second termination condition different from the first termination condition, based on the user's operation. mode can be selected.

The first mode is a control mode that accepts one instruction uttered by the user. In the first mode, when one instruction is executed in one activation of the voice interaction function, the activation mode ends. "Ending the startup mode" means, for example, that the server system 500 sends a control signal to the smart speaker 200 to end the standby state or a control signal indicating that there is no need to continue the standby state. means. Examples of controls that are set to be accepted in the first mode include controls related to the functions of refrigerator 100. Controls regarding the functions of the refrigerator 100 include, for example, changing the set temperature range of each storage compartment 120, changing the control mode of the chilled compartment 122, "quick freezing", or "quick ice making". The first termination condition is that one control is executed according to the content of the user's utterance, or that the user does not speak for a certain period of time or more after starting the voice interaction function (this is a so-called timeout, and the present invention ). In the first mode, in order to input a plurality of instructions, it is necessary to utter a trigger word and activate the voice interaction function once for each instruction.

The second mode is a control mode in which a plurality of instructions uttered by the user can be accepted in one activation of the voice interaction function. That is, the second mode is a control mode in which the activated state continues even if one instruction is received in one activation. In addition, in the second mode, even if multiple processing targets are included in one set of user's utterances, there is a function that maintains memory so that instructions for those multiple processing targets can be executed. be. Here, "one set of user's utterances" refers to a series of utterances that the user continues to make without intervening responses from the smart speaker 200 or execution of control based on the content of the utterances. In this embodiment, the instructions accepted in the second mode are instructions regarding food management. Instructions regarding food management include, for example, registration, deletion, or inventory confirmation regarding multiple food materials. The second termination condition is that an instruction to terminate the voice interaction function has been received based on a user's action such as utterance. In this embodiment, when the second mode is executed, if the user does not speak for a predetermined period of time, the smart speaker 200 outputs a voice asking for additional instructions (termination confirmation voice). If the user's response to the second termination condition satisfies a predetermined condition, it is assumed that the second termination condition is satisfied. The second termination condition will be described in detail later. In this embodiment, the control mode changing unit 525 executes the second mode and receives instructions from the user regarding registration, deletion, and confirmation of food ingredients in various DBs.

In this embodiment, the control mode changing unit 525 executes the second mode based on the user performing a predetermined action. On the other hand, the control mode changing unit 525 executes the first mode when the user does not perform a predetermined operation. The "predetermined action" is, for example, a user's utterance including a predetermined word, or an operation by a predetermined operation unit (e.g., a predetermined button) provided on the smart speaker 200 (or terminal device 400, refrigerator 100). or the door 130 of the refrigerator 100 is opened (and/or closed).

The "predetermined word" is a word related to food ingredients. "Words related to ingredients" are not limited to the name of the ingredients themselves, but also include synonyms. "Predetermined words" include words that indicate the ingredient name itself, words that indicate ingredient classification such as "vegetables" and "fruits," "ingredient management," "ingredient registration," "inventory management," "inventory registration," and so on. This may be a word related to food management such as "new purchase" or a word that requires the user to specify that it is a continuous skill.

In other words, the control mode changing unit 525 executes the first mode when the user's utterance includes a word related to the function of the refrigerator 100, and when the user's utterance includes a word related to food or a synonym of food, Execute the second mode.

In the present embodiment, in the first mode, the audio processing unit 524 terminates the activation state when the first waiting time elapses while waiting for the user to speak, and in the second mode, the voice processing unit 524 terminates the activation state while waiting for the user to speak. The activated state continues until the second standby time, which is longer than the standby time, elapses. For example, in the first mode, the audio processing unit 524 terminates the activation state after waiting for a first waiting time after the user's voice stops, and in the second mode, after the user's voice stops, the audio processing unit 524 terminates the activation state. After waiting for a second standby time that is longer than the first standby time, the activated state is ended. The first waiting time is, for example, several seconds. The second waiting time is, for example, more than twice the first waiting time, for example, 30 seconds or more, and for example, one minute or more. The second waiting time is set longer than the first waiting time because the user may open the door 130 of the refrigerator 100 and speak about food management while checking the inside.

From another perspective, in the first mode, the audio processing unit 524 executes a process according to the content of the user's utterance, waits for a third waiting time, and then moves to the next process, and in the second mode, After executing the process according to the content of the user's utterance, the process waits for a fourth standby time that is longer than the third standby time, and then moves to the next process. For example, in the first mode, the voice processing unit 524 executes a process according to the content of the user's utterance, waits for a third waiting time, and then shifts to the next process, and in the second mode, the voice processing unit 524 The process may proceed to the next process after waiting for a fourth standby time that is longer than the third standby time after the voice stops. The "next process" in the first mode is, for example, a process that provides audio guidance that the process based on the user's instructions has been completed (or has started) (e.g., a voice that indicates "quick freezing has started"). output). The "next process" in the second mode may be a process that provides audio notification that the process based on the user's instructions has been completed (for example, a process that outputs a voice that indicates "registration has been completed"), It may also be a process of outputting a voice (a voice for confirming completion, which will be described later) asking whether there are any additional instructions regarding management.

In the modified example, in the first mode, the audio processing unit 524 executes processing according to the user's utterance and then moves to the next processing without waiting time, and in the second mode, the audio processing unit 524 performs processing according to the user's utterance. After stopping, it waits for a predetermined waiting time, and then moves to the next process. Similarly to the above, the "next process" in the first mode is, for example, a process that provides audio guidance that the process based on the user's instruction has been completed (or started). The "next process" in the second mode is similar to the above, a process that provides audio notification that the process based on the user's instructions has been completed (for example, a process that outputs a voice that indicates "Registration has been completed") ), or it may be a process of outputting a voice (a voice for confirming completion, which will be described later) asking whether there are any additional instructions regarding food management.

In the present embodiment, when the second mode is executed, the voice processing unit 524 outputs a voice asking whether there is an additional instruction regarding food management after the user's voice stops, and the user's response to the voice output is The activation state is terminated when a predetermined condition is satisfied. The "predetermined conditions" include, for example, the user's response is that there is an utterance indicating that there is no additional instruction, that there is an utterance that affirms the end of the voice dialogue function, or that the user does not speak for a certain period of time or more (so-called timeout). This is the case.

<7.2 Control flow>
FIG. 23 shows a flowchart showing the flow of processing related to the voice interaction function. The following is an example in which the second mode is executed when a predetermined word is included in the user's utterance.

First, when the user speaks, the information processing unit 250 of the smart speaker 200 determines whether the user's utterance includes a trigger word (S101). The information processing unit 250 of the smart speaker 200 continues the dormant state when the trigger word is not included in the user's utterance (S101: NO). On the other hand, if the trigger word is included in the user's utterance (S101: YES), the information processing unit 250 of the smart speaker 200 activates the voice interaction function of the smart speaker 200 (S102). Thereby, the smart speaker 200 acquires the audio data of the user's utterance and transmits the acquired audio data to the server system 500.

The information processing unit 520 (hereinafter simply referred to as “information processing unit 520”) of the server system 500 analyzes the audio data received from the smart speaker 200, and interprets the user's utterances by predetermined words (for example, related to ingredients or synonyms of ingredients). (word) is included (S103).

If the user's utterance does not include the predetermined word (S103: NO), the information processing unit 520 receives the user's instruction in the first mode (S104), and executes the process corresponding to the received instruction (S105). . For example, when information processing section 520 receives an instruction regarding a function of refrigerator 100, information processing section 520 generates a control instruction for causing refrigerator 100 to execute the function, and transmits the generated control instruction to refrigerator 100.

Then, the information processing unit 520 receives the response of the refrigerator 100 to the transmitted control instruction, and if the response is normal, generates a voice indicating that the instruction received from the user has been executed (or started), Output from the smart speaker 200. After that, the information processing unit 520 causes the smart speaker 200 to end the activated state by transmitting a control signal for ending the standby state or a control signal indicating that there is no need to continue the standby state. , shifts the smart speaker 200 to a hibernation state (S106).

On the other hand, if the user's utterance includes a predetermined word (S103: YES), the information processing unit 520 switches the control mode from the first mode to the second mode, and receives the user's instruction in the second mode (S111). ), and executes processing corresponding to the received instruction (S112). For example, the information processing unit 520 executes registration, deletion, and confirmation of a plurality of ingredients in various DBs (for example, the inventory list DB 601) based on user instructions. For example, the information processing unit 520 sequentially executes processing starting from one or more user instructions whose content has been determined, without checking with the user whether there are any additional instructions. Alternatively, the information processing unit 520 may ask the user whether or not there are any additional instructions, and after confirming that there are no additional instructions, execute the multiple instructions received so far at once. good.

Next, the information processing unit 520 determines whether there is a termination instruction from the user (S113). The end instruction is, for example, a user's utterance indicating that there are no additional instructions, and is a word or phrase such as "end", "that's all", "that's all", "thank you", etc. Instead of/in addition to this, the termination instruction is given when a predetermined operation section (for example, a predetermined button) provided on the smart speaker 200 (or terminal device 400, refrigerator 100) is operated (or pressed). The operation unit may be released), the door 130 of the refrigerator 100 may be closed, etc.

When the user gives the termination instruction (S113: YES), the information processing unit 520 transmits a control signal to terminate the standby state or a control signal indicating that there is no need to continue the standby state to the smart speaker 200. As a result, the activated state of the smart speaker 200 is ended, and the smart speaker 200 is transitioned to a dormant state (S106).

On the other hand, if there is no termination instruction from the user (S113: NO), the information processing unit 520 generates a termination confirmation voice asking the user whether there are any additional instructions (for example, a voice saying "Is there anything else?"). ) is output from the smart speaker 200 (S114). Then, the information processing unit 520 determines whether the user has uttered an additional instruction (S115). In other words, even if the information processing unit 520 receives one or more instructions from the user, if there is no termination instruction from the user, the information processing unit 520 does not terminate the voice interaction function and continues the standby state.

If there is an additional instruction from the user (S115: YES), the information processing unit 520 returns to S112 and executes a process corresponding to the additional instruction. On the other hand, if there is an additional instruction from the user (S115: NO), the information processing unit 520 returns to S113 and confirms again whether there is any termination instruction from the user. Then, if there is no termination instruction from the user even after outputting the termination confirmation voice one or more times (for example, multiple times), the information processing unit 520 terminates the voice interaction function due to a timeout.

<7.3 Processing example>
The information processing unit 520 determines that the first food ingredient and the first storage location are uttered in a relationship, and the second food ingredient and the second storage location are uttered in a relationship by the end of one reception of the voice interaction function. If so, the first food material and the first storage location are processed in association with each other, and the second food material and the second storage location are processed in association with each other. For example, if the user's utterance is "Add three eggs to the refrigerator compartment, add one open horse mackerel to the freezer compartment, and delete one paprika from the vegetable compartment," the registered information management unit 522 may say, "Eggs." Processing is performed by associating "refrigerator" with the storage location for "horse mackerel", "freezer room" is processed by associating it with the storage location of "horse mackerel", and processing is performed by linking "vegetable room" with the storage location of "paprika".

If utterances regarding multiple ingredients are made before or after the utterance regarding the multiple ingredients and one storage location is uttered before or after the utterance regarding the multiple ingredients, the information processing unit 520 determines whether the information processing unit Processing is performed in association with the above one storage location. For example, if the user's utterance is "Add 3 eggs and 1 opened horse mackerel to the refrigerator," or "Add 3 eggs and 1 opened horse mackerel to the refrigerator." In some cases, the registration information management unit 522 associates and processes "refrigerating room" as a storage location for "eggs" and "horse mackerel".

In this embodiment, if the name of the food ingredient can be recognized based on the content of the user's utterance, it is not necessary to utter the information regarding the storage location and/or quantity of the food ingredient. For example, if there is no utterance regarding the storage location or quantity, the information processing unit 520 may leave those information blank and proceed with registration. Alternatively, the information processing unit 520 may automatically input the default storage location registered in the food ingredients DB 605 as the storage location. Alternatively, the information processing unit 520 may cause the smart speaker 200 to output a voice asking the user about the missing information.

In this embodiment, the food inventory status managed by the voice interaction function can also be changed by inputting the contents of the display screen 411 displayed on the terminal device 400. When the food stock status is changed by an input operation on the contents of the display screen 411, the information processing unit 520 manages the food stock stock status using the voice dialogue function based on the changed stock status.

Management of the possession status of food ingredients includes at least information on the name of the food material and the quantity of the food material. In the voice interaction function, if the name of the food ingredient is accepted based on the user's utterance, but the quantity of the food ingredient is not included in the utterance content, the information processing unit 520 inputs a predetermined name based on a predetermined rule. It is treated as if the quantity (for example, 1 piece or 1 pack) has been registered, and the registration of the food material is accepted. In the modified example, in the voice interaction function, the information processing unit 520 accepts the name of the ingredient based on the user's utterance, but if the quantity of the ingredient is not included in the utterance, the information processing unit 520 receives the name of the ingredient based on the user's utterance. If information indicating the quantity of the food ingredient cannot be obtained based on the user's utterance after outputting the questioning voice, the registration of the food food item will not be accepted.

In the present embodiment, when notifying the user of the inventory of ingredients, the information processing unit 520 first determines how many ingredients are in stock when the number of ingredients in inventory is a first predetermined number or more (for example, 6 or more). , and then outputs the names of a second predetermined number (for example, 5) of ingredients that are smaller than the first predetermined number by voice, and then reads out the names of the remaining ingredients. Generates and outputs a voice that asks for a question. For example, if there are 20 ingredients, the information processing unit 520 first outputs audibly saying, "There are 20 ingredients in total," and then assigns a predetermined priority order (for example, date of purchase, expiration date, The system outputs the names of the five highest-priority ingredients (due date) in a voice, and then outputs a voice asking the user whether to read out the remaining ingredients as well. That is, when many ingredients are registered, it is difficult to listen to them all at once, so the user is asked to speak in several parts.

In this embodiment, when deleting ingredients registered in the inventory list DB 601 and there are multiple ingredients with the same name, the registration information management unit 522 deletes the ingredients based on the purchase date, expiration date, or expiration date. , select the registration you want to delete. For example, if there are multiple foods with the same name, the registration information management unit 522 deletes registrations first, giving priority to foods with an older purchase date, foods with a near expiration date, or foods with a near expiration date. For example, if 3 tomatoes were registered on February 15th and 2 tomatoes were registered on February 16th, if the user utters "Delete 4 tomatoes", 2 tomatoes will be registered on February 16th. A total of 4 tomatoes will be deleted, including 3 tomatoes dated February 15th and 1 tomato dated February 16th. The method of deletion is not limited to this. For example, if a storage location is specified, such as "delete tomatoes from the vegetable drawer," it is recommended to delete only the ingredients in that storage location. Similarly, when checking the inventory of ingredients, if the user specifies a storage location, the quantity of ingredients may be returned only for the ingredients in that storage location. Further, when deleting the food item, if the user does not utter the quantity, the food item itself may be deleted.

In the present embodiment, if there is a missing ingredient that is not in stock or is less than a predetermined amount, the response generation unit 527 generates and outputs a voice prompting the user to add the missing ingredient to the shopping list. This saves the effort of adding ingredients to the shopping list.

Examples of utterances are shown below. U indicates the user, and S indicates the smart speaker 200.
(Example utterance 1)
U: “Hello 〇〇 (trigger word), open △△ (app name)”
S: “Yes, what is it?”
U: "Food management"
S: "Yes"
U: "Register 3 carrots"
S: “I will register 3 carrots.”
U: "3 cucumbers registered"
S: “I will register three cucumbers.”
U: "That's all"
S: "Ending food registration" (other than trigger word) voice dialogue function stopped

(Example utterance 2)
Food registration is accepted when information including at least the name of the food material and the quantity of the food material is complete, and the user is asked if there is any missing information.
U: “Hello 〇〇 (trigger word), open △△ (app name)”
S: "Yes"
U: “Please register your ingredients.”
S: "Yes"
U: "3 tomatoes"
S: "3 tomatoes"
U: "Egg"
(Predetermined time elapsed)
S: “How many eggs?”
U: "10"
S: "I have registered 10 eggs."
S: “What else?”
U: "No"
S: "Ingredients registration is finished."

<7.3 Examples>
(Example 1)
Ingredients are registered by associating the name of the ingredient with the quantity of the ingredient, so even if there is no quantity information in the utterance when instructing to register the ingredient, the voice recognition function can register the ingredient without inquiring about the quantity. Accept name registration. For example, for ingredients for which there is no quantity information, the field is left blank. Alternatively, for ingredients for which there is no quantity information, a predetermined quantity is registered. In this case, the predetermined amount is registered so that it can be set by the user, or the minimum unit amount is registered for ingredients for which there is no quantity information.

(Example 2)
Once food registration is accepted through voice recognition, the accepted content is then read out loud. The reception contents will be read aloud in a repeating manner, with each food item being read out aloud. In this case, only the names of the ingredients are read out from the received content. At the time of registration, voice guidance prompting the user to input the quantity of unknown ingredients is provided (assuming that the system has both an ingredient registration function based on voice recognition and an ingredient registration function based on manual input).

(Example 3)
For food registration, once the food registration is received in advance through voice recognition, all information on food names recognized by a series of voice recognition functions is then accepted as a food registration instruction. For example, if the verb V represents the registration of a food ingredient, and the object O represents the name of the food material to be registered, then if the user's utterance is "V (O1, O2, O3, O4)", the smart The response of the speaker 200 is "V was applied to O1, O2, O3, and O4."

On the other hand, if the user starts speaking the name of an ingredient without first hearing instructions to register the ingredient, the user should memorize the ingredient name and wait (or ask) for further instructions (registration/deletion/inventory availability). . For example, if the user's utterance is "O1, O2, O3, O4", the smart speaker 200's response is "...O1, O2, O3, O4. Do you want to use these as V1 or V2? ”.

Each of the control unit 160 of the refrigerator 100, the information processing unit 250 of the smart speaker 200, the information processing unit 440 of the terminal device 400, and the information processing unit 520 of the server system 500 described above is, for example, a CPU installed in each device. This is realized by a hardware processor such as a central processing unit (Central Processing Unit) or a graphics processing unit (GPU) executing a program (software). However, all or part of each of the control unit 160, information processing unit 250, information processing unit 440, and information processing unit 520 may be implemented using an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate). It may be realized by hardware (including a circuit section) such as an Array, or it may be realized by cooperation between software and hardware.

Although the embodiment has been described above, the content of the embodiment is not limited to the above example. For example, FIGS. 24 to 38 show modified examples of images displayed on the display screen 411 of the terminal device 400 (or the display screen 141 of the refrigerator 100). For example, FIGS. 24 and 25 are examples in which the first information I1 is displayed in a different manner instead of the first to fourth display boxes D1 to D4. Further, FIG. 25 is a pattern in which the handle portion of each door 130 is also illustrated.

26 to 31 show modified examples of the first pattern R1. That is, the pattern R1 schematically showing the refrigerator 100 may be changed depending on the shape of the refrigerator 100, the door arrangement, and the like. Display boxes D1 to D4 shown in FIGS. 12 and 13 or similar displays are added to these FIGS. 26 to 31 to link each storage chamber 120 and the food information of each storage chamber 120. It can be displayed.

32 to 38 show modified examples of the patterns of the first to fourth display boxes D1 to D4. That is, the first to fourth display boxes D1 to D4 are not limited to rectangular shapes, but may have a round shape, an elliptical shape, an elliptical shape, or the like as appropriate. Furthermore, the positions of the first to fourth display boxes D1 to D4 are not limited to the sides of the refrigerator picture, but may be above, diagonally above, below, diagonally below, or the like. For example, FIG. 35 is an example in which a part of the display box is displayed partially superimposed on a picture of a refrigerator. Moreover, the contents shown in FIGS. 12, 13, and 24 to 38 can be used in appropriate combination with each other.

According to at least one embodiment described above, it is possible to improve usability by having a processing unit that can register a plurality of types of foodstuffs by the end of one reception of the voice interaction function.

Although several embodiments of the invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention as well as within the scope of the invention described in the claims and its equivalents.

(Additional note)
[1] The information processing system of the embodiment includes a processing unit capable of registering a plurality of types of foodstuffs by the end of one reception of the voice interaction function.
[2] In the information processing system described in [1] above,
The processing unit is capable of executing a first mode in which the activated state is terminated when one instruction is received in one activation, and a second mode in which the activated state is continued even if one instruction is received in one activation. The second mode is executed for food registration.
[3] In the information processing system described in [2] above,
The processing unit executes the second mode based on the user performing a predetermined action.
[4] In the information processing system described in [3] above,
The predetermined action is to make an utterance that includes a predetermined word.
[5] In the information processing system described in [4] above,
The predetermined word is a word related to food ingredients.
[6] In the information processing system according to any one of [2] to [5] above,
The processing unit executes the first mode when the user's utterance includes a word related to the function of the refrigerator, and executes the second mode when the user's utterance includes a word related to food ingredients.
[7] In the information processing system according to any one of [2] to [6] above,
In the first mode, the processing unit terminates the activation state when the first waiting time elapses while waiting for the user to speak, and in the second mode, the processing unit terminates the activation state after the first waiting time elapses while waiting for the user to speak. The activated state continues until the second waiting time, which is also long, has elapsed.
[8] In the information processing system according to any one of [2] to [6] above,
In the first mode, the processing unit executes a process according to the content of the user's utterance, waits for a third waiting time, and then moves to the next process, and in the second mode, the processing unit executes the process according to the content of the user's utterance. After executing the process according to the above, the process waits for a fourth standby time that is longer than the third standby time, and then moves to the next process.
[9] In the information processing system according to any one of [2] to [8] above,
When the second mode is executed, the processing unit outputs a voice asking whether there are any additional instructions regarding food management after the user's voice stops, and if the user's response to the voice output satisfies a predetermined condition. If the conditions are satisfied, the startup state is terminated.
[10] In the information processing system according to any one of [1] to [9] above,
The food inventory status managed by the voice interaction function can also be changed by inputting information on the display screen displayed on a predetermined device,
When the food stock status is changed by an input operation on the contents of the display screen, the food stock stock status is managed by the audio dialogue function based on the changed stock status.
[11] In the information processing system described in [10] above,
The management of the possession status of the ingredients includes at least information on the name of the ingredients and the quantity of the ingredients,
In the voice interaction function, when the name of the food ingredient is accepted based on the user's utterance, but the quantity of the food ingredient is not included in the utterance content, it is determined that a predetermined quantity based on a predetermined rule has been registered. We handle and accept registrations for the foodstuffs.
[12] In the information processing system described in [10] above,
The management of the possession status of the ingredients includes at least information on the name of the ingredients and the quantity of the ingredients,
In the voice interaction function, if the name of an ingredient is accepted based on the user's utterance, but the quantity of the ingredient is not included in the utterance, a voice asking for the quantity of the ingredient is output, and the user's If information indicating the quantity of the food material cannot be obtained based on the utterance, registration of the food material is not accepted.
[13] In the information processing system according to any one of [1] to [12] above,
The processing unit is configured to cause the first food ingredient and the first storage location to be uttered in a relationship, and the second food ingredient and the second storage location to be uttered in a relationship by the end of one reception of the voice interaction function. If so, the first food material and the first storage location are processed in association with each other, and the second food material and the second storage location are processed in association with each other.
[14] In the information processing system according to any one of [1] to [13] above,
If utterances regarding a plurality of foodstuffs are made before or after the utterance regarding the plurality of ingredients is made before or after one reception of the voice interaction function is completed, the processing unit may be configured to The food material and the one storage location are processed in association with each other.
[15] In the information processing system according to any one of [1] to [14] above,
When notifying the user of the inventory and the number of ingredients in inventory is equal to or greater than a first predetermined number, the processing unit outputs by voice how many ingredients are in inventory, and then The names of a second predetermined number of foodstuffs, which is smaller than the predetermined number, are outputted in a voice, and then a voice is outputted asking whether the names of the remaining ingredients should also be read out.
[16] In the information processing system according to any one of [1] to [15] above,
When deleting the registration of foodstuffs in stock, and when there are multiple foodstuffs with the same name, the processing unit selects the registration to be deleted based on the purchase date, expiration date, or expiration date.
[17] In the information processing system according to any one of [1] to [16] above,
The processing unit outputs a voice prompting to add the missing ingredient to a predetermined list when there is no inventory or there is an insufficient ingredient whose inventory is less than a predetermined amount.

1... Information processing system (foodstuff management system), 100... Refrigerator, 200... Smart speaker, 250... Information processing section, 400... Terminal equipment, 440... Information processing section (processing section), 500... Server system, 520... Information processing Department (processing department).

Claims (12)

A processing unit that can register multiple types of ingredients by the end of one reception of the voice dialogue function,
Equipped with
The food inventory status managed by the voice interaction function can also be changed by inputting information on the display screen displayed on a predetermined device,
When the holding status of the food ingredients is changed by an input operation on the content of the display screen, the holding status of the food ingredients is managed by the voice interaction function based on the changed holding status,
The management of the possession status of the ingredients includes managing information on at least the name of the ingredients and the quantity of the ingredients,
In the voice dialogue function, if the name of the food ingredient is accepted based on the user's utterance, but the quantity of the food ingredient is not included in the utterance content, it is assumed that a predetermined quantity based on predetermined rules has been specified. Handle and accept registration of the food concerned,
Information processing system. If utterances regarding a plurality of foodstuffs are made before or after the utterance regarding the plurality of ingredients is made before or after one reception of the voice interaction function is completed, the processing unit may be configured to processing the food by associating it with the one storage location;
The information processing system according to claim 1. When deleting the registration of food ingredients in stock, the processing unit selects the registration to be deleted based on the purchase date, best-before date, or expiration date if there are multiple foodstuffs with the same name.
The information processing system according to claim 1 or claim 2. The processing unit is configured to cause the first food ingredient and the first storage location to be uttered in a relationship, and the second food ingredient and the second storage location to be uttered in a relationship by the end of one reception of the voice interaction function. If so, the first food material and the first storage location are processed in association with each other, and the second food material and the second storage location are processed in association with each other.
The information processing system according to any one of claims 1 to 3. When notifying the user of the inventory and the number of ingredients in inventory is equal to or greater than a first predetermined number, the processing unit outputs by voice how many ingredients are in inventory, and then Outputting the names of a second predetermined number of ingredients smaller than the predetermined number in a voice, and then outputting a voice asking whether to read out the names of the remaining ingredients.
The information processing system according to any one of claims 1 to 4. A processing unit that can register multiple types of ingredients by the end of one reception of the voice dialogue function,
Equipped with
The food inventory status managed by the voice interaction function can also be changed by inputting information on the display screen displayed on a predetermined device,
When the holding status of the food ingredients is changed by an input operation on the content of the display screen, the holding status of the food ingredients is managed by the voice interaction function based on the changed holding status,
The management of the possession status of the ingredients includes managing information on at least the name of the ingredients and the quantity of the ingredients,
In the voice interaction function, if the name of an ingredient is accepted based on the user's utterance, but the quantity of the ingredient is not included in the utterance, a voice asking for the quantity of the ingredient is output, and then the user If information indicating the quantity of the ingredient cannot be obtained based on the utterance of
Information processing system that does not accept registrations for these ingredients. A processing unit that can register multiple types of ingredients by the end of one reception of the voice dialogue function,
Equipped with
If utterances regarding a plurality of foodstuffs are made before or after the utterance regarding the plurality of ingredients is made before or after one reception of the voice interaction function is completed, the processing unit may be configured to processing the food by associating it with the one storage location;
Information processing system. A processing unit that can register multiple types of ingredients by the end of one reception of the voice dialogue function,
Equipped with
When deleting the registration of food ingredients in stock, the processing unit selects the registration to be deleted based on the purchase date, best-before date, or expiration date if there are multiple foodstuffs with the same name.
Information processing system. A processing unit that can register multiple types of ingredients by the end of one reception of the voice dialogue function,
Equipped with
The processing unit is configured to cause the first food ingredient and the first storage location to be uttered in a relationship, and the second food ingredient and the second storage location to be uttered in a relationship by the end of one reception of the voice interaction function. If so, the first food material and the first storage location are processed in association with each other, and the second food material and the second storage location are processed in association with each other.
Information processing system. A processing unit that can register multiple types of ingredients by the end of one reception of the voice dialogue function,
Equipped with
When notifying the user of the inventory, when the number of ingredients in inventory is equal to or greater than a first predetermined number, the processing unit outputs by voice how many ingredients are in inventory, and then outputs the number of ingredients in inventory, and then Outputting the names of a second predetermined number of ingredients smaller than the predetermined number in a voice, and then outputting a voice asking whether to read out the names of the remaining ingredients.
Information processing system. Register multiple types of ingredients by the end of one reception of the voice dialogue function,
including;
The inventory status of ingredients managed by the voice dialogue function can also be changed by inputting information on the display screen displayed on a predetermined device.
When the holding status of the food ingredients is changed by an input operation on the content of the display screen, the holding status of the food ingredients is managed by the voice interaction function based on the changed holding status,
The management of the possession status of the ingredients includes managing information on at least the name of the ingredients and the quantity of the ingredients,
In the voice dialogue function, if the name of the food ingredient is accepted based on the user's utterance, but the quantity of the food ingredient is not included in the utterance content, it is assumed that a predetermined quantity based on predetermined rules has been specified. Handle and accept registration of said food ingredients,
Information processing method. The computer is
Register multiple types of ingredients by the end of one reception of the voice dialogue function,
including;
The food inventory status managed by the voice interaction function can also be changed by inputting information on the display screen displayed on a predetermined device,
When the holding status of the food ingredients is changed by an input operation on the content of the display screen, the holding status of the food ingredients is managed by the voice interaction function based on the changed holding status,
The management of the possession status of the ingredients includes managing information on at least the name of the ingredients and the quantity of the ingredients,
In the voice interaction function, if the name of an ingredient is accepted based on the user's utterance, but the quantity of the ingredient is not included in the utterance, a voice asking for the quantity of the ingredient is output, and then the user If information indicating the quantity of the food cannot be obtained based on the utterance of the food, the registration of the food will not be accepted.
Information processing method.

Images