CN113251737A - Information processing system - Google Patents

Abstract

The invention provides an information processing system, which can improve the use convenience. The information processing system of the embodiment includes a processing unit. The processing unit can register a plurality of types of food materials before 1-time acceptance of the voice conversation function is completed.

Description

Information processing system

Technical Field

Embodiments of the present invention relate to an information processing system.

Background

An information processing system for managing food stored in a refrigerator has been proposed. Such an information processing system is expected to improve usability.

Documents of the prior art

Patent document

Patent document 1: japanese patent laid-open publication No. 2017-204770

Disclosure of Invention

An object of the present invention is to provide an information processing system capable of improving usability.

The information processing system of the embodiment has a processing unit. The processing unit can register a plurality of types of food materials before 1-time acceptance of the voice conversation function is completed.

Effects of the invention

According to the food material management system and the food material management method, the use convenience of the food material management system can be improved.

Drawings

Fig. 1 is a diagram showing an overall configuration of an information processing system according to an embodiment.

Fig. 2 is a front view of a refrigerator according to an embodiment.

Fig. 3 is a block diagram showing a functional configuration of the smart speaker according to the embodiment.

Fig. 4 is a block diagram showing a functional configuration of a terminal device according to the embodiment.

Fig. 5 is a block diagram showing a functional configuration of the server system according to the embodiment.

Fig. 6 is a diagram showing an example of the contents of the inventory list DB according to the embodiment.

Fig. 7 is a diagram showing an example of contents of the shopping list DB according to the embodiment.

Fig. 8 is a diagram showing an example of contents of the trash DB according to the embodiment.

Fig. 9 is a diagram showing an example of the contents of the candidate list DB according to the embodiment.

Fig. 10 is a diagram showing an example of contents of the food material DB according to the embodiment.

Fig. 11 is a diagram showing an image displayed on the display screen in the embodiment.

Fig. 12 is a diagram showing an image of a home page for managing food materials according to the embodiment.

Fig. 13 is a diagram showing an image of a modification of the home page of the food management according to the embodiment.

Fig. 14 is a diagram showing an image in which all food material lists are displayed according to the embodiment.

Fig. 15 is a diagram showing an image in which a food material list for each storage room is displayed according to the embodiment.

Fig. 16 is a diagram showing an image of a modification example of the food material list display according to the embodiment.

Fig. 17 is a diagram showing an image of a case where a food is newly registered in a food list according to the embodiment.

Fig. 18 is a diagram showing an image of a modification example of newly registering a food material in a food material list according to the embodiment.

Fig. 19 is a diagram showing an image in a case where the food material list is changed according to the embodiment.

Fig. 20 is a diagram showing an image including a trash box list according to the embodiment.

Fig. 21 is a diagram showing an image including a shopping list and a candidate list according to the embodiment.

Fig. 22 is a diagram showing another image including a shopping list according to the embodiment.

Fig. 23 is a flowchart showing a flow of processing related to the voice dialogue function according to the embodiment.

Fig. 24 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 25 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 26 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 27 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 28 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 29 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 30 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 31 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 32 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 33 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 34 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 35 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 36 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 37 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Fig. 38 is a diagram showing a modification of an image displayed on the display screen in the embodiment.

Description of reference numerals

1 … … information processing system (food material management system), 100 … … refrigerator, 200 … … smart speaker, 250 … … information processing section, 400 … … terminal device, 440 … … information processing section (processing section), 500 … … server system, 520 … … information processing section (processing section).

Detailed Description

Hereinafter, an information processing system according to an embodiment will be described with reference to the drawings. In the following description, the same reference numerals are given to the components having the same or similar functions. A repetitive description of these configurations may be omitted. "based on XX" means "based on at least XX", and may include cases based on other elements in addition to XX. "based on XX" is not limited to the case of directly using XX, and may include a case based on the result of operation or processing of XX. "XX or YY" is not limited to one of XX and YY, and may include both XX and YY. "XX" and "YY" are arbitrary elements (for example, arbitrary information). Hereinafter, the "database" is referred to as "DB". Although the following describes a case where the voice conversation function is realized, the speech from the device side is not essential. Therefore, "voice conversation" in the following description may be replaced with "voice recognition". The "bonus term" and the "consumption term" described below are examples of the "intake term" or the "1 st term", respectively.

< 1. integral constitution >

Fig. 1 is a diagram showing an overall configuration of an information processing system 1 according to an embodiment. The information processing system 1 is an example of a "material management system" and is a system for managing the stock amount of materials stocked by a user. The information processing system 1 includes, for example, a refrigerator 100, a smart speaker 200, a router 300 in a home, a terminal device 400, and a server system 500. The network NW to be described later includes, for example, 1 or more of the internet, a cellular network, a Wi-Fi network, a wan (wide Area network), a lan (local Area network), a public line, a telephone line, a radio base station, and the like. The router 300 is, for example, a wireless router, but is not limited thereto.

The refrigerator 100 is disposed at a home of a user. The refrigerator 100 is connected to a network NW via a router 300, for example, and can communicate with the server system 500. In the present embodiment, refrigerator 100 includes a display device 130 described later. In this case, refrigerator 100 corresponds to an example of "information display device". The refrigerator 100 may also include a microphone and a speaker, and have the same voice conversation function as the smart speaker 200. In this case, the refrigerator 100 corresponds to an example of a "voice dialogue device (or a voice recognition device)".

The smart speaker 200 is disposed at the home of the user. The smart speaker 200 is connected to a network NW via, for example, a router 300, and can communicate with the server system 500. The smart speaker 200 may also be able to communicate with the refrigerator 100 via the router 300, for example. The smart speaker 200 is an example of a "voice conversation device (or voice recognition device)".

The terminal device 400 is, for example, a mobile terminal device held by a user, and is a smartphone or a tablet terminal device. The terminal device 400 is connected to a network NW and can communicate with the server system 500. An application (hereinafter, sometimes referred to as "APP") for managing food materials and remotely operating the refrigerator 100 is installed in the terminal device 400. The terminal device 400 is an example of an "information display apparatus". However, the terminal apparatus 400 is not limited to the above example, and may be a personal computer, a television set having an input function, or the like. The terminal device 400 may have a microphone and a speaker, and may have the same voice conversation function as the smart speaker 200. In this case, the terminal device 400 corresponds to an example of a "voice dialogue device (or a voice recognition device)".

The server system 500 is a server system in which various data for managing food materials are recorded in advance, and is configured by a plurality of server apparatuses 501 and 502, for example. The server devices 501 and 502 are cloud servers connected to the network NW. For example, the server system 500 includes a 1 st server device 501 for controlling the voice conversation of the smart speaker 200 and a 2 nd server device 502 for controlling the appliance of the refrigerator 100. However, the server system 500 may be configured by 1 server apparatus, or may be configured by 3 or more server apparatuses. The server system 500 may include an information processing unit that performs edge calculation or fog calculation, such as an information processing unit included in the router 300 and/or an information processing unit included in a router in the network NW. The various functional units included in the server system 500, which will be described later, may be present separately in 2 or more devices.

Hereinafter, the refrigerator 100, the smart speaker 200, the terminal device 400, and the server system 500 will be described in detail.

< 2. refrigerator >

Fig. 2 is a front view showing the refrigerator 100. The refrigerator 100 includes, for example, a casing 110, a plurality of doors 130, a display device 140, an actuator 150, and a control section 160. A plurality of storage compartments (holding compartments) 120 are provided inside the casing 110. The plurality of storage compartments 120 include, for example, a refrigerating compartment 121, a freezing compartment 122, a vegetable compartment 123, an ice-making compartment 124, a small freezing compartment 125, and a main freezing compartment 126. In the present embodiment, refrigerating room 121 is disposed at the uppermost portion, vegetable room 123 is disposed below refrigerating room 121, ice making room 124 and small freezing room 125 are disposed below vegetable room 123, and main freezing room 126 is disposed below ice making room 124 and small freezing room 125. However, the configuration of the storage chamber 120 is not limited to the above example. The plurality of storage chambers 120 are closable by a plurality of doors 130.

The display device 140 is a liquid crystal display, an organic el (electro luminescence) display, or the like, and includes a display screen 141 capable of displaying images and/or videos. The display device 140 includes a touch sensor 142 superimposed on the display screen 141. The actuator 150 includes a compressor for compressing a cooling refrigerant and sending the compressed cooling refrigerant to the cooler, a blower for sending the cooled air cooled by the cooler to each storage chamber 120, and the like.

The control unit 160 controls the display device 140 and the actuator 150. For example, the control unit 160 can receive an operation of the display screen 141 by the user via the touch sensor 142. By controlling the actuator 150, the control unit 160 can perform functions of the refrigerator 100 such as changing the set temperature zones of the storage compartments 120 of the refrigerator 100, changing the control mode of the fresh-ice compartment 122, "quick freezing" for quickly cooling the freezers 126 and 126 as compared with the normal operation, and "quick ice making" for making ice faster than the normal operation.

However, the configuration of the refrigerator 100 is not limited to the above example. The refrigerator 100 may also have a camera 170 provided inside the case 110 or the door 130. In this case, the image of the food material captured by the camera 170 may be transmitted to the server system 500 and used for food material management. This will be left to the following.

< 3. Intelligent speaker >

Fig. 3 is a block diagram showing a functional configuration of the smart speaker 200. The smart speaker 200 includes, for example, a microphone 210, a speaker 220, a wireless communication module 230, a storage unit 240, and an information processing unit 250.

Microphone 210 takes the voice of the user speaking into smart speaker 200. The speaker 220 outputs voice toward the outside of the smart speaker 200. The wireless communication module 230 includes a high-frequency circuit, an antenna, and the like, and is capable of wireless communication with the router 300, for example.

The storage unit 240 stores a voice DB 241. The voice DB241 includes, for example, a simplified voice recognition DB241 a. A trigger word as a voice command for activating the smart speaker 200 is registered in the simplified speech recognition DB241 a. The "activation" means a change from a rest state (a state in which no reaction is made to the utterance of a user other than the trigger word) in which the power consumption is suppressed to a predetermined level or less to a standby state (a state in which a voice command can be received) in which a reaction (conversation) is made to the utterance of a user other than the trigger word. The "standby state" may also be referred to as the "startup state". The trigger word is a predetermined specific word or phrase "Hello, good quality".

The information processing unit 250 includes, for example, a voice receiving unit 251, a start determining unit 252, a data transmitting/receiving unit 253, and a voice output unit 254. The voice receiving unit 251 receives voice (speech of the user) input to the microphone 210 as voice data. The start determination unit 252 determines whether or not the voice dialogue function needs to be started. When the trigger word registered in the simple speech recognition DB241a is included in the speech data received by the speech receiving unit 251, the activation determination unit 252 determines that activation is necessary and activates the speech dialogue function.

The data transceiver 253 controls the wireless communication module 230 to transmit and receive control signals and data to and from the server system 500. The data transceiver 253 transmits the voice data other than the trigger word acquired by the voice receiver 251 to the server system 500. The data transceiver 253 receives the voice data for the response prepared by the server system 500 from the server system 500. The data transceiver 253 outputs the received voice data for a response to the voice output unit 254. Voice output unit 254 outputs the voice data received from data transmission/reception unit 253 to the outside as voice (speech of smart speaker 200) via speaker 220.

< 4. terminal device

Fig. 4 is a block diagram showing a functional configuration of the terminal device 400. The terminal device 400 includes, for example, a display device 410, a wireless communication module 420, a storage unit 430, and an information processing unit 440. The display device 410 is a liquid crystal display, an organic EL display, or the like, and includes a display screen 411 capable of displaying images and/or videos. The display device 410 includes a touch sensor 412 superimposed on the display screen 411. The wireless communication module 420 includes a high-frequency circuit, an antenna, and the like, and can be connected to the network NW.

The storage unit 430 stores application programs for managing food materials, remotely operating the refrigerator 100, and the like. All or a part of the information Processing unit 440 is realized by executing the application program by a hardware processor such as a cpu (central Processing unit) mounted in the terminal device 400.

The information processing unit 440 includes, for example, an operation receiving unit 441, a display device control unit 442, and a data transmitting/receiving unit 443. The information processing unit 440 is an example of a "processing unit".

The operation receiving unit 441 receives a user operation on the display screen 411 via the touch sensor 412. The data transmitting/receiving portion 443 controls the wireless communication module 420 to transmit and receive control signals and data to and from the server system 500. The data transmission/reception unit 443 transmits the information indicating the operation content of the user received by the operation reception unit 441 to the server system 500. The data transceiver 443 outputs a display-related control instruction and display data received from the server system 500 to the display device controller 442. The display device control unit 442 displays an image or video included in the display data received from the server system 500 on the display screen 411 based on the control instruction received from the server system 500.

However, the configuration of the terminal device 400 is not limited to the above example. For example, the same function as the smart speaker 200 described above as a voice conversation device may be provided. Furthermore, the terminal device 400 may also have a camera 450. In this case, the image of the food material captured by the camera 450 may be transmitted to the server system 500 and used for food material management. This will be left to the following.

< 5. Server System >

Fig. 5 is a block diagram showing a functional configuration of the server system 500. An example in which the server system 500 controls the terminal device 400 as the "information display apparatus" and the smart speaker 200 as the "voice conversation apparatus" will be described below. However, the "information display apparatus" controlled by the server system 500 may be the refrigerator 100. In this case, "terminal device 400" in the following description is replaced with "refrigerator 100". The "voice conversation apparatus" controlled by the server system 500 may also be the terminal device 400 or the refrigerator 100. In this case, "smart speaker 200" in the following description is replaced with "terminal device 400" or "refrigerator 100".

The server system 500 includes a communication module 510, an information processing unit 520, and a storage unit 530. For convenience, the functional units related to the voice conversation will be described first, and then the functional units related to the voice conversation will be described. The communication module 510 includes a communication circuit and the like, and can be connected to a network NW.

< 5.1 construction of information processing section >

The information processing unit 520 includes, for example, an instruction receiving unit 521, a registered information management unit 522, a display control unit 523, a voice processing unit 524, a control mode changing unit 525, a control instruction generating unit 526, and a response generating unit 527. However, a part or all of these functional units may be provided in the terminal device 400 or the refrigerator 100 instead of the server system 500. The information processing unit 520 is an example of a "processing unit".

The instruction receiving unit 521 receives information indicating the user operation (the user operation on the display screen 411) received by the operation receiving unit 441 of the terminal apparatus 400 from the terminal apparatus 400 via the communication module 510. The instruction receiving unit 521 outputs the information received from the terminal device 400 to the registration information management unit 522.

The registration information management unit 522 updates various DBs (the stock list DB601, the shopping list DB602, the candidate list DB604, the trash box DB603, the stock DB607, the user setting DB606, the remote operation registration DB609, and the like) described later, based on the information received from the instruction receiving unit 521. In addition, in the present embodiment, the registration information management unit 522 receives, from the voice processing unit 524, information indicating the content (instruction made by the user by speaking) analyzed by the voice processing unit 524 described later. The registration information management unit 522 updates the various DBs based on the information received from the voice processing unit 524. The registration information management unit 522 is described in detail later. In the following description, the term "user operation" or "user input" means a user operation (operation of the terminal device 400 by the user) received by the instruction receiving unit 521 or a user operation (operation performed by speaking to the smart speaker 200) analyzed by the voice processing unit 524 described later.

The display control unit 523 generates display data and control signals including images or videos displayed on the display screen 411 of the terminal device 400 based on the contents of the various DBs. The display controller 523 transmits the generated display data and control signal to the terminal device 400 via the communication module 510. The control signal generated by the display control unit 523 is a control signal for operating the display device control unit 442 of the terminal device 400. Here, the phrase "display on the display screen" is not limited to the meaning that the image or video is displayed on the display screen 411 by the display control unit 442 of the terminal device 400, and may include a case where the image or video is displayed on the display screen 411 by externally supplying display data and/or a control signal to the terminal device 400.

The voice processing unit 524 receives the voice data acquired by the smart speaker 200 from the smart speaker 200 via the communication module 510. For example, the voice processing unit 524 receives voice data other than the trigger word among the voice data acquired by the smart speaker 200. The voice processing unit 524 analyzes the voice data received from the smart speaker 200. For example, the voice processing unit 524 performs voice recognition processing to convert voice data into text, and performs natural language processing on the converted voice data to interpret the meaning. The voice processing unit 524 outputs the analysis result of the voice data to the control pattern changing unit 525, the control instruction generating unit 526, and the response generating unit 527.

The control mode changing unit 525 changes the operation state of the smart speaker 200 based on the analysis result of the voice data obtained by the voice processing unit 524. The control mode changing unit 525 will be described in detail later.

Based on the analysis result of the voice data obtained by the voice processing unit 524, the control instruction generating unit 526 receives an instruction to the refrigerator 100 (for example, an instruction to change the set temperature zone of each storage room 120, change the control mode of the fresh food room 122, to perform "quick freezing" or "quick ice making") when the user speaks that the instruction is included. Control instruction generating unit 526 generates a control instruction corresponding to the received instruction for refrigerator 100. Control instruction generating unit 526 transmits the generated control instruction to refrigerator 100 via communication module 510. Control unit 160 of refrigerator 100 drives actuator 150 based on the control instruction generated by control instruction generating unit 526, thereby realizing the control desired by the user.

When there is a content to be answered by the utterance of the user analyzed by the voice processing unit 524 or when there is a content to be reported to the user, the response generation unit 527 refers to the voice DB610 (described later) and generates voice data to be output from the smart speaker 200. The response generation section 527 transmits the generated voice data to the smart speaker 200 via the communication module 510. The smart speaker 200 responds or reports to the user with voice by driving the speaker 220 based on the voice data generated by the response generation unit 527.

< 5.2 various DB >

The storage unit 530 stores an inventory list DB601, a shopping list DB602, a trash box DB603, a candidate list DB604, a food material DB605, a user setting DB606, a stock DB607, a history DB608, a remote operation registration DB609, and a voice DB 610. The inventory list DB601, shopping list DB602, trash box DB603, candidate list DB604, user setting DB606, stock DB607, and history DB608 are prepared for each user.

Although the above-described examples of the plurality of DBs are shown as the DB for managing the food materials for convenience of explanation, the DBs may be integrally provided and managed by assigning identification information (for example, a tag) indicating that the identification information is displayed in a specific list (shopping list or candidate list) described later to register each food material. The process of registering, deleting, and confirming various DBs (for example, registering, deleting, and confirming food materials) described below is executed by the registration information manager 522.

< 5.2.1 inventory List DB >

Fig. 6 is a diagram showing an example of the contents of the inventory list DB 601. The "stock list" is a list for managing food materials stocked by the user. The "stock list" exemplifies "list 1". In the following description, the "stock list" may be referred to as a "food material list". The inventory list DB601 registers ingredients that exist as an inventory in the home of the user. In the stock list DB601, not only the food stored inside the refrigerator 100 but also food stored outside the refrigerator 100 (for example, stored at normal temperature) is registered. The stock list DB601 is an example of "stock information" indicating the stock amount of the food material stocked by the user.

In the inventory list DB601, "management ID", "food name", "food classification", "storage place", "purchase date", "expiration date" (or "expiration date"), "number", "unit", and "history" are registered in association with each food. The term "for each food material" means, for example, the same food material purchased at the same time. In other words, even if the food materials are of the same type, the food materials purchased on a certain time and day and the food materials purchased on other time and day are registered as different food materials. Here, the "purchase date" means a date on which a specific food is newly registered, that is, a date on which the stock amount is increased, and is used to include a case where the food is not purchased by the approach.

By "food material classification" is meant the differentiation of food materials. In the present embodiment, 16 food material categories are set as the "food material category" as follows: "vegetables", "fruits", "meats", "fish and marine algae", "egg and dairy products", "bean and bean products", "processed foods", "rice and bread", "noodles", "drinks", "frozen foods", "desserts", "seasonings", "spices", "side meals", and "others". However, the above-described division is merely an example, and the division of the food material classification can be set as appropriate.

The "storage location" means a location where food materials are stored. In the present embodiment, the following 4 types of storage locations are set as the "storage locations: "refrigerating chamber", "vegetable chamber", "freezing chamber", and "others". The "refrigerating chamber" corresponds to a region where the refrigerating chamber 121 and the fresh ice chamber 122 are combined. The "vegetable room" corresponds to vegetable room 123. The "freezer compartment" corresponds to a region where the small freezer compartment 125 and the main freezer compartment 126 are combined. The "others" correspond to the storage places outside the refrigerator 100.

However, the above-described division is an example, and the division of the storage location can be set as appropriate. For example, refrigerating compartment 121 and fresh-ice compartment 122 may be separated into different storage locations. The small freezer compartment 125 and the main freezer compartment 126 may be divided into different storage locations. Further, the storage locations may be classified into more detailed regions such as "the uppermost layer of the refrigerating compartment" and "the right door pocket of the refrigerating compartment". The "storage location" may also be referred to as a "storage room" or a "storage area". Hereinafter, for convenience of explanation, the concept of the freezing compartment 122 may be included in the case of the "refrigerating compartment 121", and the concept of the small freezing compartment 125 may be included in the case of the freezing compartment 126.

The "number" is the latest number of food materials registered. In the "history", when the food material is reduced, information indicating the number of times and days of the reduction is registered.

The registration information management unit 522 performs registration, deletion, and confirmation of the food material with respect to the inventory list DB601, for example, based on an operation by the user. In the present embodiment, the registration information management unit 522 notifies the terminal device 400 that there is a food material approaching the reward period (or consumption period) a predetermined number of days before (for example, 1 day before) the reward period (or consumption period) of the food material registered in the inventory list DB 601. The notification is displayed as a push notification on the display screen 411 of the terminal apparatus 400. When the appreciation period (or consumption period) of the food registered in the inventory list DB601 has elapsed, the registration information management unit 522 automatically deletes the food having the appreciation period (or consumption period) elapsed from the inventory list DB601, and moves the registration of the food to the trash DB603 and the candidate list DB604, which will be described later. The "stock list" is displayed as a "food material list" on the display screen 411 of the terminal device 400 under the control of the server system 500. The details of the display example of the "stock list (food material list)" will be described later.

< 5.2.2 shopping List DB >)

Fig. 7 is a diagram showing an example of the contents of the shopping list DB 602. The "shopping list" is a list that can be referred to by the user to prevent forgetting to purchase the food item when the user purchases the food item, and is a list of food items for which the purchase plan (e.g., the user's intention to purchase the food item) of the user is confirmed. The "shopping list" is an example of the "3 rd list" and is a list having a higher possibility of purchase than a candidate list described later. However, the "shopping list" is not limited to the materials registered in accordance with the user's explicit purchase intention, and may include materials registered under estimation based on the user's stock intention (an intention to store a certain material as a stock). The ingredients registered in the "shopping list" are once registered in a "candidate list" described later, for example, and then moved from the "candidate list" to the "shopping list" based on the meaning of the user (operation by the user), and registered in the "shopping list". However, the food materials registered in the "shopping list" may be directly registered in the "shopping list" based on the operation of the user without going through the candidate list. The "shopping list" is displayed on the display screen 411 of the terminal device 400 under the control of the server system 500. Details of the display example of the "shopping list" will be described later.

The food materials displayed as a "shopping list" are registered in the shopping list DB 602. In the shopping list DB602, "management ID", "food name", "food classification", "number", and "unit" are registered in association with each food. The registration information management unit 522 performs registration, deletion, and confirmation of ingredients with respect to the shopping list DB602 based on an operation by the user.

< 5.2.3 garbage can DB >

Fig. 8 is a diagram showing an example of the contents of the trash DB 603. The trash DB603 temporarily stores the ingredients deleted from the stock list DB 601. For example, a food material that is originally registered in the inventory list DB601 and consumed and is deleted from the inventory list DB by a user operation is registered in the trash DB 603. In the present embodiment, among the food materials originally registered in the inventory list DB, food materials that have been automatically deleted from the inventory list DB by the registration management unit 522 because the appreciation period (or consumption period) has passed are also registered in the trash box DB 603. The food registered in the trash DB603 can be registered again in the inventory list DB by the user's operation. The food registered in the trash DB603 is automatically deleted from the trash DB603 after a predetermined period (for example, 3 days) has elapsed since the registration date registered in the trash DB 603. Even if a material whose appreciation period (or consumption period) has passed is registered in the stock list DB601, the material may be deleted by waiting for a user operation without being automatically deleted from the stock list DB 601. Similarly, the food registered in the trash DB603 may be deleted by waiting for a user operation without setting a period until the automatic deletion. In the trash DB603, "management ID", "food name", "food classification", and "registration date to the trash DB 603" are registered in association with each food.

< 5.2.4 candidate list DB >)

Fig. 9 is a diagram showing an example of the contents of the candidate list DB 604. The "candidate list" is a list that the user can assist in referring to prevent forgetting to purchase, and is a list of candidate food materials for purchase whose purchase plan (e.g., explicit purchase intention) has not been confirmed by the user. The "candidate list" exemplifies "list 2". The "candidate list" is an example of information that urges the user to purchase food materials based on the decrease in the stock of food materials. The "candidate list" is purchase proposal information, and is generated by, for example, a deletion operation of the food registered in the inventory list DB 601. The "candidate list" is displayed on the display screen 411 of the terminal apparatus 400 under the control of the server system 500. Details of the display example of the "candidate list" will be described later. The food materials displayed as the "candidate list" are registered in the candidate list DB 604. In the candidate list DB604, "management ID", "food name", "food classification", and "registration date to the candidate list DB 604" are registered in association with each food. When the number of specific food materials registered in the inventory list DB601 is reduced to a predetermined number before the number of the food materials is reduced to zero as described below, the number of the food materials (the number of inventory) may be registered in the candidate list DB604 so as to be associated with the food materials.

In the present embodiment, when the number of the materials registered in the inventory list DB601 is reduced and a predetermined condition is satisfied, the reduced materials are automatically registered in the candidate list DB 604. The "predetermined condition" is, for example, that the number of the reduced food materials is less than a predetermined threshold value. In the present embodiment, the "predetermined condition" is that the number of the reduced food materials becomes zero (that is, the stock of the food materials is not available). The "predetermined condition" is an example of the "1 st condition".

For example, when the originally registered food is deleted in the inventory list DB601 and registered in the trash DB603, the same food as the food registered in the trash DB603 is automatically registered in the candidate list DB 604. The "deletion of registration" means that the stock of the specific food material is lost (changed from an existing state to an absent state). That is, in the candidate list DB604, there are automatically registered: the food material that is consumed by the originally registered food material in the inventory list DB601 and deleted from the inventory list DB by the user's operation, and the food material that is automatically deleted from the inventory list DB by the registration information management section 522 because the appreciation period (or consumption period) has passed among the originally registered food materials in the inventory list DB. Both the shopping list DB602 and the candidate list DB604 register ingredients with stock that is missing (or reduced). The shopping list DB602 and the candidate list DB604 are updated at different timings from each other. When the user performs a deletion operation for registering the inventory list DB601 for a certain food material in accordance with the fact that the food material is eaten or thrown away, the server system 500 according to the present embodiment does not need to display the food material, but can be effectively used as data for recommending shopping.

The registration information management unit 522 can accept an operation by a user to move the food registered in the candidate list DB604 to the shopping list DB 602. For example, the user can select a material to be moved by clicking (clicking) on the display of the material in the candidate list described later, and move the selected material to the shopping list. When the selected material is registered in the shopping list DB602, the registration of the material is deleted from the candidate list DB604 as the movement source.

Here, in a state where a certain food material is already registered in the candidate list DB604, the same food material may be directly registered in the shopping list DB602 by a user operation (that is, in a case where the food material is registered in the shopping list DB602 without going through the candidate list DB 604). In this case, the registration information management unit 522 registers the food material in the shopping list DB602 and deletes the registration of the food material from the candidate list DB 604. That is, with respect to the ingredients already registered in the shopping list DB602, even when the number of stocks in the stock list DB601 satisfies the above-described "predetermined condition", the ingredients are not registered in the candidate list DB 604.

When the number of the ingredients registered in the inventory list DB601 is reduced and the predetermined condition is satisfied and when the ingredient is a special ingredient (for example, an ingredient registered in the stock DB 607) registered in advance by the user, the registration information management unit 522 may directly register the ingredient in the shopping list DB602 without registering the ingredient in the candidate list DB 604.

In the present embodiment, the maximum number (for example, 20) of food materials that can be registered in the candidate list DB604 is determined. When a predetermined amount (for example, the maximum number) of ingredients has already been registered in the candidate list DB604, the registration information management unit 522 deletes the old registration of the candidate list DB604 in sequence in accordance with the case where a new ingredient registration is added to the candidate list DB 604.

The registration information management unit 522 deletes the old registration of the candidate list DB604 in order of priority based on 1 or more of the registration date with the candidate list DB604, the purchase date of the food material, the taste expiration date of the food material, and the consumption expiration date of the food material. For example, the registration information management unit 522 deletes the old registrations of the ingredients registered in the candidate list DB604 in the order of the purchase date of the ingredients from old to new, the order of the appreciation period of the ingredients from old to new, or the order of the consumption period of the ingredients from old to new. In addition, when the number of specific food materials registered in the inventory list DB601 is reduced to a predetermined number before the number becomes zero, and the food materials are registered in the candidate list DB604 as described above, the food materials stored in the inventory list DB601 may be preferentially deleted in the order of the expiration date of the food materials from the new to the old, or in the order of the expiration date of the food materials from the new to the old.

The registration information management unit 522 deletes the old registration of the candidate list DB604 in the order of priority based on the food material classification to which the food material belongs, instead of or in addition to the above order of priority. For example, when a food material belongs to a specific food material category, the registration information management unit 522 reduces the priority of deleting the food material (or excludes the food material from the deletion target) so that the food material is not easily deleted from the candidate list DB 604. The "specific food material classification" is a food material classification having characteristics similar to those of a conventional food material, and examples thereof include seasonings and spices.

Instead of or in addition to the priority order, the registration information management unit 522 deletes the old registration of the candidate list DB604 in accordance with the past stock frequency based on the food materials and/or the priority order of whether or not the food materials are included in the food material group designated by the user. When the past stock frequency of the food material is equal to or higher than the threshold value, the registration information management unit 522 lowers the priority of deleting the food material (or excludes the food material from the deletion target) so that the food material is not easily deleted from the candidate list DB 604. The past stock frequency can be obtained based on the history DB608 described later. When a food is included in the food group designated by the user, the registration information management unit 522 lowers the priority of deletion of the food (or excludes the deletion target) so that the food is not easily deleted from the candidate list DB 604. The "designated food material group" is, for example, a food material group registered in the stock DB 607.

By providing the candidate list, it is possible to reduce forgetting to register a shopping list. Further, the candidate list functions as an auxiliary function for registering in the shopping list, and the work load of the registration work in the shopping list can be reduced.

< 5.2.5 food material DB >

Fig. 10 is a diagram showing an example of the contents of the material DB 605. The material DB605 is a default DB set for common use by a plurality of users. The material DB605 is used for automatic input of material information described later. The ingredients registered in the ingredient DB are registered with a large number of ingredients generally distributed. In the food material DB605, for each food material, "food material name", "acceptance candidate of food material name", "food material classification", "icon of food material classification", "default storage location (recommended storage location)", and "appreciation period (or consumption period)" are registered in correspondence. The "food material classification", "icon of food material classification", "default storage place", and "appreciation period (or consumption period)" are examples of the "additional information", respectively. The material DB605 is an example of "relationship information" in which material names and additional information are associated with each other, and is an example of "1 st information". However, the "1 st information" mentioned here is a concept different from the "1 st information I1" described later. In the present embodiment, the icon of the food material classification is registered as one of the additional information, but an image indicating the food material classification may be used instead. The material DB605 is information in which a general correspondence relationship between a material and a storage location of the material (a correspondence relationship not depending on each user, but a correspondence relationship of a standard set in advance as a content to be applied commonly to each user unless a user specifies it) is registered. This correspondence relationship is an example of "1 st correspondence relationship".

As the "acceptance candidates for the food material name", 1 or more acceptance candidates in which an input error of the name of the food material is considered are registered. For example, when the food material is "プチトマト" ("プチ" means mini in chinese, "トマト" means tomato in chinese, "プチトマト" is katakana expression, and the corresponding hiragana is "ぷちとまと"), in addition to "プチトマト" (katakana expression), the food material is registered with "ぷちとまと" (hiragana expression) and "トマト". As the "default storage location", a storage location inside refrigerator 100 or outside refrigerator 100 is registered, and is a storage location suitable for storing food. As described above, in the case where refrigerating room 121 has a shelf, the "storage place" may be a division indicating a part of places different from other places among specific storage rooms 120, such as the "uppermost layer of refrigerating room 121". The "acceptance candidate of a food material name" may be used only for association with the "additional information" (that is, use of the "relationship information"), and the name itself registered by the user may be used as the name of the food material indicated to the user without being converted into the "acceptance candidate of a food material name". For example, when the user inputs "o" とまと "(tomato produced in chinese, o), additional information such as the food material classification" vegetables "and the storage location" vegetable room "prepared in advance in association with" トマト "may be registered in the stock list DB601 as the case where" トマト "is registered as the" food material name acceptance candidate ", and the name of the food material indicated to the user such as" o "とまと" may be used without being processed on the display screen 411 of the terminal device 400. Alternatively, instead of the above-described method, only a part of what is considered to be expression errors may be changed (i.e., "とまと" may be changed to "トマト") as in "o" トマト "or the like, and the change may be used for display.

Here, the refrigerator 100 has different configurations of the storage chambers 120 according to models. Therefore, the storage location recommended for the food may vary depending on the model. Therefore, in the present embodiment, the material DB605 is prepared for each model of the refrigerator 100, for example. In other words, the material DB605 is different according to models of the refrigerator 100. The registration information management unit 522 specifies the model of the refrigerator 100 of the user based on the model information of the refrigerator 100 obtained by referring to the remote operation registration information DB609 described later, and selects the material DB605 corresponding to the specified model. Note that, in the case of preparing information that differs depending on the model of the refrigerator 100, the information may be limited to a part of the information indicating the storage location in the food DB605, not to the entire food DB 605.

The description is continued with reference to fig. 2. The storage unit 530 stores the user setting DB606, the stock DB607, the history DB608, the remote operation registration DB609, and the voice DB610 in addition to the above-described DBs.

The user setting DB606 is information registered by the user. In the user setting DB606, a user-specific change setting for the material DB is registered in order to provide flexibility in automatic input of material information to be described later. For example, the user setting DB606 registers a correspondence relationship unique to the user between the food and the storage location. This correspondence relationship is an example of "2 nd correspondence relationship". Specifically, in the food material DB605, "refrigerator" is registered as a default storage location for "chicken". When the user purchases chicken meat and stores the chicken meat in the freezer compartment more than in the refrigerator compartment, the user can register the "chicken meat" and the "freezer compartment" in the user setting DB606 in association with each other. The user setting DB606 is another example of "relationship information" in which the name of the food material is associated with the additional information, and is an example of "2 nd information". However, the "2 nd information" mentioned here is a concept different from the "2 nd information I2" described later. The login information management unit 522 performs login, deletion, and confirmation of the user setting DB606 based on the operation of the user.

The stock DB607 registers ingredients frequently used by the user (for example, ingredients whose stock is not desired to be empty). The stock DB607 is registered with a material (special material) which the user "announces" the desire to stock. The registration information management unit 522 performs registration, deletion, and confirmation of the stock DB607 based on an operation by the user.

The history DB608 can be used instead of the user setting DB606 or in combination with the user setting DB 606. The history DB608 has registered therein a history of the user (a history of use of the refrigerator 100) related to storage of the food material, for example, without relying on explicit registration of the user. The registration information management unit 522 can obtain information on which storage room 120 is often used for a certain food by referring to the history DB 608. The history DB608 is generated based on the result of registration of the user to the inventory list DB601 and/or the result of shooting by the camera 170 of the refrigerator 100.

In the remote operation registration DB607, "management ID", "model", and "management ID of registered terminal device for remote operation" are registered in association with each refrigerator 100. The display control unit 523 specifies the model of the refrigerator 100 of the user from the remote operation registration DB607 based on the management ID of the terminal device 400 received from the terminal device 400, and changes the pattern of the refrigerator 100 displayed on the display screen 411 of the terminal device 400 based on the specified model of the refrigerator 100. The details of this will be described later.

A dictionary for explaining the meaning of the utterance of the user, voice information for generating voice data for a sounding application, and the like are registered in voice DB 610. The storage unit 530 stores image data of icons for classifying food materials.

< 6. display of picture >

Next, screen display displayed on the display screen 411 of the terminal apparatus 400 when the terminal apparatus 400 is used to manage the ingredients of the refrigerator 100 will be described. The screen display described below is realized by the display device control unit 442 of the terminal device 400 performing control based on the display data and the control instruction transmitted from the server system 500.

< 6.1 Home Page of management Screen of refrigerator >

Fig. 11 is a diagram showing an image displayed on the display screen 411. To the left in fig. 11 is an image S1 showing a home page displayed in the case where the application is started in the terminal apparatus 400. The image S1 includes a pattern R0 schematically representing the refrigerator 100, and a plurality of displays D11 to D16. The pattern R0 includes patterns R0a to R0f of the respective storage compartments 120 of the refrigerator 100, which are schematically shown. Patterns R0a to R0f are patterns that sequentially show refrigerating compartment 121, fresh food compartment 122, vegetable compartment 123, ice-making compartment 124, small freezing compartment 125, and main freezing compartment 126, respectively. Displays D11 to D16 show the setting contents (operation states) of the storage compartments 120.

The display controller 523 displays an image S2 indicating a menu so as to overlap with the image S1 by the user pressing the display D17 indicating "setting" located at the upper right of the image S1. The display controller 523 causes the display screen 411 to display an image S3 (fig. 13) of the home page of the food material management by "food material management" included in the image S2 operated (for example, clicked) by the user.

< 6.2 Home Page for food Material management >

Fig. 12 shows an image S3 of the home page of food material management. The image S3 includes the 1 st pattern R1, the 2 nd pattern R2, the 1 st to 4 th display frames D1 to D4, and the 1 st to 4 th icons K1 to K4.

In the present embodiment, the images S3 are displayed in association with the patterns R1a to R1e schematically showing the storage rooms (storage places) 120 included in the 1 st pattern R1 of the refrigerator 100, and the 1 st information I1 related to the food stored in the storage room 120. The image S3 is an example of "display No. 1". The information included in each of the 1 st to 4 th display frames D1 to D4 is an example of the "1 st information I1". The 1 st information I1 includes information on the entire food stored in each storage room 120. For example, the 1 st information I1 is the number of food materials stored in each storage room 120.

However, the 1 st information I1 may be information indicating the filling degree (degree of use) of each storage chamber 120. The "filling degree" is information based on a calculation result obtained by dividing the number of food materials stored in each storage chamber 120 by the number of food materials when each storage chamber 120 is fully filled, and is expressed by a filling rate (usage rate) or a filling amount ("large", "normal", or small "), or the like. The filling rate of each storage chamber 120 can be estimated with high accuracy from the type and number of the ingredients registered as currently being stored, provided that the volume and size occupied by the ingredients are specified for each type of food. Instead of the volume and size of the food materials, each combination of the type and number of food materials that can be stored in each storage chamber 120 may be determined in advance. The 1 st information I1 may include the names of the food materials stored in the storage compartments 120, instead of or in addition to the number of food materials or the filling degree.

An example of the present embodiment will be described in detail below. The 1 st pattern R1 is a pattern schematically representing the refrigerator 100. The 1 st pattern R1 includes patterns R1a to R1e schematically showing each storage chamber 120 of the refrigerator 100, in addition to the outer shape of the refrigerator 100. The 1 st pattern R1 is formed of, for example, a line indicating the outer shape of the refrigerator 100 and a line indicating the boundary between the storage compartments 120. The 1 st pattern R1 includes a pattern R1a indicating a refrigerator compartment 121, a pattern R1b indicating a vegetable compartment 123, a pattern R1c indicating an ice making compartment 124, a pattern R1d indicating a small freezer compartment 125, and a pattern R1e indicating a main freezer compartment 126.

In the present embodiment, pattern 1R 1 includes 1 pattern R1a corresponding to refrigerator compartment 121 and fresh air compartment 122, instead of pattern indicating fresh air compartment 122. In the present embodiment, the boundary line between the left and right refrigerating chamber doors is omitted in the 1 st pattern R1 even in the side-by-side refrigerator 100. Thus, the 1 st pattern R1 becomes a simple pattern. However, the 1 st pattern R1 may include a pattern indicating the fresh food compartment 122, or may have a boundary line between the left and right refrigerating compartment doors 130 in the side-by-side refrigerator 100. The details of the modification of the 1 st pattern R1 will be described later. The 1 st pattern R1 does not necessarily indicate the size of each door of the actual refrigerator 100 correctly, and the size ratio may be adjusted appropriately for the sake of visibility or for easy display of information.

The display controller 523 specifies the model of the user 'S refrigerator 100 based on the remote operation registration DB609, and includes the 1 st pattern R1 of the arrangement of the storage room 120 corresponding to the model of the user' S refrigerator 100 in the image S3. In this case, if the arrangement of the doors 130 of the refrigerator 100 is the same, the display controller 523 can use the same 1 st pattern R1 even if the size ratio is slightly different.

The 2 nd pattern R2 is disposed outside the 1 st pattern R1. For example, the 2 nd pattern R2 is disposed on the side of the 1 st pattern R1. In the present embodiment, the 2 nd pattern R2 is disposed at the side of the 1 st pattern R1 at the same height as the bottom of the refrigerator included in the 1 st pattern R1. The 2 nd pattern R2 schematically shows a storage place outside the refrigerator 100. The 2 nd pattern R2 includes, for example, a pattern indicating a box-shaped storage section.

The 1 st to 4 th display frames D1 to D4 include, for example, patterns representing bubble charts. The 1 st display frame D1 includes 1 st information I1 about food materials stored in the refrigerating compartment 121 and the freezing compartment 122. For example, information indicating the number of food items stored in refrigerating compartment 121 and freezing compartment 122 is displayed in display box 1D 1. The 1 st display frame D1 is disposed on the side of the pattern R1a of the refrigerating compartment included in the 1 st pattern R1. From other points of view, the 1 st display frame D1 is disposed at the same height as the pattern R1a of the refrigerating chamber included in the 1 st pattern R1.

The 2 nd display frame D2 includes the 1 st information I1 related to the food material held in the vegetable room 123. For example, information indicating the number of food items stored in the vegetable room 123 is displayed in the 2 nd display frame D2. The 2 nd display frame D2 is disposed on the side of the pattern R1b of the vegetable room 123 included in the 1 st pattern R1. From other points of view, the 2 nd display frame D2 is disposed at the same height as the pattern R1b of the vegetable room 123 included in the 1 st pattern R1.

The 3 rd display frame D3 contains the 1 st information I1 related to the food materials stored in the freezing chamber 125, 126. For example, information indicating the number of food items stored in the freezing chambers 125 and 126 is displayed in the 3 rd display frame D3. The 3 rd display frame D3 is disposed on the side of the patterns R1D and R1e of the freezing chambers 125 and 126 included in the 1 st pattern R1. From other points of view, the 3 rd display frame D3 is disposed at the same height as the patterns R1D, R1e of the freezing compartments 125, 126 included in the 1 st pattern R1.

The 4 th display frame D4 includes 1 st information I1 related to food materials (food materials not stored in the refrigerator 100) stored outside the refrigerator 100. For example, information indicating the number of food items stored outside refrigerator 100 is displayed in 4 th display box D4. The 4 th display frame D4 is disposed directly above the 2 nd pattern R2. The 1 st information I1 included in the 4 th display frame D4 is an example of "out-of-box information".

The information displayed in the 1 st to 4 th display frames D1 to D4 is generated by the display controller 523 based on the inventory list DB601, for example. With such a configuration, each storage room 120 of the refrigerator 100 can be visually linked with the food material information, and the convenience of use of the inventory management system can be improved.

In the present embodiment, when the operation of the user for specifying the 1 st storage location (1 st storage room) included in the 1 st pattern R1 is accepted for the 1 st pattern R1, the display controller 523 displays the 2 nd information I2 (see fig. 15) more detailed than the 1 st information I1 on the display screen 411 with respect to the food stored in the 1 st storage location. The "operation by the user for specifying the 1 st storage location" is, for example, clicking a pattern indicating the 1 st storage location on the 1 st pattern R1, but is not limited to this, and may be clicking on the display frames D1 to D4 corresponding to the respective storage rooms 120, or may be a designation by a comment via the smart speaker 200.

The display controller 523 may change any of the pattern, color, and size of the storage location corresponding to the storage room 120 that is clicked in the 1 st pattern R1. With this configuration, it is easy to grasp that the click is performed.

In the present embodiment, when an operation of a user specifying the 1 st storage location included in the 1 st pattern R1 is accepted for the 1 st pattern R1, a food material list of food materials stored in the 1 st storage location is displayed on the display screen 411. In the present embodiment, when an operation of a user for specifying the 1 st storage location included in the 1 st pattern R1 is accepted for the 1 st pattern R1 in a state where the image S3 is displayed on the display screen 411, the display controller 523 switches the display of the display screen 411 to the image S5 in which the 2 nd information I2 (see fig. 14) more detailed than the 1 st information I1 is included in the food stored in the 1 st storage room. The image S5 exemplifies "display No. 2".

More specifically, when the user's operation for specifying refrigerating compartment 121 is accepted for pattern 1R 1 (when pattern R1a or 1 st display frame D1 is clicked by the user), display controller 523 displays the food item list of the food items stored in refrigerating compartment 121. Similarly, when the user's operation for specifying vegetable room 123 is accepted for pattern 1R 1 (when the user clicks on pattern R1b or 2 nd display box D2), display controller 523 displays the food item list of the food items stored in vegetable room 123. When the user operation for specifying freezing chambers 125 and 126 is accepted for the 1 st pattern R1 (when the user clicks on the patterns R1D and R1e or the 3 rd display frame D3), the display controller 523 displays the food material list of the food materials stored in the freezing chambers 125 and 126. When the user operation for specifying the external storage location is accepted for the 1 st pattern R1 (when the 2 nd pattern R2 or the 4 th display frame D4 is clicked by the user), the display controller 523 displays the food material list of the food materials stored outside the refrigerator 100. These material lists are examples of the "2 nd information I2", respectively. Details of the food material list will be left to later.

In the present embodiment, the login information manager 522 and the display controller 523 may receive different control instructions when the operation of the 1 st form of the user is received for each storage room 120 (the patterns R1a to R1e or the 1 st to 4 th display frames D1 to D4) included in the 1 st pattern R1, and when the operation of the 2 nd form of the user is received for each storage room 120 (the patterns R1a to R1e or the 1 st to 4 th display frames D1 to D4) included in the 1 st pattern R1. For example, when the operation of the 1 st form of the user is accepted for each storage room 120 included in the 1 st pattern R1, the display controller 523 displays the 2 nd information I2 related to the food stored in the selected storage room 120 by, for example, screen conversion. On the other hand, when the operation of the 2 nd form of the user is accepted for each storage room 120 included in the 1 st pattern R1, the registration information manager 522 and the display controller 523 collectively perform a predetermined process on all the food materials stored in the selected storage room 120.

The "operation of the 1 st mode" is, for example, clicking (clicking) on the display screen 411. The "operation of the 2 nd mode" is, for example, a long press, a double click, a heavy press (in the case of a pressure-sensitive display), a swipe, a slide (flick), or the like on the display screen 411. The "predetermined process" may be a process different from the process corresponding to the "operation of the 1 st form", and a common operation is preferably set for the ingredients stored in the selected storage room 120. For example, the registration of all the ingredients stored in the selected storage room 120 is collectively deleted, or a reception window for collectively deleting is opened. By providing the function of deleting the contents collectively, the contents of the refrigerator 100 can be easily sorted. Further, the function to be collectively deleted is improved in operability when it is associated with the pattern or the display frame of each storage room 120 included in the 1 st pattern R1.

From another point of view, when the operation of the 1 st form of the user is accepted for each storage room 120 included in the 1 st pattern R1, the registration information management unit 522 and the display control unit 523 open the food material management screen (registration screen/display screen) of the selected storage room 120, and even if the operation is performed for the 2 nd form of the same place, if the operation is a long press, the processing related to the collective deletion of the food material registrations in the storage place can be performed.

The 1 st icon K1 includes, for example, an arrow-shaped pattern, and is disposed on the side of the 1 st pattern R1. When the user operates (for example, clicks) the 1 st icon K1, the screen is switched to a screen displaying a list of all food materials in stock. The details of this will be described later.

The 2 nd to 4 th icons K2 to K4 are arranged at the lower end of the image S3. When the user operates (e.g., clicks) the 2 nd icon K2, the screen is switched to a search screen in which the stored food can be searched. When the user operates (for example, clicks) the 3 rd icon K3, the screen is switched to the one that displays the shopping list. The 3 rd icon K3 is an example of "display of an operation unit for switching to a screen on which a material which urges a user to purchase is displayed". This will be left to the following. When the user operates (for example, clicks) the 4 th icon K4, the screen is switched to the one that displays the trash can list. The trash box list is a list of food materials registered in the trash box DB 603. The trash box list is generated by the display control unit 523 with reference to the trash box DB603, for example.

In the present embodiment, the 2 nd to 4 th icons K2 to K4 are disposed below the 1 st pattern R1. Instead of this, the 2 nd to 4 th icons K2 to K4 may be arranged above the 1 st pattern R1. For example, the 3 rd icon K3 may be disposed on the upper right of the 1 st pattern R1. The 3 rd icon K3 is an example of the "2 nd operation unit". A screen including a shopping list displayed by the operation of the 3 rd icon K3 is an example of "display No. 4".

The door arrangement of the refrigerator 100 and the patterns of the 1 st to 4 th display frames D1 to D4 in the 1 st pattern R1 can be displayed in colors as appropriate, and can be displayed in a manner easily visible.

Fig. 13 shows an image S3A of a modification of the home page for managing food materials. In the present modification, the display controller 523 displays the 1 st pattern R1A instead of the 1 st pattern R1. The 1 st pattern R1A is a pattern in which the refrigerator 100 is modeled in a state where the right and left refrigerating chamber doors are opened and the inside of the refrigerating chamber 121 is visible. The 1 st pattern R1A includes a pattern R1aa representing the fresh food compartment 122.

The display controller 523 displays an image S3A as an image of a home page for managing food materials (i.e., in place of the image S3). In another example, the display controller 523 may display the image S3 as an image of a home page for managing food materials, and switch to the image S3A of the present modification example by the user operating (for example, clicking) the pattern R1a indicating the refrigerator compartment 121 or the 1 st display frame D1 in the image S3. In this case, pattern 1R 1A is not limited to a pattern corresponding to the entire refrigerator 100, and may also display a pattern in which refrigerating compartment 121 and fresh food compartment 122 are enlarged (a pattern in which other storage compartments 120 are omitted).

< 6.3 food Material List >

Fig. 14 is an image S4 showing a list of all food materials L0. The all-food-material list L0 is a list showing all the food materials (all the food materials stored inside and outside the refrigerator 100) registered in the inventory list DB 601. In the all-food-material list L0, icons indicating food material categories, food material names, purchase dates, prize periods (or consumption periods), numbers, and units are displayed for each food material. The food materials are arranged in a predetermined order regardless of the storage location in which the food materials are stored. The "predetermined order" is, for example, the order of the names of the food materials (the order of あいうえお), the order of the purchase dates from old to new (or the order of new to old), and the order of the appreciation period (consumption period) from near to far (or the order of far to near). The user can change the predetermined order (rearrange the display order of the food materials) by operating (for example, clicking) the operation unit G0 included in the image S4. However, the present invention is not limited to this, and the function of rearranging the storage locations may be provided in parallel for each storage location in accordance with the setting operation by the user.

When the 1 st icon K1 of the image S3 (see fig. 12) is operated by the user, the display controller 523 displays the image S4. That is, the display control unit 523 generates display data including all the food material lists L0 based on the inventory list DB601 and transmits the display data to the terminal device 400.

Fig. 15 is an image S5 showing a list of food materials for each storage room 120. For example, fig. 15 is an image of a food material list L2 in which the vegetable room 123 is displayed. The food material list L2 in the vegetable room 123 is a list displayed by selecting the food materials stored in the vegetable room 123 from among the food materials registered in the inventory list DB 601. In the food material list L2, icons indicating food material categories, food material names, purchase dates, expiration dates (or consumption dates), numbers, and units are displayed for each food material. The food materials are arranged in a predetermined order in the same manner as the list of all food materials L0. The user can change the predetermined order (rearrange the display order of the food materials) by operating (for example, clicking) the operation unit G0 included in the image S5.

When the pattern R1b of the vegetable room 123 included in the 1 st pattern R1 of the image S3 (see fig. 12) or the 2 nd display frame D2 is operated (for example, clicked) by the user, the display controller 523 displays the image S5. That is, the display control unit 523 generates display data including the food item list L2 of the vegetable room 123 based on the inventory list DB601 and transmits the display data to the terminal device 400. The same applies to the food item list of refrigerating room 121 and the food item lists of freezing rooms 125 and 126.

Fig. 16 is an image S5A showing a modification of the food material list. In the present modification, the image S5A includes the display of the operation portions O1, O3, and O4 for switching to another display. The other display to be switched includes, in place of the 2 nd information I2 relating to the food stored in the vegetable room 123, the 2 nd information I2 relating to the food stored in the other storage location.

For example, image S5A includes operation units O1, O2, O3, and O4 for switching food item list L1 in refrigerating room 121, food item list L2 in vegetable room 123, food item lists L3 in freezing rooms 125 and 126, and food item lists L4 of other (stored outside refrigerator 100). When food item list L2 of vegetable room 123 is displayed, food item list L1 of refrigerating room 121, food item list L3 of freezing rooms 125 and 126, and other food item list L4 are examples of "display No. 3". When the user operates the operation units O1, O2, O3, and O4, the 4 food material lists L1, L2, L3, and L4 are switched. The display of the operation units O1, O2, O3, and O4 is commonly set in the image in which the food material list of any storage room 120 is displayed.

< 6.4 New Login to food Material List >)

Fig. 17 shows an image S6 displayed when a new registration of a food material is performed to the food material list (inventory list DB 601). In the present embodiment, for example, when the user inputs a name of a food material, the registration information management unit 522 automatically inputs a food material classification, a storage location, a date of purchase, a date of sale (or a time limit of consumption), an icon of the food material classification, and a unit. The food material classification, storage location, date of purchase, date of sale, icon of food material classification, and unit are examples of the "additional information".

In detail, the registration information management unit 522 automatically inputs the food classification, the default storage location, and the unit registered in the food DB605 in association with the food name as "food classification", "storage location", and "unit", respectively, based on the input food name. When the registration information management unit 522 receives a user input for registering additional information in association with the input of the name of the food material, the input content of the user is prioritized over the food material DB605, and the food material is registered in association with the additional information set by the user.

In the present embodiment, when information registered in association with an input food material name (for example, a storage location in which a setting is changed) exists in the user setting DB606, the registered information management unit 522 inputs the information of the user setting DB606 more preferentially than the food material DB 605. That is, the registration information management unit 522 can refer to the material DB605 indicating a general correspondence relationship between the material and the storage location and the user setting DB606 indicating a user-specific correspondence relationship between the material and the storage location. The registration information management unit 522 is set to a state in which a storage location is input based on the content of the user setting DB606 when the user setting DB606 exists, and is set to a state in which a storage location is input based on the food material DB605 when the user setting DB606 does not exist. The registration information management unit 522 acquires information of an icon indicating the corresponding food material classification from the storage unit 530 and automatically registers the information based on the food material classification obtained by referring to the food material DB. The registration information management unit 522 automatically registers the date on which the food item name is input as the "purchase date", and automatically registers the date of the appreciation period (or consumption period) estimated based on the purchase date. With this configuration, the number of steps for registering the food can be reduced.

The registration information management unit 522 may automatically register additional information based on the history DB608 instead of the user setting DB606 or in combination with the user setting DB 606. That is, the registration information management unit 522 may be configured to input the storage location based on the history DB608 when the history DB608 of the target material exists, and to input the storage location based on the material DB605 when the history DB608 of the target material does not exist. In this way, when the additional information is input not based on the material DB605 but based on the history DB608 in the registration of the specific material, it is also possible to set the condition that the frequency of registration of the material is equal to or more than a predetermined frequency. Of course, the frequency of registration of the specific food material can be acquired by referring to the history DB 608.

The registration information manager 522 sets the state where the 1 st storage location is input as additional information when a predetermined condition is not satisfied for a certain food, and sets the state where the 2 nd storage location is input as additional information when the predetermined condition is satisfied. The "predetermined condition" is a condition relating to the degree of filling in the 1 st or 2 nd storage location, or the amount of food registered as stock. For example, when "refrigerator compartment" is associated with "pork" in material DB605, if the filling rate of refrigerator compartment 121 is higher than the threshold value, it is estimated that the user has rejected putting into refrigerator compartment 121 and put into freezer compartment 126, and registered "freezer compartment" as additional information for the pork. In another example, when the "cold room" is associated with "pork" in the material DB605, and when the amount of pork is higher than the threshold value, the "freezer" is registered as additional information for the pork as the case of performing the frozen storage.

Even if the same material is used, the registration information manager 522 may change the automatically inputted prize period (or consumption period) between the case of storing in the refrigerating room 121 and the case of storing in the freezing rooms 125 and 126. For example, even if the same material is used, the registration information manager 522 may set the number of days until the appreciation period (or consumption period) when stored in the refrigerating room 121 to the 1 st day (for example, 3 days), and set the number of days until the appreciation period (or consumption period) when stored in the freezing rooms 125 and 126 to the 2 nd day (for example, 1 week) longer than the 1 st day.

The registration information management unit 522 can accept a user operation of changing additional information after the additional information is input based on the food material DB 605. When receiving such a user operation, the registration information manager 522 replaces the additional information automatically registered in the inventory list DB601 with the content based on the user operation.

In the above-described example, when the user inputs a food name to the terminal device 400, for example, the registration information management unit 522 automatically inputs additional information based on the food DB605 and/or the user setting DB606 without inputting information other than the food name from the outside. The input of the food material name to the terminal device 400 by the user is an example of "input of information indicating the name of the food material from the outside".

However, the "input of information indicating the name of the food material from the outside" is not limited to the manual input by the user. For example, the registration information management unit 522 may receive an image captured by the camera 170 of the refrigerator 100 or the camera 450 of the terminal device 400, specify a food material based on the received image, and automatically input a food material name, a food material classification, a storage room, a purchase date, a taste expiration date (or consumption expiration date), an icon of the food material classification, a unit, and the like corresponding to the specified food material. In this case, the registration information management unit 522 specifies the food from the received image, for example, based on a learned model that is mechanically learned for the image recognition of the food. Further, instead of manually inputting a food name by the user, the registration information management unit 522 accepts an input of a food name by the user speaking to the smart speaker 200, and automatically inputs a food classification, a storage room, a purchase date, a taste expiration date (or consumption expiration date), an icon of a food classification, a unit, and the like corresponding to the accepted food name.

From one perspective, the food material DB605 may include a 1 st DB in which food material names are associated with food material categories, and a 2 nd DB in which food material categories are associated with storage locations. In this case, the registration information management unit 522 may specify the food material classification corresponding to the food material based on the information indicating the name of the food material input from the outside and the 1 st DB, specify the storage location using the specified food material classification and the 2 nd DB, and automatically input the specified storage location as the additional information.

Fig. 18 shows an image S6A of a modified example of newly registering a food material in the food material list. In the present modification, the characters representing the contents input by the user and the characters representing the contents automatically input by the registration information management unit 522 are displayed in different forms. For example, characters indicating the contents input by the user are displayed as black characters, and characters indicating the contents automatically input by the registration information management unit 522 are displayed in a form different from the black characters (blue characters, gray characters, etc., and dotted lines in fig. 18) as the assumed registration state. In this case, the login information management unit 522 may set the contents displayed in the different modes to the formal login state based on the confirmation operation performed by the user on the contents. According to such a modification, it is possible to easily distinguish and confirm the content input by the user himself from the content input automatically.

< 6.4 Change in food Material List >

Fig. 19 shows an image S7 when the food material list is changed (deleted/moved). Fig. 19 illustrates a food material list L2 of the vegetable room 123. The user can select a food material by operating (for example, clicking) the display of the food material to be selected from the food material list L2. When the food material is selected, the 1 st to 3 rd icons K5 to K7 are displayed at the lower end of the image S7.

The 1 st icon K5 is an icon for moving the registration of the selected food material to another storage room. When the 1 st icon K5 is operated (e.g., clicked) by the user, the display controller 523 displays a display S8 for selecting the destination storage room superimposed on the image S7. The registration information management unit 522 changes the storage location associated with the food material in the inventory list DB601 based on the selection result of the display S8 by the user. Thus, the selected food material is deleted from the food material list in the storage location before the movement.

The 2 nd icon K6 is an icon for adding a food material to a shopping list (shopping list DB602) described later. When the 2 nd icon K6 is operated (e.g., clicked) by the user, the display control unit 523 adds the selected food material to the shopping list DB 602. In this case, the registration of the ingredients in the ingredient list is maintained as it is.

The 3 rd icon K7 is an icon for moving the registration of the selected food material to the trash. When the 3 rd icon K7 is operated (e.g., clicked) by the user, the registration information management unit 522 deletes the selected material from the inventory list DB601 and registers the deleted material in the trash DB 603. As described above, in the present embodiment, when the food deleted from the inventory list DB601 is registered in the trash DB603, the same food is registered in the candidate list DB 604.

< 6.5 dustbin List >

Fig. 20 shows an image S8 including a trash box list L5. The trash box list L5 is a list in which the food materials registered in the trash box DB603 are displayed. In the trash box list L5, icons indicating food material categories, food material names, purchase dates, prize periods (or consumption periods), numbers, and units are displayed for each food material. The image S8 shows a display D21 that notifies the user that the food material moved to the trash box list L5 was automatically deleted after a predetermined number of days (for example, 3 days) has passed since the movement.

< 6.6 shopping List and candidate List >

FIG. 21 shows an image S9 including a shopping list L6 and a candidate list L7. Shopping list L6 is a list in which the food materials registered in shopping list DB602 are displayed. In the shopping list L6, icons indicating the categories of food materials, names of food materials, predetermined numbers of purchased food materials, and units are displayed for each food material. The display control unit 523 generates display data including the shopping list L6 based on the shopping list DB602 and transmits the display data to the terminal device 400.

On the other hand, the candidate list L7 is a list in which the food materials registered in the candidate list DB604 are displayed. In the candidate list L7, icons indicating the food material categories and food material names are displayed for each food material. The display control unit 523 generates display data including the candidate list L7 based on the candidate list DB604 and transmits the display data to the terminal apparatus 400.

In the present embodiment, the shopping list L6 and the candidate list L7 are displayed on the same screen on the display screen 411. For example, the shopping list L6 and the candidate list L7 are displayed in an upper and lower arrangement. The "same screen" is a single continuous screen in which the display contents are continuous and which can be displayed without switching the entire display contents in accordance with an external operation. The "same screen" is not limited to the case of being displayed at the same time at all times, and is a screen displayed when an operation (so-called scroll operation) is performed in which the display content is sequentially changed partially and partially without switching the entire display content in accordance with an operation of a user sliding on the screen. For example, the following cases are also included: a case where there are many food materials registered in the shopping list K6 and the display screen 411 is occupied by the shopping list L6; and a case where the candidate list L7 is displayed next after the shopping list L6 by scrolling the shopping list L6 downward. In other words, the "same screen" also includes a case where only a part (for example, a lower end part) of the shopping list L6 and the candidate list L7 are simultaneously displayed, and the like.

For example, in a state where shopping list L6 and candidate list L7 are displayed simultaneously, registration information manager 522 accepts an operation by the user to move the registration of the food material included in candidate list L7 to shopping list L6. That is, the user can move the food material displayed in the candidate list L7 to the shopping list L6.

Fig. 22 is a diagram showing another image S10 including a shopping list L6. The display controller 523 can receive a user operation to set the candidate list L7 to be non-displayed or to make the candidate list L7 less noticeable (e.g., to make the color thinner) than the shopping list L6 on the screen on which the shopping list L6 and the candidate list L7 can be aligned and displayed. In the present embodiment, the candidate list L7 can be closed based on the user operation. For example, when the user performs an operation (e.g., clicks) on the display of the item indicating the candidate list L7 in the image S9, the display controller 523 switches the image S0 including both the shopping list L6 and the candidate list L7 to the image S10 in which the shopping list L6 is displayed in a state where the candidate list L7 is closed. This enables the user to make a purchase while confirming only the shopping list L6, and can suppress erroneous purchases and the like as compared with the case where the candidate list L7 is displayed.

< 7. Voice dialog function >

< 7.1 Overall construction of control >

Next, a voice conversation function of the server system 500 according to the present embodiment will be described. In the present embodiment, the information processing unit 520 of the server system 500 can register or delete a plurality of ingredients before 1-time reception of the voice conversation function is completed. In other words, the speech processing unit 524 can continuously receive a plurality of instructions included in 1-line utterance (a series of utterances, a series of conversations) of the user. Further, activation of the voice conversation function (which may be referred to as "voice recognition function" as described above) means transition to a state in which an utterance of an arbitrary content by a user can be accepted. When the voice conversation function is activated, an instruction included in the speech content of the user is executed or the speech of the user is responded. When the voice conversation function is ended, the user does not accept the speech except for the trigger word which is a password for starting the voice conversation function. A period from one time of starting the voice conversation function to the time of ending the voice conversation function is referred to as "1 reception of the voice conversation function".

Specifically, the voice processing unit 524 makes a speech of the trigger word to the smart speaker 200 to activate the voice dialogue function, and then ends the voice dialogue function by satisfying a predetermined termination condition. The control mode changing unit 525 can select, based on the user's motion, the 1 st mode in which the speech dialogue function is ended under the 1 st end condition and the 2 nd mode in which the speech dialogue function is ended under the 2 nd end condition different from the 1 st end condition.

The 1 st mode is a control mode for receiving 1 instruction from the user to speak. Mode 1 the start-up mode is terminated after 1 instruction has been executed in 1 start-up of the speech dialog function. "ending the start mode" means, for example, that a control signal for ending the standby state or a control signal indicating that the standby state does not need to be continued is transmitted from the server system 500 to the smart speaker 200. Examples of the control received and set in mode 1 include control related to the function of refrigerator 100. The control related to the function of refrigerator 100 is, for example, a change in a set temperature zone of each storage chamber 120, a change in a control mode of fresh air chamber 122, "quick freezing" or "quick ice making" or the like. The 1 st termination condition is that 1 control is executed according to the content of speech of the user, or that no speech of the user is made after a certain time or longer after the voice conversation function is started (so-called timeout, which corresponds to the "1 st standby time" in the present invention). In the 1 st mode, in order to input a plurality of instructions, it is necessary to speak a trigger word every time an instruction is made and to activate the voice conversation function once and again.

The 2 nd mode is a control mode capable of being managed by a plurality of instructions made by the speech of the user in 1 start of the voice conversation function. That is, the 2 nd mode is a control mode capable of continuing the activation state even if 1 instruction is received in 1 activation. In the 2 nd mode, there is a function of storing in advance instructions for a plurality of processing targets even when the user has a plurality of processing targets in the utterance content in 1 st line. The "1-line utterance of the user" referred to herein means a series of utterances that are continuously performed by the user without sandwiching the response of the smart speaker 200 and the execution of the control based on the utterance content. In the present embodiment, the instruction received in the 2 nd mode is an instruction relating to food material management. Examples of the reference relating to the management of food materials are registration, deletion, or confirmation of stock relating to a plurality of food materials. The 2 nd termination condition is that an instruction to terminate the voice conversation function is received based on an action of the user such as an utterance. In the present embodiment, when the mode 2 is executed and there is no utterance of the user for a predetermined time, a voice inquiring whether or not there is an addition instruction (voice for end confirmation) is output from the smart speaker 200, and when the response of the user to the voice satisfies a predetermined condition, it is regarded that the end condition 2 is satisfied. The details of the 2 nd termination condition will be described later. In the present embodiment, the control pattern changing unit 525 executes the 2 nd pattern and accepts an instruction from a user regarding registration, deletion, and confirmation of materials for various DBs.

In the present embodiment, the control mode changing unit 525 executes the 2 nd mode based on the user performing a predetermined operation. On the other hand, the control mode changing unit 525 executes the 1 st mode when the user does not perform a predetermined operation. The "predetermined action" is, for example, a user making a comment including a predetermined word, a predetermined operation portion (for example, a predetermined button) provided in smart speaker 200 (or terminal device 400 or refrigerator 100) being operated, or door 130 of refrigerator 100 being opened (and/or closed).

The "prescribed word" is a word relating to the food material. The term "related to the food material" is not limited to the name of the food material itself, and includes a similar meaning term and the like. The "prescribed word" is: words representing the name of the food material itself; words such as "vegetables" and "fruits" that indicate food material classification; terms related to food material management such as "food material management", "food material registration", "stock management", "stock registration", "new purchase"; or let the user specify words that are sequential skills, etc. The continuous skill means the operation in the 2 nd mode.

In other words, the control mode changing unit 525 executes the 1 st mode when the words related to the functions of the refrigerator 100 are included in the utterance of the user, and executes the 2 nd mode when the words related to the near-meaning words of the food materials or the food materials are included in the utterance of the user.

In the present embodiment, the voice processing unit 524 terminates the active state after 1 st standby time elapses during the standby of the utterance of the user in the 1 st mode, and continues the active state until 2 nd standby time longer than the 1 st standby time elapses during the standby of the utterance of the user in the 2 nd mode. For example, the voice processing unit 524 terminates the active state after waiting for the 1 st standby time from the stop of the voice of the user in the 1 st mode, and terminates the active state after waiting for the 2 nd standby time longer than the 1 st standby time from the stop of the voice of the user in the 2 nd mode. The 1 st standby time is, for example, several seconds. The 2 nd standby time is, for example, a time which is twice or more the 1 st standby time, for example, 30 seconds or more, for example, 1 minute or more. Since the user may open the door 130 of the refrigerator 100 and make a comment about the food management while checking the inside, the 2 nd standby time is set to be longer than the 1 st standby time.

From another viewpoint, the voice processing unit 524 waits for a 3 rd standby time and then shifts to the next processing after performing the processing corresponding to the utterance content of the user in the 1 st mode, and waits for a 4 th standby time longer than the 3 rd standby time and then shifts to the next processing after performing the processing corresponding to the utterance content of the user in the 2 nd mode. For example, the speech processing unit 524 may perform processing according to the speech content of the user in the 1 st mode, wait for the 3 rd standby time and then shift to the next processing, and wait for the 4 th standby time longer than the 3 rd standby time from when the speech of the user stops in the 2 nd mode and then shift to the next processing. The "next processing" in the 1 st mode is, for example, processing for guiding a case where the processing instructed by the user has ended (or started) by voice (for example, output of a voice indicating that "quick freezing has started"). The "next process" in the 2 nd mode may be a process of guiding the completion of the process instructed by the user by voice (for example, a process of outputting a voice indicating "registration completion"), or a process of outputting a voice inquiring the presence or absence of an addition instruction related to the food material management (a voice of completion confirmation described later).

In the modification, the voice processing unit 524 performs processing corresponding to the utterance of the user and then shifts to the next processing without waiting for a predetermined time in the 1 st mode, and shifts to the next processing after waiting for a predetermined time from the stop of the voice of the user in the 2 nd mode. The "next process" in the 1 st mode is, for example, a process of providing guidance by voice to a case where the process based on the user's instruction has ended (or started), as described above. The "next process" in the 2 nd mode may be a process of guiding the completion of the process by the user's instruction by voice (for example, a process of outputting a voice indicating "registration completed"), or a process of outputting a voice inquiring the presence or absence of an additional instruction related to the material management (a voice of completion confirmation described later), as in the above.

In the present embodiment, when the 2 nd mode is executed, the voice processing unit 524 outputs a voice inquiring whether or not there is an addition instruction related to the material management after the voice of the user is stopped, and ends the activated state when the response of the user to the output of the voice satisfies a predetermined condition. The "predetermined condition" is, for example, a case where, as the response of the user, there is a speech indicating that no instruction is added, there is a speech confirming the termination of the voice conversation function, or there is no speech of the user for a certain time or longer (so-called timeout).

< 7.2 control flow >

Fig. 23 is a flowchart showing a flow of processing related to the voice dialogue function. The following is an example in which the 2 nd mode is executed when a predetermined word is included in the speech of the user.

First, when the user speaks, the information processing unit 250 of the smart speaker 200 determines whether or not the trigger word is included in the utterance of the user (S101). When the trigger word is not included in the utterance of the user (S101: no), the information processing unit 250 of the smart speaker 200 continues the resting state. On the other hand, when the trigger word is included in the utterance of the user (yes in S101), the information processing unit 250 of the smart speaker 200 activates the voice dialogue function of the smart speaker 200 (S102). In this way, smart speaker 200 acquires speech data of the utterance of the user, and transmits the acquired speech data to server system 500.

The information processing unit 520 (hereinafter, simply referred to as "information processing unit 520") of the server system 500 analyzes the voice data received from the smart speaker 200, and determines whether or not a predetermined word (for example, a word related to a food material or a synonym of a food material) is included in the utterance of the user (S103).

When the user' S speech does not include a predetermined word (no in S103), the information processing unit 520 receives an instruction from the user in the 1 st mode (S104), and executes a process corresponding to the received instruction (S105). For example, when receiving an instruction related to a function of refrigerator 100, information processing unit 520 generates a control instruction for causing refrigerator 100 to execute the function, and transmits the generated control instruction to refrigerator 100.

Then, information processing unit 520 receives a response to refrigerator 100 in response to the transmitted control instruction, and if the response is normal, generates a voice indicating that the instruction received from the user is executed (or started), and outputs the voice from smart speaker 200. After that, the information processing unit 520 transmits a control signal for ending the standby state or a control signal indicating that the standby state is not required to be continued to the smart speaker 200, thereby ending the active state of the smart speaker 200 and causing the smart speaker 200 to transition to the resting state (S106).

On the other hand, when a predetermined word is included in the utterance of the user (yes in S103), the information processing unit 520 switches the control mode from the 1 st mode to the 2 nd mode, receives an instruction from the user in the 2 nd mode (S111), and executes processing corresponding to the received instruction (S112). For example, the information processing unit 520 performs registration, deletion, and confirmation of a plurality of food materials with respect to various DBs (for example, the inventory list DB601) based on an instruction from the user. For example, the information processing unit 520 may execute the processing in order from an instruction having a specified content among 1 or more instructions of the user without confirming the presence or absence of the additional instruction to the user. Alternatively, the information processing unit 520 may inquire of the user whether or not an addition instruction is present, confirm that no addition instruction is present, and then collectively execute the plurality of instructions received thus far.

Next, the information processing unit 520 determines whether or not an end instruction is given from the user (S113). The end is a word or phrase such as "end", "above", "this is all", "bailey", and the like, indicating that the user has not been instructed to add. Instead of these instructions or in addition to them, the end instruction may be a case where a predetermined operation portion (e.g., a predetermined button) provided in the smart speaker 200 (or the terminal device 400 or the refrigerator 100) is operated (or an operation portion that is originally pressed is released), a case where the door 130 that is originally opened of the refrigerator 100 is closed, or the like.

When an end instruction is given from the user (yes in S113), the information processing unit 520 transmits a control signal for ending the standby state or a control signal indicating that the standby state is not required to be continued to the smart speaker 200, thereby ending the active state of the smart speaker 200 and causing the smart speaker 200 to transition to the resting state (S106).

On the other hand, if there is no termination instruction from the user (no in S113), the information processing unit 520 outputs a voice (for example, "do else. Then, the information processing unit 520 determines whether or not there is an utterance of an addition instruction from the user (S115). In other words, even when 1 or more instructions are accepted from the user, the information processing unit 520 continues the standby state without terminating the voice conversation function when there is no termination instruction from the user.

When an addition instruction is given from the user (yes in S115), the information processing unit 520 returns to S112 and executes the processing corresponding to the addition instruction. On the other hand, when there is no additional instruction from the user (NO in S115), the information processing unit 520 returns to S113 to check again whether or not there is an instruction to end the user. Then, if there is no instruction to end the user even after the voice for end confirmation is output 1 or more times (for example, a plurality of times), the information processing unit 520 ends the voice dialogue function due to an timeout.

< 7.3 example of processing >

Before completing the reception of the voice conversation function for 1 time, the information processing unit 520 processes the 1 st food in association with the 1 st storage location and processes the 2 nd food in association with the 2 nd storage location when the 1 st food is spoken in association with the 1 st storage location and the 2 nd food is spoken in association with the 2 nd storage location. For example, when the user says "add 3 eggs to the refrigerating room, add 1 decapterus maruadsi to the freezing room, and delete 1 paprika from the vegetable room", the log information manager 522 associates and processes the "refrigerating room" as the storage location of the "eggs", associates and processes the "freezing room" as the storage location of the "decapterus maruadsi", and associates and processes the "vegetable room" as the storage location of the "paprika".

The information processing unit 520 performs processing for associating a plurality of food materials with 1 storage location when utterances related to the plurality of food materials are performed before 1 time of reception of the voice conversation function is completed and utterances related to the plurality of food materials are performed before or after 1 storage location. For example, in the case where the user says "add 3 eggs and 1 middle-piece decapterus maruadsi" to the refrigerating room "or" add 3 eggs and 1 middle-piece decapterus maruadsi "to the refrigerating room", the registration information manager 522 associates "refrigerating room" as the storage locations of "eggs" and "decapterus maruadsi".

In the present embodiment, when the name of the food material can be recognized based on the content of the utterance of the user, the utterance relating to the storage location and/or the number of food materials is not essential. For example, when there is no comment about the storage location or the number of comments, the information processing unit 520 may continue the registration with the information thereof as an empty field. Alternatively, the information processing unit 520 may automatically input a default storage location registered in the food material DB605 for the storage location. Alternatively, the information processing unit 520 may output a voice asking the user from the smart speaker 200 for insufficient information.

In the present embodiment, the stock status of the ingredients managed by the voice dialogue function can be changed by an input operation to the content of the display screen 411 displayed on the terminal device 400. The information processing unit 520, after changing the stock status of the food material by an input operation on the content of the display screen 411, manages the stock status of the food material by the voice dialogue function based on the changed stock status.

The management of the stock status of food materials at least comprises information of the names of the food materials and the number of the food materials. In the voice conversation function, when the name of a material is received based on the utterance of the user but the number of the materials is not included in the content of the utterance, the information processing unit 520 performs processing in accordance with the case where a predetermined number (for example, 1 or 1 packet) based on a predetermined rule is registered, and receives registration of the material. In the modification, in the voice dialogue function, when the name of the food material is received based on the utterance of the user but the number of the food material is not included in the utterance content, the information processing unit 520 does not receive registration of the food material when outputting a voice inquiring about the number of the food material and when failing to acquire information indicating the number of the food material based on the utterance of the user.

In the present embodiment, when the stock of food materials is reported to the user and the number of the food materials in stock is not less than the 1 st predetermined amount (e.g., not less than 6), the information processing unit 520 outputs, by voice, a few food materials that exist as stock, outputs, by voice, the name of a food material of the 2 nd predetermined amount (e.g., 5) that is less than the 1 st predetermined amount, and generates and outputs a voice asking whether or not the names of the remaining food materials are also recited. For example, when there are 20 food items, the information processing unit 520 outputs "all 20 food items" by voice, outputs the names of 5 food items having a high priority by voice in a predetermined priority order (for example, date of purchase, time limit of enjoyment, and time limit of consumption), and outputs a voice for confirming to the user whether or not the remaining food items are also read. That is, when a large number of food materials are registered, it is troublesome to listen to all of them at once, and therefore, the user speaks the food materials in several portions at a time.

In the present embodiment, when a material registered in the inventory list DB601 is deleted and a plurality of materials with the same name exist, the registration information management unit 522 selects a registration to be deleted based on the purchase date, the date of appreciation, or the time limit of consumption. For example, when there are a plurality of materials with the same name, the registration information management unit 522 preferentially deletes the registration from the material with the old date of purchase, the material with the near-future date of appreciation, or the material with the near-future date of consumption. For example, when 3 tomatoes are registered on day 2/15 and 2 tomatoes are registered on day 2/16, if the user makes a statement "delete 4 tomatoes", 3 tomatoes on day 2/15, 1 tomato on day 2/16, and a total of 4 tomatoes are deleted. The manner of deletion is not limited to this. For example, when a storage location is designated as "delete tomatoes from a vegetable room", the deletion may be performed only for the food in the storage location. Similarly, in the stock confirmation of the food materials, when the storage location is designated by the user, the number of food materials may be answered only for the food materials in the storage location. In addition, in the material deletion, when the user does not speak the number, the material itself may be deleted.

In the present embodiment, when there is a shortage of food materials that are not stored or are stored at a predetermined amount or less, the response generation unit 527 generates and outputs a voice prompting the shortage of food materials to be added to the shopping list. This can save additional labor and time for adding food to the shopping list.

The following illustrates a speech example. U denotes a user and S denotes the smart speaker 200.

(speech example 1)

U: "Hello good quality (trigger word), open Δ (name of APP)"

S: "what is good, what is i needed to do? "

U: food material management "

S: "good"

U: "register 3 carrots"

S: "3 carrots are registered"

U: "Login 3 cucumbers"

S: "3 cucumbers have been registered"

U: 'above'

S: end of food registration (other than trigger word) voice conversation function stop

(speech example 2)

And accepting the food material registration when the information of at least the food material name and the number of the food materials is complete, and inquiring the user if the information is insufficient.

U: "Hello good quality (trigger word), open Δ (name of APP)"

S: "good"

U: please register food material "

S: "good"

U: '3 tomatoes'

S: '3 tomatoes'

U: egg "

(lapse of a predetermined time)

S: "several eggs? "

U：“10”

S: "10 eggs are registered. "

S: "are there others? "

U: "not have"

S: ending food material registration "

< 7.3 example >

(example 1)

The method includes registering food materials by associating food material names with the number of the food materials, and accepting registration of the food material names without inquiring the number from the voice recognition function side even if there is no information on the number in the utterance contents at the time of the food material registration instruction. For example, the food items having no information are registered in an empty column. Alternatively, the ingredients having no information are registered in a predetermined amount. In this case, the predetermined amount can be registered by setting by the user, or the amount of the food having no information can be registered in the minimum unit amount.

(example 2)

If the food material registration is accepted through the voice recognition, the accepted content is read. The reading of the accepted content is a chorus formula in which the reading is performed for each registration of the food material. In this case, the spoken content is only the food material name. At the time of registration, voice guidance for prompting a user to input an unknown number of food materials is performed (on the premise, both a food material registration function by voice recognition and a food material registration function by manual input are provided).

(example 3)

In the food registration, when an instruction to register a food is accepted in advance by voice recognition, information of a name of the food recognized by a series of voice recognition functions is all accepted as a food registration instruction. For example, when the instruction to register the food material is expressed by the verb V and the name of the food material to be registered is expressed by the target language O, if the utterance of the user is "V (O1, O2, O3, O4)," the response of the smart speaker 200 is "V is performed to O1, O2, O3, O4".

On the other hand, if the utterance of the food name is started without previously hearing the instruction of the registration of the food, the food name is temporarily stored, and a subsequent instruction (presence or absence of registration, deletion, or stock) is waited for (or inquired about). For example, in the case where the utterance of the user is "O1, O2, O3, O4", the response of the smart speaker 200 is "yes · · O1, O2, O3, O4, to cafe? Do they go to V1? Or proceed to V2? ".

The control unit 160 of the refrigerator 100, the information Processing unit 250 of the smart speaker 200, the information Processing unit 440 of the terminal device 400, and the information Processing unit 520 of the server system 500 described above are each realized by executing a program (software) by a hardware processor such as a cpu (central Processing unit) or a gpu (graphics Processing unit) mounted on each device. However, all or a part of the control unit 160, the information processing unit 250, the information processing unit 440, and the information processing unit 520 may be realized by hardware (including circuit units) such as an asic (application Specific Integrated circuit), a pld (Programmable Logic device), an fpga (field Programmable Gate array), or the like, or may be realized by cooperation of software and hardware.

The embodiments have been described above, and the contents of the embodiments are not limited to the above examples. For example, fig. 24 to 38 show modified examples of images displayed on the display screen 411 of the terminal apparatus 400 (or the display screen 141 of the refrigerator 100). For example, fig. 24 and 25 show examples in which the 1 st information I1 is displayed in a different form instead of the 1 st to 4 th display frames D1 to D4. Fig. 25 is a pattern also illustrating the handle portion of each door 130.

Fig. 26 to 31 show modifications of the 1 st pattern R1. That is, the pattern R1 schematically showing the refrigerator 100 may be changed according to the shape, door arrangement, and the like of the refrigerator 100. In fig. 26 to 31, the food material information of each storage room 120 and each storage room 120 can be displayed in a linked manner by displaying the display frames D1 to D4 shown in fig. 12 and 13 or displaying the same.

Fig. 32 to 38 show modifications of the patterns of the 1 st to 4 th display frames D1 to D4. That is, the 1 st to 4 th display frames D1 to D4 are not limited to rectangular shapes, and may be circular shapes, elliptical shapes, oblong shapes, and the like as appropriate. The arrangement positions of the 1 st to 4 th display frames D1 to D4 are not limited to the side of the pattern of the refrigerator, and may be upward, obliquely upward, downward, obliquely downward, or the like. For example, fig. 35 is an example in which a part of the display frame is displayed so as to overlap with a part of the pattern of the refrigerator. Note that the contents shown in fig. 12, 13, and 24 to 38 can be combined with each other as appropriate.

According to at least one embodiment described above, by providing a processing unit capable of registering a plurality of types of food materials before 1-time acceptance of the voice conversation function is completed, it is possible to improve usability.

Several embodiments of the present invention have been described, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in other various forms, and respective omissions, substitutions, and changes can be made without departing from the spirit of the invention. These embodiments and modifications are included in the scope and spirit of the invention, and are also included in the invention described in the claims and the equivalent scope thereof.

Claims (17)

1. An information processing system in which, among other things,

the disclosed food processing device is provided with a processing unit that can register a plurality of types of food materials before 1-time acceptance of a voice conversation function is completed.

2. The information processing system of claim 1,

the processing unit is capable of executing a 1 st mode in which the activation state is terminated after 1 instruction is accepted during 1 activation and a 2 nd mode in which the activation state is continued even if 1 instruction is accepted during 1 activation, and executes the 2 nd mode for registration of a food material.

3. The information processing system of claim 2,

the processing unit executes the 2 nd mode based on a situation that a user performs a predetermined operation.

4. The information processing system of claim 3,

the prescribed action is to make a statement that includes a prescribed word.

5. The information processing system of claim 4,

the prescribed word is a word related to the food material.

6. The information processing system of any one of claims 2 to 5,

the processing unit executes the 1 st mode when a word related to a function of the refrigerator is included in a user utterance, and executes the 2 nd mode when a word related to a food material is included in the user utterance.

7. The information processing system of any one of claims 2 to 6,

the processing unit terminates the active state after a 1 st standby time elapses while the user is waiting for speech in the 1 st mode, and continues the active state until a 2 nd standby time longer than the 1 st standby time elapses while the user is waiting for speech in the 2 nd mode.

8. The information processing system of any one of claims 2 to 6,

the processing unit waits for a 3 rd standby time and then shifts to the next processing after performing the processing corresponding to the utterance content of the user in the 1 st mode, and waits for a 4 th standby time longer than the 3 rd standby time and then shifts to the next processing after performing the processing corresponding to the utterance content of the user in the 2 nd mode.

9. The information processing system of any one of claims 2 to 8,

the processing unit outputs a voice asking whether or not there is an addition instruction related to material management after the voice of the user stops when the 2 nd mode is executed, and ends the activated state when a response of the user to the output of the voice satisfies a predetermined condition.

10. The information processing system of any one of claims 1 to 9,

the stock status of the food material managed by the voice dialogue function can be changed by an input operation performed on the content displayed on the display screen of the predetermined device,

after the stock status of the food material is changed by an input operation performed on the content of the display screen, the stock status of the food material is managed by the voice dialogue function based on the changed stock status.

11. The information processing system of claim 10,

the management of the stock status of food materials at least comprises information of the names of the food materials and the number of the food materials,

in the voice conversation function, when the name of a material is received based on the utterance of the user but the number of the materials is not included in the utterance content, the registration of the material is received in accordance with a predetermined number of cases registered based on a predetermined rule.

12. The information processing system of claim 10,

the management of the stock status of food materials at least comprises information of the names of the food materials and the number of the food materials,

in the voice conversation function, when the name of the food material is received based on the utterance of the user but the number of the food material is not included in the content of the utterance, if the voice inquiring about the number of the food material is output and information indicating the number of the food material is not obtained based on the utterance of the user, the registration of the food material is not received.

13. The information processing system of any one of claims 1 to 12,

the processing unit processes the 1 st food in association with the 1 st storage location and processes the 2 nd food in association with the 2 nd storage location when the 1 st food is spoken in a manner associated with the 1 st storage location and the 2 nd food is spoken in a manner associated with the 2 nd storage location before 1 time of reception of the voice conversation function is completed.

14. The information processing system of any one of claims 1 to 13,

the processing unit processes the plurality of food materials in association with 1 storage location when utterances related to the plurality of food materials are made and utterances related to the plurality of food materials are made 1 storage location before or after 1 reception of the voice conversation function is completed.

15. The information processing system of any one of claims 1 to 14,

when the stock is reported to the user and the number of the stocked food materials is equal to or more than the 1 st predetermined amount, the processing unit outputs a few food materials as the stock by voice, then outputs the name of the 2 nd predetermined amount of food materials smaller than the 1 st predetermined amount by voice, and then outputs a voice asking whether the names of the remaining food materials are to be read aloud.

16. The information processing system of any one of claims 1 to 15,

the processing unit selects a registration to be deleted based on the purchase date, the appreciation period, or the consumption period when the registration of the stored food is deleted and when there are a plurality of food items with the same name.

17. The information processing system of any one of claims 1 to 16,

the processing unit outputs a voice prompting the insufficient food material to be added to a predetermined list when there is no stock or the stock is less than a predetermined amount.

Images