JP2022130059A - Program, terminal device and terminal device control method - Google Patents

Abstract

To provide a program, a terminal device and a terminal device control method which enable a user to intuitively assess what kind of food is stored in a refrigerator.SOLUTION: A program causes a computer of a terminal device mounted with a display section to function as a display control section which displays: a refrigerator interior image representing an interior of a refrigerator viewed from a front side thereof on the display section; and a food image representing food stored in the refrigerator in an area on the refrigerator interior image corresponding to a storage area of the food inside the refrigerator.SELECTED DRAWING: Figure 16

Description

The present disclosure relates to a program, a terminal device, and a terminal device control method.

In Patent Document 1, when an object passing through an opening of the refrigerator is detected, a camera installed on the top of the refrigerator takes a picture, and based on the photographed result, information about the food stored in the refrigerator is sent to the refrigerator or a terminal. A device display technology is disclosed.

JP 2019-070476 A

The present disclosure provides a program, a terminal device, and a control method of the terminal device that allow the user to intuitively grasp what kind of food is stored in the refrigerator.

A program according to the present disclosure causes a computer of a terminal device having a display unit to display, on the display unit, an interior image showing a case in which the interior of a refrigerator is viewed from the front of the refrigerator, and in the displayed interior image and a display control unit for displaying a food image representing the food in an area corresponding to the storage area of the food stored in the storage.

Further, the terminal device according to the present disclosure displays a display section, and an interior image showing a case in which the interior of a refrigerator is viewed from the front of the refrigerator by the display section. a display control unit for displaying a food image showing the food in an area corresponding to the food storage area stored therein.

In addition, the control method of the terminal device according to the present disclosure displays a refrigerator interior image showing a case in which the interior of the refrigerator is viewed from the front of the refrigerator. A food image showing the food is displayed in an area corresponding to the accommodation area of .

The program, the terminal device, and the control method of the terminal device according to the present disclosure display an image of the food in an area corresponding to the storage area of the food stored in the refrigerator in the image showing the inside of the refrigerator. Therefore, the user can intuitively grasp what kind of food is stored in the refrigerator.

The figure which shows the structure of the food management system in Embodiment 1 The side view which looked at the camera unit installed in the refrigerator in Embodiment 1 from the right side FIG. 4 is a perspective view of the position regulating member in contact with the front edge of the main housing according to Embodiment 1, viewed from the front left side; Fig. 2 is a front view of the refrigerator and the camera unit according to Embodiment 1 as seen from the front side; FIG. 2 is a plan view of the refrigerator and the camera unit according to Embodiment 1 as viewed from the right side; 2 is a block diagram showing the configuration of the camera unit according to Embodiment 1; FIG. Block diagram showing the configuration of the terminal device and the food management server according to Embodiment 1 A diagram showing an example of a refrigerating room management database according to Embodiment 1 A diagram showing an example of an image inside a refrigerator according to the first embodiment. A diagram showing an example of a withdrawal management database in Embodiment 1 A diagram showing an example of an adjustment screen according to the first embodiment. A diagram showing an example of a central position recording sheet according to the first embodiment. 4 is a flowchart showing the operation of the camera unit according to Embodiment 1; Flowchart showing the operation of the food management system according to Embodiment 1 A diagram for explaining the entering/leaving determination image according to the first embodiment. A diagram showing an example of a refrigerator home screen according to Embodiment 1 FIG. 4 shows a memo input procedure according to the first embodiment; A diagram showing an example of a containment list screen according to the first embodiment. A diagram showing an example of a new addition list screen in Embodiment 1 Diagram for explaining the procedure for adding food information in Embodiment 1 A diagram showing an example of a deletion target list screen according to the first embodiment Flowchart showing the operation of the food management system according to Embodiment 1 A diagram showing an example of a vegetable compartment home screen according to Embodiment 1 A diagram showing an example of a food image list screen according to Embodiment 1 Flowchart showing the operation of the food management system according to Embodiment 1 Block diagram showing configurations of a terminal device and a food management server according to Embodiment 2

(Knowledge, etc. on which this disclosure is based)
At the time when the inventors came up with the present disclosure, there was a technology in which a terminal device displayed a list of foods stored in a refrigerator. However, the inventors found a problem that the user cannot intuitively grasp what kind of food is stored in the refrigerator by simply displaying a list of foods as in the conventional technology. In order to solve the problem, the subject of the present disclosure has been constructed.
Accordingly, the present disclosure provides a program, a terminal device, and a control method of the terminal device that allow the user to intuitively grasp what kind of food is stored in the refrigerator.

Hereinafter, embodiments will be described in detail with reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially the same configurations may be omitted.
It should be noted that the accompanying drawings and the following description are provided to allow those skilled in the art to fully understand the present disclosure and are not intended to limit the claimed subject matter thereby.

(Embodiment 1)
First, Embodiment 1 will be described.
[1-1. Constitution]
[1-1-1. Configuration of food management system]
FIG. 1 is a diagram showing the configuration of a food management system 1000. As shown in FIG.

The food management system 1000 is a system for managing the storage date and time of storage in the refrigerator 1 for the food stored in the refrigerator 1 .
The refrigerator 1 corresponds to an example of the storage of the present disclosure. Food corresponds to an example of an article of the present disclosure.

A food management system 1000 includes a refrigerator 1 . A refrigerator 1 has a main box body 10 with an open front surface. A refrigerator compartment 11 , an ice making compartment 12 , a fresh freezing compartment 13 , a freezing compartment 14 and a vegetable compartment 15 are formed in the main box body 10 . A left door 11A and a rotary right door 11B are provided at the front opening of the refrigerator compartment 11 . The ice making compartment 12, the fresh freezing compartment 13, the freezing compartment 14, and the vegetable compartment 15 are provided with drawers 12A, 13A, 14A, and 15A for storing foods, respectively. The drawers 14A, 15A of the present disclosure comprise two cases CA, an upper case JCA and a lower case GCA.
The upper case JCA corresponds to an example of the first case of the present disclosure. The lower case GCA corresponds to an example of the second case of the present disclosure.

In the following description, when the left door 11A and the right door 11B are not distinguished from each other, they are referred to as "doors" and denoted by "11C". Moreover, in the following description, when the drawers 12A, 13A, 14A, and 15A are not distinguished, they are referred to as "drawers" and denoted by "16A".

A food management system 1000 includes a camera unit 2 . The camera unit 2 is a device that takes a picture and detects food that is put in and out of the refrigerator 1 from the taken image. Camera unit 2 is provided on upper surface 10B of main housing 10 of refrigerator 1 .
The camera unit 2 corresponds to an example of the imaging device disclosed herein.

A food management system 1000 includes a terminal device 3 . The terminal device 3 is, for example, a smart phone or a tablet terminal. The terminal device 3 is installed with an application program for managing food in the refrigerator 1, and communicates with the food management server 4 by the function of this application program.
In the following description, this application program is referred to as a "food management application" and denoted by "311".
The food management application 311 corresponds to an example of the program and application program of the present disclosure.

In FIG. 1, a solid line indicates a user P who is at home, and a dotted line indicates a user P who has left home H. As shown in FIG. The terminal device 3 communicates with the food management server 4 via the communication device 5 or not via the communication device 5 by the function of the food management application 311 when used by the user P who is at home. In addition, when the terminal device 3 is used by the user P who has left home H and cannot establish a communication connection with the communication device 5, the function of the food management application 311 enables the food management server to operate without the communication device 5. Communicate with 4.

The communication device 5 connects to a global network GN including the Internet, a telephone network, and other communication networks, and communicates with the food management server 4 connected to the global network GN. The communication device 5 is an interface device for connecting the terminal device 3 to the global network GN.

The food management system 1000 has a food management server 4 . Food management server 4 is a server device that manages food in refrigerator 1 and is connected to global network GN. In each figure, the food management server 4 is represented by one block, but this does not necessarily mean that the food management server 4 is composed of a single server device. For example, the food management server 4 may be configured including a plurality of server devices with different processing contents.

Next, configurations of the camera unit 2, the terminal device 3, and the food management server 4 will be described.

[1-1-2. Configuration of camera unit]
The camera unit 2 will be described with reference to FIGS. 2 to 4. FIG.
2-4, the X, Y, and Z axes are illustrated. The X-axis, Y-axis, and Z-axis are orthogonal to each other. The Z-axis indicates the vertical direction. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the horizontal direction, and also indicates the width direction of the refrigerator 1 and the camera unit 2 . The Y-axis indicates the front-rear direction, and also indicates the depth direction of the refrigerator 1 and the camera unit 2 . The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the forward direction. The positive direction of the Z-axis indicates the upward direction.

FIG. 2 is a side view of the camera unit 2 installed in the refrigerator 1 as seen from the right side. FIG. 3 is a perspective view of the position regulating member 205 in contact with the front edge 10A of the main housing 10, viewed from the front left side.

As shown in FIGS. 2 and 3, the camera unit 2 includes a box-shaped main body 201 installed on the upper surface 10B of the main box body 10 . Further, the camera unit 2 includes a photographing member 202 extending forward from the main body 201 above the front portion of the main body 201 . The photographing member 202 includes a refrigerator camera 203 and a drawer camera 204 . Each of the refrigerator camera 203 and the drawer camera 204 is provided in front of the main body 201 in the lower part of the photographing member 202 .

The main body 201 includes a position regulating member 205 that abuts against the front edge 10A of the main housing 10 to regulate the position of the photographing member 202 in the front-rear direction of the refrigerator 1 . The position regulating member 205 is a position regulating piece that extends downward from the bottom surface 206 of the main body 201 in the installed state. The position regulating member 205 extends in the left-right direction of the main body 201 on the front surface of the main body 201 and has a mark 207 in the center of the main body 201 in the left-right direction. This mark 207 indicates the central position of the camera unit 2 in the horizontal direction.

The length of the position regulating member 205 is formed shorter than the length from the upper surface 10B of the main box body 10 to the packing provided between the refrigerator 1 and the door 11C in the installed state. More specifically, the length of the position regulating member 205 is formed shorter than the length from the upper surface 10B of the main box body 10 to the door packing provided substantially parallel to the upper surface 10B on the upper inner surface of the door 11C in the installed state. It is

The main body 201 is fixed at a position regulated by a position regulating member 205 by means of a built-in magnet 208 . The means for fixing the main body 201 is not limited to the magnet 208, and other means such as double-sided tape may be used.

FIG. 4 is a front view of the refrigerator 1 and the camera unit 2 as seen from the front side. In the refrigerator 1 shown in FIG. 4, the left door 11A and the right door 11B are open. FIG. 5 is a plan view of the refrigerator 1 and the camera unit 2 as seen from the right side. Refrigerator 1 shown in FIG. 5 is in a state in which drawer 15A is in an open state, that is, in a drawn-out state. In the camera unit 2 shown in FIGS. 4 and 5, the mark 207 is positioned substantially in the center of the refrigerator 1 in the horizontal direction, and the position regulating member 205 is in contact with the front edge 10A.

The camera unit 2 has a refrigerator camera 203 and a drawer camera 204 .
The refrigerating compartment camera 203 captures an image of food entering and leaving the refrigerating compartment 11 by capturing an image of the lower front of the refrigerator 1 from above.

For example, the photographing range of the refrigerator camera 203 includes the range A1 shown in FIG. 4 when viewed from the front. The range A1 is a range including the door pocket 111 provided in the door 11C and the front opening of the refrigerator compartment 11 when the door 11C is open. In addition, the photographing range of refrigerator camera 203 includes range A3 shown in FIG. 5 in a side view. Range A3 is a range that includes the open door 11C to the front end of the shelf 112 in the front-rear direction, and a range that includes the opening of the refrigerator compartment 11 in the vertical direction.

The drawer camera 204 photographs the food being put into and taken out of the drawer-type accommodation chamber by photographing the lower front of the refrigerator 1 from the upper front.

For example, the photographing range of the drawer camera 204 includes the range A2 shown in FIG. 4 when viewed from the front. Range A2 is a range including drawer 16A in front view of refrigerator 1 . In addition, the photographing range of the drawer camera 204 includes the range A4 shown in FIG. 5 in a side view. Range A4 is a range that includes drawer 16A in the most pulled out state in the front-rear direction.

The camera unit 2 has a left door ranging sensor 209 , a right door ranging sensor 210 , and a drawer ranging sensor 211 .

The left door distance measuring sensor 209 is a distance measuring sensor for detecting the open/closed state of the left door 11A. The left door distance measuring sensor 209 detects the distance from the left door 11A as the open/closed state of the left door 11A.

The right door ranging sensor 210 is a ranging sensor for detecting the open/closed state of the right door 11B. The right door distance measuring sensor 210 detects the distance from the right door 11B as the open/closed state of the right door 11B.

The drawer distance measuring sensor 211 is a distance measuring sensor for detecting the open/closed state of the drawer 16A. Note that the open state of the drawer 16A means that the drawer 16A is pulled out from the corresponding drawer-type accommodation chamber. Further, when the drawer 16A is in the closed state, it means that the drawer 16A is housed in the corresponding drawer type housing chamber. The drawer distance measuring sensor 211 detects the distance from the drawer 16A as the open/closed state of the drawer 16A.
In this embodiment, the drawer ranging sensor 211 detects the open/closed state of each of the drawers 14A and 15A.

The camera unit 2 has a human sensor 212 . The human sensor 212 outputs the detection value to the camera unit control section 20 . A human sensor 212 is provided at a predetermined position of the photographing member 202 .
The human sensor 212 corresponds to an example of the "sensor" of the present disclosure.

The arrangement order and installation positions of the sensors and cameras in FIGS. 4 and 5 are shown for convenience of explanation of the camera unit 2. 5 are not limited to the arrangement order and installation position.

FIG. 6 is a block diagram showing the configuration of the camera unit 2. As shown in FIG.
The camera unit 2 includes a camera unit control section 20 , a camera unit communication section 21 , a sensor section 22 , an imaging section 23 and a power supply section 24 .

The camera unit control section 20 includes a camera unit processor 220 such as a CPU (Central Processing Unit) or an MPU (Micro-processing unit), and a camera unit storage section 230 . The camera unit control section 20 controls each section of the camera unit 2 by having the camera unit processor 220 read out and execute a control program stored in the camera unit storage section 230 . The camera unit processor 220 executes a control program stored in the camera unit storage unit 230 to control a camera unit communication control unit 221, a detection value processing unit 222, an imaging control unit 223, a recording control unit 224, and a food detection unit. 225.

The camera unit storage section 230 includes a memory that stores programs executed by the camera unit processor 220 and data processed by the camera unit processor 220 . The camera unit storage section 230 stores a control program executed by the camera unit processor 220, a camera unit ID 4112 that is identification information of the camera unit 2, and various other data. The camera unit storage section 230 has a non-volatile storage area. In addition, the camera unit storage section 230 may include a volatile storage area and configure a work area for the camera unit processor 220 .

The camera unit communication section 21 is a communication interface provided with communication circuits, connectors, and the like conforming to a predetermined communication standard, and communicates with the food management server 4 and the terminal device 3 under the control of the camera unit control section 20 . In this embodiment, the communication standard used by the camera unit communication section 21 is exemplified by the wireless communication standard, but may be the wired communication standard.

The sensor unit 22 includes a left door ranging sensor 209, a right door ranging sensor 210, a drawer ranging sensor 211, and a motion sensor 212, and outputs detection values to the camera unit control unit 20 for each sensor. .

The photographing unit 23 includes a refrigerator compartment camera 203 and a drawer camera 204 . The imaging unit 23 outputs the imaging results of each of the refrigerator camera 203 and the drawer camera 204 to the camera unit control unit 20 . The refrigerator camera 203 and the drawer camera 204 capture moving images.
Hereinafter, when the refrigerator camera 203 and the drawer camera 204 are not distinguished from each other, they will be referred to as a "camera" and denoted by "213".

The power supply section 24 includes hardware such as a power supply circuit, and supplies power to each section of the camera unit 2 . In this embodiment, the power cable 7 of the camera unit 2 is connected to the commercial AC power supply 8, and the power supply unit 24 supplies power to each of the camera units 2 based on the power supplied from the external commercial AC power supply 8. to power the

The supply source for supplying power to the power supply unit 24 is not limited to the external commercial AC power supply 8, and may be the refrigerator 1 or a battery. If the supply source is the former, the power supply unit 24 and the refrigerator 1 are provided with hardware such as a port that complies with the USB power supply standard. In the former case, the USB cable of the camera unit 2 is connected to the USB port of the refrigerator 1, and the refrigerator 1 supplies power to the power supply unit 24 via the USB cable. If the supply source is the latter, the camera unit 2 is equipped with a battery.

As described above, the camera unit control section 20 functions as a camera unit communication control section 221 , detection value processing section 222 , photography control section 223 , recording control section 224 and food detection section 225 .
The food detection unit 225 corresponds to an example of an article detection unit.

The camera unit communication control section 221 communicates with the food management server 4 via the camera unit communication section 21 .

The detection value processing unit 222 determines whether each of the left door 11A, the right door 11B, and the drawers 14A and 15A is open or closed based on the detection values output from the sensor unit 22 . When the detection value processing unit 222 determines that either the left door 11A or the right door 11B is open, the door open state information indicating that the door 11C is open is sent to the recording control unit 224 and food detection. Output to unit 225 . Further, when the detection value processing unit 222 determines that both the left door 11A and the right door 11B are closed, the detection value processing unit 222 transmits door closed state information indicating that the door 11C is closed to the recording control unit 224 and the food product. Output to the detection unit 225 . When the detection value processing unit 222 determines that one of the drawers 14A and 15A is open, the detection value processing unit 222 outputs drawer open state information indicating that one of the drawers 14A and 15A is open to the recording control unit 224 and the recording control unit 224. Output to the food detection unit 225 . Further, when the detection value processing unit 222 determines that both the drawers 14A and 15A are closed, the detection value processing unit 222 notifies the recording control unit 224 and the food detection unit of the drawer closed state information indicating that the drawers 14A and 15A are closed. Output to unit 225 .

Further, the detection value processing unit 222 determines whether or not there is a person around the refrigerator 1 based on the detection value of the human sensor 212 output from the sensor unit 22 . When the detection value processing unit 222 determines that a person exists around the refrigerator 1 , the detection value processing unit 222 outputs presence/absence information indicating that a person exists to the imaging control unit 223 . When the detection value processing unit 222 determines that no person exists around the refrigerator 1 , the detection value processing unit 222 outputs presence/absence information indicating that no person exists to the imaging control unit 223 .

When the detection value processing unit 222 outputs presence/absence information indicating the presence of a person, the photographing control unit 223 causes the refrigerator camera 203 and the drawer camera 204 to start photographing. When the detection value processing unit 222 outputs presence/absence information indicating that no person exists, the photographing control unit 223 causes the refrigerator camera 203 and the drawer camera 204 to finish photographing.

When the detection value processing unit 222 outputs the door open state information, the recording control unit 224 starts recording the photographed result of the refrigerator camera 203 . When the detection value processing unit 222 outputs the door closed state information, the recording control unit 224 ends the recording of the photographing result of the refrigerator camera 203 that has started. In addition, when the detection value processing unit 222 outputs the drawer open state information, the recording control unit 224 starts recording the photographed result of the drawer camera 204 . When the detection value processing unit 222 outputs the drawer closed state information, the recording control unit 224 ends the recording of the photographed result of the drawer camera 204 that has started. After finishing the recording, the recording control section 224 generates a moving image from the start to the end of the recording as one recording file 231 and stores it in the camera unit storage section 230 .

The food detection unit 225 detects food that has been put into or taken out of the refrigerator 1 based on the recorded file 231 stored in the camera unit storage unit 230 .

FIG. 7 is a block diagram showing configurations of the terminal device 3 and the food management server 4. As shown in FIG.

[1-1-3. Food management server configuration]
First, the configuration of the food management server 4 will be described.
The food management server 4 includes a server control section 40 and a server communication section 41 .

The server control unit 40 includes a server processor 400 such as a CPU or MPU, and a server storage unit 410 . The server control unit 40 controls each unit of the food management server 4 by causing the server processor 400 to read and execute a control program stored in the server storage unit 410 . The server processor 400 functions as a server communication control unit 401 and an information processing unit 402 by executing control programs stored in the server storage unit 410 .

The server storage unit 410 has a memory that stores programs executed by the server processor 400 and data processed by the server processor 400 . The server storage unit 410 stores a control program executed by the server processor 400, a refrigerator compartment food management database 411, a withdrawal food management database 412, and various other data. The server storage unit 410 has a non-volatile storage area. The server storage unit 410 may comprise a volatile storage area and constitute a work area of the server processor 400 .

FIG. 8 is a diagram showing an example of one record stored in the refrigerator compartment food management database 411. As shown in FIG.
The refrigerating compartment food management database 411 is a database that stores various types of information related to foods stored in the refrigerating compartment 11 .
In the following description, one record stored in the refrigerating compartment food management database 411 is referred to as a "refrigerating compartment food management record" and denoted by "RR".

Refrigerator compartment food management record RR includes account ID 4111, camera unit ID 4112, refrigerator model information 4113, image data in refrigerator compartment 4114, storage availability information 4115, food image data 4116, storage date and time information 4117, division area number 4118, memo It has input presence/absence information 4119 , memo information 4130 , deletion information 4131 , and deletion date/time information 4132 .

Account ID 4111 is identification information for identifying an account assigned to user P who uses food management application 311 .

Refrigerator model information 4113 is information indicating the model of refrigerator 1 used by user P indicated by account ID 4111 associated with the same record.

The refrigerating compartment interior image data 4114 is image data of an image showing the interior of the refrigerating compartment 11 of the refrigerator 1 . An image showing the interior of the refrigerator compartment 11 is hereinafter referred to as a "image inside the refrigerator compartment" and is denoted by "RG". The refrigerator interior image data 4114 indicates the refrigerator interior image RG corresponding to the model indicated by the associated refrigerator model information 4113 in the same record.
The refrigerator interior image RG corresponds to an example of the refrigerator interior image.

FIG. 9 is a diagram showing an example of the refrigerator interior image RG.
The refrigerating compartment interior image RG shown in FIG. 9 includes a left door pocket image RG1, a refrigerating compartment front image RG2, and a right door pocket image RG3. In the refrigerating compartment interior image RG shown in FIG. 9, a left door pocket image RG1, a refrigerating compartment front image RG2, and a right door pocket image RG3 are arranged in this order from the left in the figure. The left door pocket image RG1 is an image showing the door pocket 111 formed on the left door 11A. Refrigerating chamber front image RG2 is an image of refrigerating chamber 11 viewed from the front when left door 11A and right door 11B are open. The right door pocket image RG3 is an image showing the door pocket 111 formed on the right door 11B.

The image area of the refrigerator interior image RG is divided into a plurality of areas. Each of these areas is referred to as a "divided area" and is labeled with "BA". The refrigerator interior image RG shown in FIG. 9 is divided into 15 divided areas BA. A number different from each other is assigned to each of the divided areas BA. In the following description, this number is referred to as a division area number and is denoted by "4118". Note that FIG. 9 shows for convenience which division area number is assigned to which division area BA. Number 4118 is not displayed.

Returning to the description of FIG. 8, the entering/leaving information 4115 is information indicating whether or not there is a new entering/leaving.

The food image data 4116 is image data of food. In the present embodiment, the image indicated by the food image data 4116 is an image captured by camera 203 for refrigerator compartment.

The storage date/time information 4117 is information indicating the storage date/time when the food was stored in the refrigerator 1 .

The refrigerator compartment food management record RR has a divided area number 4118 for each food image data 4116 .

The memo input presence/absence information 4119 is information indicating whether or not there is memo input.

Memo information 4130 is information indicating an input memo.

The deletion information 4131 is information indicating whether or not the associated sub-record is a sub-record to be deleted from the refrigerator compartment food management record RR. A sub-record is a record stored in the refrigerator compartment food management record RR, and has food image data 4116 , storage date/time information 4117 , division area number 4118 , memo input presence/absence information 4119 , and memo information 4130 .

The deletion date/time information 4132 is information indicating the date/time when the corresponding sub-record was targeted for deletion. When the deletion information 4131 indicates an object to be deleted, the information processing section 402 (to be described later) stores the sub-record and the deletion information 4131 and the deletion date/time information 4132 associated with this sub-record at a predetermined timing in the refrigerator compartment food management record RR. delete from

FIG. 10 is a diagram showing one record stored in the withdrawal food management database 412. As shown in FIG.
The drawer food management database 412 is a database that stores various information related to the food in the drawer type accommodation room. Note that the withdrawal food management database 412 is a database related to the vegetable compartment 15 in the description with reference to FIGS. 23-25.
In the following description, one record stored in the food withdrawal management database 412 is referred to as a "food withdrawal management record" and is denoted by "HR".

The withdrawal food management record HR has account ID 4111 , camera unit ID 4112 , update date/time information 4121 , upper case information 4122 and lower case information 4124 .

The update date/time information 4121 is information indicating the date/time when the content of the food withdrawal management record HR was updated.

The upper case information 4122 includes upper case image data 4123 , food image data 4116 , memo input presence/absence information 4119 , and memo information 4130 .

The upper case image data 4123 is image data of the upper case image JCG, which is the image of the upper case JCA. The upper case image JCG is an image captured by the drawer camera 204 .
The upper case image JCG corresponds to an example of the first case image of the present disclosure.

In the present embodiment, the image indicated by the food image data 4116 included in the upper case information 4122 is an image extracted from the upper case image JCG.

The lower case information 4124 includes lower case image data 4125 , food image data 4116 , memo input presence/absence information 4119 , and memo information 4130 .

The lower case image data 4125 is image data of the lower case image GCG, which is the image of the lower case GCA. A lower case image GCG is an image captured by the drawer camera 204 .
The lower case image GCG corresponds to an example of the second case image of the present disclosure.
Hereinafter, when the upper case image JCG and the lower case image GCG are not distinguished from each other, they are referred to as a "case image" and denoted by "CG".

In the present embodiment, the image indicated by the food image data 4116 included in the lower case information 4124 is an image extracted from the lower case image GCG.

Returning to the description of FIG. 7, the server communication unit 41 is a communication interface having a configuration related to communication such as a radio circuit and an antenna according to a predetermined communication standard, and is connected to a device connected to the global network GN according to the predetermined communication standard. connect. In this embodiment, the server communication section 41 communicates with the camera unit 2 and the terminal device 3 .

As described above, the server control section 40 functions as the server communication control section 401 and the information processing section 402 .

The server communication control section 401 communicates with the camera unit 2 and the terminal device 3 via the server communication section 41 .

The information processing section 402 performs various types of information processing based on information received by the server communication control section 401 from the camera unit 2 and the terminal device 3 .

[1-1-4. Configuration of terminal device]
Next, a functional configuration of the terminal device 3 will be described.
The terminal device 3 includes a terminal control section 30 , a terminal communication section 31 and a touch panel 32 .
The touch panel 32 corresponds to an example of the display section.

The terminal control unit 30 includes a terminal processor 300 such as a CPU or MPU, and a terminal storage unit 310 . The terminal control unit 30 controls each unit of the terminal device 3 by causing the terminal processor 300 to read and execute a control program stored in the terminal storage unit 310 . A food management application 311 is installed in the terminal device 3 . Terminal processor 300 functions as application execution unit 301 by reading and executing food management application 311 .
Terminal processor 300 corresponds to an example of a computer of the present disclosure. Also, the application execution unit 301 corresponds to an example of the display control unit of the present disclosure.

The terminal storage unit 310 has a memory that stores programs executed by the terminal processor 300 and data processed by the terminal processor 300 . The terminal storage unit 310 stores a control program executed by the terminal processor 300, a food management application 311, an account ID 4111, and various other data. The terminal storage unit 310 has a nonvolatile storage area. In addition, the terminal storage unit 310 may include a volatile storage area and constitute a work area of the terminal processor 300 .

The terminal communication unit 31 is a communication interface having a configuration such as a radio circuit and an antenna according to a predetermined communication standard, and communicates with devices connected to the global network GN according to the predetermined communication standard. In this disclosure, the wireless communication standard is exemplified as the communication standard used by the terminal communication unit 31 .

The touch panel 32 includes a display panel such as a liquid crystal display panel, and a touch sensor provided over the display panel or integrally therewith.

As described above, the terminal control section 30 functions as the application execution section 301 .
The application execution unit 301 transmits and receives various information to and from the food management server 4 via the terminal communication unit 31 .

Further, the application execution unit 301 causes the touch panel 32 to display a user interface related to food management of the refrigerator 1 . In the following description, this user interface is referred to as "application UI" and denoted by "320". The application execution unit 301 displays the application UI 320 on the touch panel 32 to provide the user P with various information related to the food in the refrigerator 1 and receive various inputs related to the food in the refrigerator 1 from the user P.

[1-2. motion]
Next, the operation of each part related to food management system 1000 will be described.

[1-2-1. Installation of camera unit]
The camera unit 2 is separate from the refrigerator 1. - 特許庁Therefore, the camera unit 2 is installed by the user P on the upper surface 10B of the refrigerator 1 . As described above, the position regulating member 205 regulates the position of the camera unit 2 in the front-rear direction of the refrigerator 1 , so the user P adjusts the position of the refrigerator 1 in the left-right direction when installing the camera unit 2 .
A plurality of methods for adjusting the position of the camera unit 2 in the horizontal direction of the refrigerator 1 are exemplified below.

[1-2-1-1. First position adjustment]

In the first position adjustment, user P uses application UI 320 .
In the first position adjustment, the application executing unit 301 causes the touch panel 32 to display the application UI 320 of the adjustment screen GM1.

FIG. 11 is a diagram showing an example of the adjustment screen GM1.
The adjustment screen GM1 is provided with a shooting result display area HA1. The photographing result display area HA1 is a rectangular area, and is an area for displaying the photographing result of either the refrigerator camera 203 or the drawer camera 204 . A line L1 extending in the vertical direction is provided at substantially the center in the left-right direction in the figure in the imaging result display area HA1.

The adjustment screen GM1 includes adjustment method information J1. The adjustment method information J1 prompts the user to adjust the position so that the horizontal center of the refrigerator 1 in the photographed result display area HA1 is aligned with the line L1 provided in the photographed result display area HA1.

The user P installs the camera unit 2 in the refrigerator 1 according to the adjustment method indicated by the adjustment method information J1. As a result, the camera unit 2 allows each of the refrigerator compartment camera 203 and the drawer camera 204 to include the foods entering and leaving the refrigerator 1 in the photographing range. Further, the camera unit 2 can detect opening/closing of the corresponding door 11C or drawer 16A by each distance measuring sensor.

In the first position adjustment, the camera unit 2 transmits to the terminal device 3 the photographed result of either the refrigerator compartment camera 203 or the drawer camera 204 . In addition, in the first position adjustment, the camera unit 2 and the terminal device 3 directly communicate via the food management server 4 or by short-range wireless communication or the like.

The camera unit communication control section 221 of the camera unit 2 transmits the photographing result of either the refrigerator compartment camera 203 or the drawer camera 204 to the terminal device 3 . Then, the application executing unit 301 displays the received photographing result in the photographing result display area HA1.

[1-2-1-2. Second position adjustment]
In the second position adjustment, the user P uses the center position recording paper CY.

FIG. 12 is a diagram showing an example of the central position recording sheet CY.
The center position recording sheet CY is a sheet on which a center mark MR2 indicating the center position of the refrigerator 1 in the horizontal direction is recorded. In addition to the center mark MR2, a left edge mark MR1 indicating the left edge position of the refrigerator 1 in the left-right direction and a right edge mark MR3 indicating the right edge position of the refrigerator 1 in the left-right direction are recorded on the center position recording sheet CY.

The central position recording paper CY is packed together with the refrigerator 1 and delivered to the user P. Note that the camera unit 2 may or may not be included together with the central position recording sheet CY and the refrigerator 1 . The central position recording sheet CY packed together with the refrigerator 1 has a length corresponding to the width of the refrigerator 1 from the left end mark MR1 to the right end mark MR3.

In the second position adjustment, the user P uses the central position recording paper CY to grasp where the central position of the refrigerator 1 in the horizontal direction is. For example, the user P attaches the center position recording sheet CY to the refrigerator 1 so that the left end mark MR1 is at the left end of the refrigerator 1 and the right end mark MR3 is at the right end of the refrigerator 1, and the center mark MR2 is used as a mark to mark the left and right sides of the refrigerator 1. Find the center position in the direction. Then, the user P adjusts the position of the camera unit 2 so that the mark 207 of the position regulating member 205 of the camera unit 2 is positioned at the grasped central position.

The second positional adjustment enables the camera unit 2 to perform various detections in the same manner as the first positional adjustment.

[1-2-1-3. Third position adjustment]
In the third position adjustment, a mark indicating the central position of the refrigerator 1 in the horizontal direction is attached to a predetermined position of the refrigerator 1 . In the third position adjustment, the mark 207 of the position regulating member 205 and the mark attached to the refrigerator 1 are used.

In the third position adjustment, the user P moves the camera unit 2 so that the position of the mark on the refrigerator 1 and the position of the mark 207 on the position regulating member 205 match in the horizontal direction of the refrigerator 1. Adjust position.

By the third positional adjustment, the camera unit 2 can perform various detections in the same manner as the first positional adjustment.

Next, the operation of the food management system 1000 after the camera unit 2 is installed in the refrigerator 1 will be described.
[1-2-2. Operations related to recording]
First, the operation related to recording will be described.
13 is a flowchart FA showing the operation of the camera unit 2. FIG.

In the operation shown in FIG. 13 , the sensor section 22 outputs detection values of each sensor to the camera unit control section 20 . In the operation shown in FIG. 13 , the detection value processing unit 222 executes processing based on various detection values received from the sensor unit 22 .

Detection value processing unit 222 determines whether or not there is a person around refrigerator 1 based on the detection value of human sensor 212 (step SA1).

When the detection value processing unit 222 determines that a person exists around the refrigerator 1 (step SA1: YES), it outputs presence/absence information indicating the presence of a person to the imaging control unit 223 (step SA2).

When presence/absence information indicating the presence of a person is input from detection value processing unit 222, shooting control unit 223 causes refrigerator camera 203 and drawer camera 204 to start shooting (step SA3).

Next, the detection value processing unit 222 determines whether any of the left door 11A, the right door 11B, the drawers 14A and 15A is open based on the detection values of various sensors output from the sensor unit 22. (step SA4).

When the detection value processing unit 222 determines that none of the left door 11A, the right door 11B, and the drawers 14A and 15A is open (step SA4: NO), the process of step SA16 is performed.

When the detection value processing unit 222 determines that any one of the left door 11A, the right door 11B, and the drawers 14A and 15A is open (step SA4: YES), the detection value processing unit 222 determines whether the door 11C is open or whether the drawer 14A, 15A is opened (step SA5).

When the detection value processing unit 222 determines that the door 11C is open (step SA5: door), the detection value processing unit 222 outputs door open state information to the recording control unit 224 (step SA6).

Next, when the door open state information is input from the detection value processing unit 222, the recording control unit 224 starts recording the photographed result of the refrigerator camera 203 (step SA7).

Next, the detection value processing section 222 determines whether or not the door 11C is closed (step SA8).

When the detection value processing unit 222 determines that the door 11C is closed (step SA8: YES), it outputs door closed state information to the recording control unit 224 (step SA9).

Next, when the door closed state information is input from the detection value processing unit 222, the recording control unit 224 ends the recording of the imaging result of the refrigerator camera 203 (step SA10). As a result, one recording file 231 of the refrigerator camera 203 is generated in the camera unit storage section 230 .

Returning to the description of step SA5, when the detection value processing unit 222 determines that one of the drawers 14A and 15A is in the open state (step SA5: drawer), the detection value processing unit 222 outputs the drawer open state information to the recording control unit 224 ( Step SA11).

Next, when the drawer closed state information is input from the detection value processing unit 222, the recording control unit 224 starts recording the photographed result of the drawer camera 204 (step SA12).

Next, the detection value processing unit 222 determines whether or not one of the open drawers 14A and 15A is closed (step SA13).

When detection value processing unit 222 determines that one of open drawers 14A and 15A is closed (step SA13: YES), detection value processing unit 222 outputs drawer closed state information to recording control unit 224 (step SA14).

Next, when the drawer closed state information is input from the detection value processing unit 222, the recording control unit 224 ends the recording of the imaging result of the drawer camera 204 (step SA15). As a result, one recorded file 231 of the withdrawal camera 204 is generated in the camera unit storage section 230 .

Detected value processing unit 222 determines whether or not there is a person around refrigerator 1 (step SA16).

When the detection value processing unit 222 determines that there is a person around the refrigerator 1 (step SA16: YES), the processing after step SA4 is performed again.

On the other hand, when the detection value processing unit 222 determines that no person exists around the refrigerator 1 (step SA16: NO), it outputs presence/absence information indicating that no person exists to the photographing control unit 223 (step SA17).

When presence/absence information indicating that there is no person is input from detection value processing unit 222, shooting control unit 223 causes refrigerator camera 203 and drawer camera 204 to finish shooting (step SA18).

[1-2-3. Operation related to food detection]
Next, an operation related to food detection will be described.
FIG. 14 is a flow chart showing the operation of the food management system 1000. FIG.
In FIG. 14, a flowchart FB shows the operation of the camera unit 2, and a flowchart FC shows the operation of the food management server 4. As shown in FIG.

The food detection section 225 of the camera unit 2 determines whether or not the recorded file 231 is stored in the camera unit storage section 230 (step SB1).

When the food detection unit 225 determines that the recording file 231 is stored in the camera unit storage unit 230 (step SB1: YES), the stored recording file 231 is either the recording file 231 of the refrigerator camera 203 or the drawer. It is determined whether it is the recorded file 231 of the camera 204 (step SB2). The determination in step SB2 is based on the information output by the detection value processing section 222. FIG.

When the food detection unit 225 determines that the food is the recorded file 231 of the refrigerator camera 203 (step SB2: refrigerator camera), the food detection unit 225 detects the food based on the stored recorded file 231 (step SB3).

At step SB3, for example, the food detection unit 225 detects food from the video recording file 231 of the refrigerator camera 203 based on the feature amount such as shape and color. Data necessary for food detection is stored in a storage area that can be acquired by the food detection unit 225 .

Food detection unit 225 determines whether or not food has been detected from recorded file 231 of refrigerator camera 203 (step SB4).

When food detection unit 225 determines that food cannot be detected from recorded file 231 of refrigerator camera 203 (step SB4: NO), the process proceeds to step SB10.

On the other hand, when the food detection unit 225 can detect the food from the recorded file 231 of the refrigerator camera 203 (step SB4: YES), the food detection unit 225 performs the entering/leaving determination process (step SB5).

The entering/leaving determination process is a process for determining whether the food detected in step SB3 is the food stored in the refrigerator 1 or the food taken out from the refrigerator 1. FIG. In the warehousing/leaving determination process, the warehousing/leaving determination image HG is superimposed on the moving image indicated by the recording file 231 of the refrigerator camera 203 .

FIG. 15 is a diagram for explaining the entering/leaving determination image HG.
The entering/leaving judgment image HG shown in FIG. 15 is superimposed on the moving image in which the user P places the egg pack into the refrigerator 1 .

The entering/leaving determination image HG is provided with a boundary line L2.
Boundary line L2 has boundary lines L21, L22, L23, L24, and L25. The boundary line L21 is a line that extends rightward from the left end of the entering/leaving determination image HG in the figure at the upper part of the entering/leaving determination image HG. The boundary line L22 is a line connecting the right end of the boundary line L21 and the left end of the boundary line L23. The boundary line L23 is a line that extends rightward from the position in the horizontal direction of the right end of the boundary line L21 at the approximate center in the vertical direction of the entering/leaving determination image HG. The boundary line L24 is a line connecting the right end of the boundary line L23 and the left end of the boundary line L24. The boundary line L25 is a line extending rightward from the right end of the boundary line L23 in the left-right direction to the right end of the entering/leaving determination image HG in the upper portion of the entering/leaving determination image HG.

The image area of the entering/leaving determination image HG is divided by a boundary line L2 into a concave area HG1 that is a concave area and a convex area HG2 that is a convex area.

The food detection unit 225 performs image analysis on the moving image on which the entering/leaving determination image HG is superimposed to determine whether the food detected in step SB3 has progressed from the convex region HG2 to the concave region HG1, or has moved from the concave region HG1. It is determined whether or not the state has progressed to the shape region HG2. When the food detection unit 225 determines that the food detected in step SB3 has progressed from the convex region HG2 to the concave region HG1, the food detection unit 225 determines that the food detected in step SB3 has entered the refrigerator 1 . On the other hand, when the food detection unit 225 determines that the food detected in step SB3 has progressed from the concave region HG1 to the convex region HG2, the food detection unit 225 determines that the food detected in step SB3 has left the refrigerator 1 .

Returning to the description of the flowchart FB in FIG. 14, the food detection unit 225 performs division area number identification processing (step SB6).

The division area number specifying process is a process of specifying the division area BA corresponding to the accommodation area where the food detected in step SB3 was received and delivered, and identifying the division area number 4118 assigned to the identified division area BA.

For example, the food detection unit 225 identifies, by image analysis, the housing area of the refrigerator compartment 11 into which the food detected in step SB3 has been loaded/unloaded. Also, for example, a mark for identifying a storage area is attached to the tip of each shelf 112 and each door pocket 111, and the food detection unit 225 detects the position of this mark and the detected food in the moving image. Identify containment areas based on relationships. Next, the food detection unit 225 determines which divided area BA of the refrigerator interior image RG corresponds to the identified accommodation area, and identifies the divided area number 4118 assigned to the identified divided area BA. Note that the correspondence between the accommodation area, the divided area BA, and the divided area number 4118 is stored in the camera unit storage section 230 as data.

After performing the division area number identification process, the food detection unit 225 cuts out an image of each food detected in step SB3 from the recorded file 231 of the refrigerator camera 203 (step SB7). Hereinafter, the image of the food cut out from the result of photographing by the camera 213 is referred to as a "photographed food image" and is denoted by "FG".
The photographed food image FG corresponds to an example of the food image of the present disclosure.

Next, food detection unit 225 generates first update information (step SB8).

The first update information generated in step SB8 includes the camera unit ID 4112 of the camera unit storage section 230. FIG.
Further, the first update information generated in step SB8 includes date and time information indicating the date and time when the recording file 231 of the refrigerator camera 203 was generated.
The first update information generated in step SB8 includes the food image data 4116 indicating the photographed food image FG cut out in step SB7 and the determination result of the storage/delivery determination process for each food detected in step SB3. It includes the entering/leaving determination result information and the division area number 4118 specified in the division area number specifying process.

Next, the camera unit communication control section 221 transmits the first update information generated by the food detection section 225 to the food management server 4 (step SB9).

The food detection unit 225 deletes the recorded file 231 from the camera unit storage unit 230 (step SB10).

Returning to the description of step SB2, when the food detection unit 225 determines that the file is the recorded file 231 of the drawer camera 204 (step SB2: drawer camera), the upper part of the moving image of the recorded file 231 of the drawer camera 204 It is determined whether or not at least one of the case JCA and the lower case GCA has been drawn by a predetermined threshold or more (step SB11).

For example, the food detection unit 225 distinguishes the case CA in the moving image based on feature amounts such as shape and color. Then, the food detection unit 225 determines whether or not the upper case JCA and the lower case GCA have been pulled out of the refrigerator 1 by a predetermined threshold or more by image analysis of the recorded file 231 .
Further, for example, the upper case JCA and the lower case GCA each have a mark identifying the case CA at a predetermined position. Distinguish case CA. Then, the food detection unit 225 determines whether or not the refrigerator 1 has been pulled out by a predetermined threshold value or more by calculating the moving distance of this mark in the moving image.

When the food detection unit 225 determines that neither the upper case JCA nor the lower case GCA has been pulled out by the predetermined threshold value or more (step SB11: NO), the food detection unit 225 performs the process of step SB18.

On the other hand, when the food detection unit 225 determines that at least one of the upper case JCA and the lower case GCA has been pulled out by a predetermined threshold value or more (step SB11: YES), the food detection unit 225 cuts out the case image CG from the recorded file 231 (step SB12). .

In step SB12, the food detection unit 225 cuts out the upper case image JCG from the recording file 231 when only the upper case JCA is extracted by a predetermined threshold value or more.
In step SB12, the food detection unit 225 cuts out the lower case image GCG from the recording file 231 when only the lower case GCA is extracted by a predetermined threshold value or more.
In step SB12, the food detection unit 225 cuts out the upper case image JCG and the lower case image GCG from the recording file 231 when the upper case JCA and the lower case GCA are extracted by a predetermined threshold value or more.

Next, the food detection unit 225 detects food from the clipped case image CG (step SB13).

Next, the food detection unit 225 determines whether or not food has been detected from the case image CG (step SB14).

When the food detection unit 225 determines that the food cannot be detected (step SB14: NO), the process of step SB16 is performed.

On the other hand, when the food detection unit 225 determines that food has been detected (step SB14: YES), the food detection unit 225 cuts out a photographed food image FG for each detected food from the case image CG (step SB15).

Next, food detection unit 225 generates second update information (step SB16).

When only the upper case image JCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage section 230, the upper case image data 4123 indicating the cut out upper case image JCG, and the upper case image JCG. It includes food image data 4116 representing the photographed food image FG extracted from the case image JCG. If a negative determination is made in step SB14, food image data 4116 is not included in this second update information.

Further, when only the lower case image GCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage section 230, the lower case image data 4125 indicating the cut out lower case image GCG, and , food image data 4116 representing the photographed food image FG extracted from the lower case image GCG. If a negative determination is made in step SB14, food image data 4116 is not included in this second update information.

Further, when the upper case image JCG and the lower case image GCG are cut out, the second update information generated in step SB16 includes the camera unit ID 4112 in the camera unit storage section 230, the upper case image data group, and the lower case image Contains data groups. The upper case image data group includes upper case image data 4123 representing the clipped upper case image JCG and food image data 4116 representing the photographed food image FG clipped from the upper case image JCG. The lower case image data group includes lower case image data 4125 representing the cut-out lower case image GCG and food image data 4116 representing the photographed food image FG cut out from the lower case image GCG. If a negative determination is made in step SB14, food image data 4116 is not included in this second update information.

Next, the camera unit communication control section 221 transmits the second update information generated by the food detection section 225 to the food management server 4 (step SB17).

Next, the food detection section 225 deletes the recorded file 231 from the camera unit storage section 230 (step SB18).

As shown in flowchart FC, the server communication control unit 401 of the food management server 4 determines whether or not update information of either the first update information or the second update information has been received (step SC1).

When it is determined that the server communication control unit 401 has received update information (step SC1: YES), the information processing unit 402 updates the database based on the update information received by the server communication control unit 401 (step SC2).

Step SC2 will be described in detail.
When the update information received by the server communication control unit 401 is the first update information, the information processing unit 402 identifies the refrigerator compartment food management record RR having the camera unit ID 4112 included in the first update information. Then, the information processing unit 402 updates the entry/exit presence/absence information 4115 of the specified refrigerating compartment food management record RR to information indicating "present".

Next, when the warehousing/dispatching determination result information included in the first update information indicates warehousing, the information processing unit 402 generates a subrecord, and associates blank deletion information 4131 and deletion date/time information 4132 with the generated subrecord. , is stored in the identified refrigerator compartment food management record RR. The sub-records generated here are food image data 4116 included in the first update information, storage date/time information 4117 indicating the same date and time as the date/time information included in the first update information, divided area number 4118 included in the first update information, and blanks. memo input existence information 4119 and blank memo information 4130 .

Further, when the entering/leaving determination result information included in the first update information indicates delivery, the information processing unit 402 determines the photographed food image FG indicated by the food image data 4116 included in the first update information and the photographed food image having a high degree of matching. A sub-record containing food image data 4116 indicating FG is identified from the refrigerator compartment food management record RR. This identification is performed for each food image data 4116 included in the first update information. Then, the information processing section 402 updates the deletion information 4131 associated with the specified sub-record to information indicating that the sub-record is to be deleted. The information processing unit 402 also updates the deletion date/time information 4132 associated with the identified sub-record to information indicating the current date/time as the deletion date/time.

Further, step SC2 will be described in detail.
When the update information received by the server communication control unit 401 is the second update information, the information processing unit 402 identifies the withdrawal food management record HR having the camera unit ID 4112 included in the second update information.

In the case of the second update information when only the upper case image JCG is cut out, the information processing unit 402 converts the upper case image data 4123 and the food image data 4116 included in the upper case information 4122 of the identified food management record HR to It updates to the data which the 2nd update information has.
In the case of the second update information when only the lower case image GCG is cut out, the information processing unit 402 converts the lower case image data 4125 and the food image data 4116 included in the lower case information 4124 of the identified food management record HR to It updates to the data which the 2nd update information has.
If the upper case image JCG and the lower case image GCG are the second update information when they are cut out, the information processing unit 402 processes the upper case image data 4123 and the food The image data 4116 is appropriately updated to the data contained in the upper case image data group. In addition, the information processing section 402 updates the lower case image data 4125 and the food image data 4116 included in the lower case information 4124 of the specified withdrawn food management record HR to the data included in the lower case image data group.

In addition, the information processing section 402 updates the update date/time information 4121 of the specified withdrawal food management record HR to information indicating the date and time when the withdrawal food management record HR was updated.

[1-2-4. Operation related to application UI]
Next, operations related to the application UI 320 will be described.
In the present embodiment, the application UI 320 displays a screen related to the refrigerator compartment 11 and a screen related to the drawer-type storage compartment.

[1-2-4-1. Screen related to the refrigerator compartment]

The screens related to the refrigerator 11 include a refrigerator home screen GM2, a refrigerator list screen GM3, and a food information addition list screen GM4.

FIG. 16 is a diagram showing an example of the refrigerator home screen GM2.

The refrigerator home screen GM2 has a food image display area HA2.
The food image display area HA2 is an area for displaying the photographed food image FG. The food image display area HA2 displays a refrigerator interior image RG, and displays a photographed food image FG superimposed on the refrigerator interior image RG. Each of the photographed food images FG is superimposed and displayed in the divided area BA indicated by the divided area number 4118 corresponding to the food image data 4116 in the refrigerator interior image RG.

The food image display area HA2 in FIG. 16 displays the refrigerator interior image RG shown in FIG. For example, in the refrigerator compartment food management record RR, it is assumed that the food image data 4116 representing the photographed food image FG1 in FIG. 16 is associated with the divided area number 4118 of "4". In this case, the photographed food image FG1 is displayed superimposed on the divided area BA to which the divided area number 4118 of "4" is assigned.

The default food image display area HA2 displays the refrigerator interior image RG in a state where the left door pocket image RG1 and the right door pocket image RG3 are cut off. In the food image display area HA2, a scroll operation in the horizontal direction of the drawing is accepted from the user P. FIG. The food image display area HA2 changes the display range of the refrigerator interior image RG according to the scroll amount and scroll direction of the received scroll operation.

For example, assume that the scroll direction is to the right and the scroll amount is such that the left door pocket image RG1 is displayed up to the left end. In this case, the food image display area HA2 displays the refrigerator interior image RG in which the right door pocket image RG3 is completely cut off. The photographed food image FG and the memo input icon MIC follow the change in the display range of the refrigerator interior image RG, in other words, move along with the movement of the superimposed divided area BA. The memo input icon MIC will be described later.

In the food image display area HA2, when displaying a plurality of photographed food images FG in one divided area BA, they are displayed side by side in the horizontal direction of the figure. The food image display area HA2 can simultaneously display up to five photographed food images FG side by side in one divided area BA. The food image display area HA2 receives a scroll operation from the user P when there are six or more photographed food images FG to be displayed in one divided area BA. A divided area BA of 1 displays photographed food images FG after the sixth sheet when a scroll operation is received. In the food image display area HA2, when there are six or more photographed food images FG to be displayed in one divided area BA and a horizontal scrolling operation is accepted, the maximum number of images that can be displayed side by side is maintained. The photographed food images FG after the sixth one are displayed according to the amount of scrolling. At this time, the food image display area HA2 hides the first to fifth displayed photographed food images FG according to the amount of scrolling and the number of photographed food images FG to be displayed. After the sixth and subsequent photographed food images FG, the first and subsequent photographed food images FG are displayed. Note that the memo input icon MIC moves following the movement of the associated photographed food image FG. That is, when the associated photographed food image FG is hidden by the scrolling operation, the memo input icon MIC is also hidden.

Each photographed food image FG displayed in the food image display area HA2 accepts input of a memo. The memo is information that the user P wants to leave for the food in the refrigerator 1 .

FIG. 17 is a diagram showing a memo input procedure.

The user P performs a predetermined touch operation such as a long press operation on the photographed food image FG for which he/she wishes to enter a memo on the refrigerating room home screen GM2. When receiving a predetermined touch operation on one photographed food image FG, the application executing unit 301 displays a memo edited image MHG in association with the photographed food image FG on which the predetermined touch operation is performed.

The memo edit image MHG has a memo input area MNA, a memo save button B1, a memo delete button B2, and a food delete button B3.

The memo input area MNA is an area for inputting a memo. Application execution unit 301 displays a software keyboard on touch panel 32 when memo input area MNA is touch-operated. Then, application execution unit 301 inputs a memo corresponding to the operation on the software keyboard into memo input area MNA.

The memo save button B1 is a software button for saving a memo input in the memo input area MNA. The memo save button B1 cannot be touch-operated when no memo is input in the memo input area MNA. When the memo save button B1 is touch-operated, the application executing unit 301 displays the photographed food image FG with the memo input in the display mode of the photographed food image FG without the memo input on the screen related to the refrigerator compartment 11. Display in a different display mode. In the present embodiment, the application execution unit 301 displays the frame of the photographed food image FG to which the memo has been input in a different color, and indicates that the memo has been input to the photographed food image FG to which the memo has been input. A memo input icon MIC, which is an icon shown in FIG. The memo input icon MIC can be touch-operated. When the memo input icon MIC is touch-operated, the input memo is displayed on the application UI 320 .
The memo input icon MIC corresponds to an example of the memo input image of the present disclosure.

The memo erase button B2 is a button for erasing the input memo. When the memo erase button B2 is touch-operated, the memo input to the photographed food image FG is erased.

The food delete button B3 is a button for deleting the photographed food image FG from the food image display area HA2.

Returning to the description of the refrigerator compartment home screen GM2 in FIG. 16, the food image display area HA2 has a list display button B4. The list display button B4 is a touch-operable software button.

When the list display button B4 is touch-operated, the application UI 320 transitions to the refrigerating room list screen GM3.

The refrigerating room list screen GM3 includes an accommodation list screen GM31, a new addition list screen GM32, and a deletion target list screen GM33.

The accommodation list screen GM31 will be described.
FIG. 18 is a diagram showing the accommodation list screen GM31.

The contained list screen GM31 has a contained food information display area HA3. The stored food information display area HA3 displays a list of the first food information J21 for each food stored in the refrigerator compartment 11 . The first food information J21 includes a photographed food image FG, storage date/time information 4117, and divided area information J3. The divided area information J3 is a refrigerator interior image RG in which the divided area BA is indicated by color.

When a memo is entered in the photographed food image FG of the first food information J21, the frame is displayed in a different color, and the memo input icon MIC is displayed in association with it. It should be noted that, in the stored food information display area HA3, it is possible to perform a scrolling operation in the vertical direction of the figure. The user P can display the first food information J21 that cannot be displayed in the stored food information display area HA3 by performing a scroll operation in the stored food information display area HA3.

The accommodation list screen GM31 has a first switching button B5 and a second switching button B6.

The first switching button B5 is a software button for switching the arrangement order of the first food information J21 displayed in the contained food information display area HA3 to the next order.
That is, when the first switching button B5 is touch-operated, the stored food information display area HA3 changes the first food information J21 having the photographed food image FG with the memo input to the photographed food image FG without the memo input. is displayed higher than the first food information J21 having It should be noted that displaying in a higher position means displaying so that the amount of scrolling when displaying is reduced. When there are a plurality of pieces of first food information J21 having photographed food images FG to which memos have been input, the stored food information display area HA3 displays the food information J2 with the older storage date and time indicated by the storage date and time information 4117 at a higher level. do. Further, when there are a plurality of pieces of first food information J21 having photographed food images FG with no memo input, the stored food information display area HA3 displays the first food information J21 with the older storage date and time indicated by the storage date and time information 4117 as a higher rank food item. to display.
Note that the accommodation list screen GM31 on the left side of FIG. 18 shows the accommodation list screen GM31 when the first switching button B5 is touch-operated.

The second switching button B6 is a software button for switching the arrangement order of the first food information J21 displayed in the contained food information display area HA3 to the next order.
That is, when the second switching button B6 is touch-operated, the stored food information display area HA3 displays the first food information J21 side by side for each divided area BA. The order in which the divided areas BA are arranged may be an order predetermined by the user P. The stored food information display area HA3 displays the first food information J21 in one divided area BA in the same order as when the first switching button B5 is touch-operated.
Note that the accommodation list screen GM31 on the right side of FIG. 18 shows the accommodation list screen GM31 when the second switching button B6 is touch-operated.

The accommodation list screen GM31 has an accommodation list display button B7, a new addition list display button B8, and a deletion target list display button B9.

The display accommodation list button B7 is a software button for switching the screen displayed by the application UI320 to the accommodation list screen GM31.
The new addition list display button B8 is a software button for switching the screen displayed by the application UI 320 to the new addition list screen GM32.
The deletion target list display button B9 is a software button for switching the screen displayed by the application UI 320 to the deletion target list screen GM33.

FIG. 19 is a diagram showing an example of the new addition list screen GM32.
The newly added list screen GM32 has a newly added food information display area HA4. The newly added food information display area HA4 displays a list of the first food information J21 related to the new food stored in the refrigerator compartment 11 . The food newly stored in the refrigerating compartment 11 means the food stored within a predetermined period (for example, three days) from the current date and time. The newly added food information display area HA4 displays the first food information J21 of the food corresponding to the newly accommodated food in order of newest accommodation date and time indicated by the accommodation date and time information 4117. FIG.

Each piece of first food information J21 displayed in the newly added food information display area HA4 receives a predetermined touch operation on the photographed food image FG. On the newly added list screen GM32, the first edited image HNG1 is displayed in association with the photographed food image FG subjected to the predetermined touch operation.

The first edited image HNG1 has a food item deletion button B10, an area correction button B11, and a return button B12.
The food delete button B10 is a software button for deleting the food indicated by the photographed food image FG associated with the first edited image HNG1 from the refrigerator compartment home screen GM2 when touch-operated.
The area correction button B11 is a software button for correcting the division area BA indicated by the division area information J3 when touch-operated.
The return button B12 is a software button for stopping the display of the first edited image HNG1.

The newly added food information display area HA4 has a food information addition button B13 at the end of the list of the first food information J21. The food information addition button B13 is a software button for manually adding new first food information J21.
A procedure for adding the first food information J21 will be described with reference to FIG.

FIG. 20 is a diagram for explaining the procedure for adding the first food information J21.

When the food information addition button B13 is touch-operated, the food information addition list screen GM4 is displayed on the application UI320. The food information addition list screen GM4 has a plurality of add buttons B14, which are software buttons. The food information addition list screen GM4 also has a blank image BG and a food name input area SNA for inputting food names. When the blank image BG is touch-operated, the blank image BG becomes an image showing the food name input in the food name input area SNA.

When the add button B<b>14 is touch-operated on the food information addition list screen GM<b>4 , the refrigerator interior image RG is displayed on the application UI<b>320 . Here, the displayed refrigerator interior image RG is an image in which each of the divided areas BA is assigned a divided area number 4118, and each of the divided areas BA can be selected by a touch operation. When the divided area BA is selected in the refrigerator interior image RG, the following first food information J21 is added to the application UI320.

That is, the first food information J21 to be added includes the image of the food corresponding to the touch-operated add button B14, the storage date and time information 4117 indicating the date and time when the first food information J21 was added as the storage date and time, and the selected division. This is divided area information J3 indicating area BA. The food image indicated by the added first food information J21 may be an icon. In the present embodiment, it is assumed that the image superimposed on the refrigerator interior image RG of the refrigerator home screen GM2 is the shadow food image FG. An icon is displayed in the indoor image RG together with the photographed food image FG.

In addition, in the procedure for adding the first food information J21, there may be a procedure for the user P to input the storage date and time. In this case, the first food information J21 to be added includes accommodation date/time information 4117 indicating the accommodation date/time input by the user P. FIG.

Returning to the description of FIG. 19, the new addition list screen GM32 has an accommodation list display button B7, a new addition list display button B8, and a deletion target list display button B9.

FIG. 21 is a diagram showing an example of the deletion target list screen GM33.

The deletion target list screen GM33 has a deletion target food information display area HA5. The deletion target food information display area HA5 lists the second food information J22 for each deletion target food. The food to be deleted indicates the food to be deleted from the refrigerating compartment home screen GM2 and the food determined to have been taken out of the refrigerating compartment 11 . The second food information J22 includes a photographed food image FG, deletion date/time information 4132, and divided area information J3. The deletion target list screen GM33 displays the second food information J22 in descending order of deletion date and time.

Each of the second food information J22 displayed in the deletion target food information display area HA5 receives a predetermined touch operation by the photographed food image FG. On the deletion target list screen GM33, the second edited image HNG2 is displayed in association with the photographed food image FG subjected to the predetermined touch operation.

The second edited image HNG2 has a delete cancel button B15 and a return button B16.
The delete cancel button B15 is a software button for making the photographed food image FG associated with the second edited image HNG2 an image displayed on the refrigerator compartment home screen GM2.
The return button B16 is a software button for stopping the display of the second edited image HNG2.

[1-2-4-2. Operation of food management system]
Next, the operation of the food management system 1000 when displaying a screen related to the refrigerator compartment 11 will be described.

FIG. 22 is a flow chart showing the operation of the food management system 1000. FIG. In FIG. 22, a flow chart FD shows the operation of the terminal device 3, and a flow chart FE shows the operation of the food management server 4. FIG.

The application execution unit 301 of the terminal device 3 determines whether or not to request the refrigerator compartment food management record RR from the food management server 4 (step SD1).

For example, when the application UI 320 receives a touch operation instructing display of the refrigerator compartment home screen GM2, the application execution unit 301 makes an affirmative determination in step SD1.
Further, for example, when a touch operation instructing display update is performed on the refrigerator compartment home screen GM2, the application execution unit 301 makes an affirmative determination in step SD1.

When the application execution unit 301 determines to request the refrigerator management record RR (step SD1: YES), the application execution unit 301 transmits first record request information requesting the refrigerator management record RR to the food management server 4 (step SD2). . Account ID 4111 stored in terminal storage unit 310 is included in the first record request information transmitted in step SD2.

As shown in flowchart FF, the server communication control unit 401 of the food management server 4 receives the first record request information through the server communication unit 41 (step SE1).

Next, information processing section 402 identifies refrigerator food management record RR including account ID 4111 included in the first record request information received by server communication control section 401 from refrigerator compartment food management database 411 (step SE2).

Then, server communication control section 401 transmits refrigerator compartment food management record RR specified by information processing section 402 to terminal device 3 (step SE3). In addition, the information processing unit 402 updates the incoming/outgoing information 4115 of the transmitted refrigerating compartment food management record RR to information indicating "none".

Referring to flowchart FD, upon receiving the refrigerator compartment food management record RR from the food management server 4 (step SD3), the application execution unit 301 of the terminal device 3 executes the refrigerator compartment 11 based on the refrigerator compartment food management record RR. is generated (step SD4).

Step SD4 will be described in detail.
Application execution unit 301 generates refrigerator home screen GM2 that displays refrigerator interior image RG indicated by refrigerator interior image data 4114 included in refrigerator food management record RR in food image display area HA2.
Application execution unit 301 also generates refrigerator home screen GM2 in which photographed food image FG indicated by food management image data 4116 is superimposed on divided area BA indicated by divided area number 4118 for each subrecord.
Further, when memo input presence/absence information 4119 indicates that there is a memo, application execution unit 301 generates refrigerator home screen GM2 displaying memo input icon MIC in association with photographed food image FG for each subrecord.

Furthermore, step SD4 will be described in detail.
Application execution unit 301 generates first food information J21 for each food image data 4116 . This first food information J21 includes the photographed food image FG indicated by the food image data 4116, the storage date/time information 4117 associated with this food image data 4116, and the divided areas indicated by the color of the divided area BA indicated by the divided area number 4118. Contains information J3. Then, the application execution unit 301 generates an accommodation list screen GM31 that displays a list of the first food information J21 related to the sub-records associated with the deletion information 4131 that does not indicate deletion targets. In addition, the application executing unit 301 displays a list of the first food information J21 that the storage date and time indicated by the storage date and time information 4117 indicates a date and time within a predetermined period from the current time, among the first food information J21 displayed on the storage list screen GM31. A new addition list screen GM32 is generated.

Furthermore, step SD4 will be described in detail.
Application execution unit 301 generates second food information J22 for each food image data 4116 . The second food information J22 includes the photographed food image FG indicated by the food image data 4116, the deletion date/time information 4132 associated with the food image data 4116, and the divided area information indicated by the color of the divided area BA indicated by the divided area number 4118. Including J3. Then, the application execution unit 301 generates a deletion target list screen GM33 that lists the generated second food information J22.

Returning to the description of the flowchart FD, the application execution unit 301 displays a screen related to the refrigerator compartment 11 according to the touch operation (step SD5). It should be noted that, if the entry/exit presence/absence information 4115 of the refrigerator compartment food management record RR received in step SD3 indicates "yes", when the refrigerator compartment home screen GM2 is displayed for the first time in step SD5, the application executing unit 301 displays the refrigerator compartment home screen GM2. The home screen GM2 pops up to the effect that there is a new entry/exit.

Next, the operation of the food management system 1000 when operated on the screen related to the refrigerator compartment 11 will be described.

When the memo save button B1 is touch-operated, the application execution unit 301 transmits to the food management server 4 information for updating the sub-record related to the photographed food image FG to be operated. This information includes memo input presence/absence information 4119 indicating that there is a memo, and memo information 4130 indicating the memo that has been input. The food management server 4 appropriately updates the subrecords based on the received information.

When the memo delete button B2 is touch-operated, the application execution unit 301 transmits to the food management server 4 information for updating the sub-record related to the photographed food image FG to be operated. This information includes memo input presence/absence information 4119 indicating that no memo has been input. The food management server 4 appropriately updates the subrecords based on the received information.

When the food deletion buttons B3 and B10 are touch-operated, the application execution unit 301 transmits to the food management server 4 information for updating the sub-record regarding the photographed food image FG to be operated. This information includes deletion information 4131 indicating deletion targets. The food management server 4 appropriately updates the subrecords based on the received information.

When the area correction button B11 is touch-operated and the division area BA is corrected, the application execution unit 301 transmits to the food management server 4 information for updating the sub-record related to the photographed food image FG to be operated. This information includes the division area number 4118 of the division area BA after correction. The food management server 4 appropriately updates the subrecords based on the received information.

When the first food information J21 is added by the addition procedure shown in FIG. 20, the application execution unit 301 transmits to the food management server 4 information for adding a subrecord related to the added first food information J21. The food management server 4 appropriately updates the subrecords based on the received information.

When the deletion cancellation button B15 is touch-operated, the application execution unit 301 transmits to the food management server 4 information for updating the sub-record related to the photographed food image FG to be operated. This information includes deletion information 4131 indicating blank. The food management server 4 appropriately updates the subrecords based on the received information.

[1-2-4-3. Screen related to the drawer-type containment room]
Next, the screen related to the drawer-type accommodation room will be described. In the explanation using FIGS. 23 to 25, the vegetable compartment 15 is exemplified as the drawer type storage compartment. In this description, the database shown in FIG. 10 is a database that stores various information related to food in the vegetable compartment 15 .

Screens related to the vegetable compartment 15 include a vegetable compartment home screen GM5 and a food image list screen GM6.

FIG. 23 is a diagram showing an example of the vegetable compartment home screen GM5.

The vegetable room home screen GM5 has a case image display area HA6.
The case image display area HA6 is an area for displaying the case image CG. The case image display area HA6 displays the upper case image JCG superimposed on the lower case image GCG. In the default case image display area HA6, the upper case image JCG is superimposed on the entire image area of the lower case image GCG, and only the upper case image JCG is displayed.

The case image display area HA6 accepts a scroll operation in the vertical direction of the figure. The case image display area HA6 changes the display range of the upper case image JCG according to the scroll amount and scroll direction of the accepted scroll operation. For example, when the scroll direction of the accepted scroll operation is upward, the case image display area HA6 displays the upper case image JCG partially cut off. In the case image display area HA6, when the upper case image JCG moves upward, the lower case image GCG superimposed on the upper case image JCG is displayed.

Each of the case images CG displayed in the case image display area HA6 accepts memo input. Note that the memo input procedure is the same as in the refrigerator compartment 11, except that the image operated by the user P is the photographed food image FG or the case image CG. Note that, as shown in FIG. 23, the memo input icon MIC associated with the upper case image JCG moves following the movement of the upper case image JCG.

The vegetable room home screen GM5 has a list display button B17. The list display button B17 is a touch-operable software button.

When the list display button B17 is touch-operated, the food image list screen GM6 is displayed on the application UI320.

The vegetable room home screen GM5 may have a switching button for switching the case image CG displayed in the case image display area HA6 to the upper case image JCG or the lower case image GCG. In this configuration, the application execution unit 301 switches the case image CG to be displayed to the upper case image JCG or the lower case image GCG in accordance with the operation of the switch button.

FIG. 24 is a diagram showing an example of the food image list screen GM6.

The food image list screen GM6 has update date/time information 4121 . The food image list screen GM6 in FIG. 24 has update date/time information 4121 indicating the update date/time of “12/1 10:05”.

The food image list screen GM6 displays a list of photographed food images FG for each case. The photographed food image FG displayed in association with the item of the upper case JCA is an image cut from the upper case image JCG. The photographed food image FG displayed in association with the item of the lower case GCA is an image cut from the lower case image GCG.

[1-2-4-5. Operation of food management system]
Next, the operation of the food management system 1000 when displaying a screen related to the vegetable compartment 15 will be described.

FIG. 25 is a flow chart showing the operation of the food management system 1000. FIG. In FIG. 11, a flowchart FF shows the operation of the terminal device 3, and a flowchart FJ shows the operation of the food management server 4. FIG.

The application execution unit 301 of the terminal device 3 determines whether or not to request the withdrawal management record HR from the food management server 4 (step SF1).

For example, when the application UI 320 receives a touch operation instructing display of the vegetable room home screen GM5, the application execution unit 301 makes an affirmative determination in step SF1.
Further, for example, when a touch operation instructing update of the display is performed on the vegetable room home screen GM5, the application execution unit 301 makes an affirmative determination in step SF1.

When the application execution unit 301 determines to request the withdrawal management record RR (step SF1: YES), the application execution unit 301 transmits second record request information requesting the withdrawal management record HR to the food management server 4 (step SF2). Account ID 4111 stored in terminal storage unit 310 is included in the second record request information transmitted in step SF2.

Referring to flowchart FJ, server communication control unit 401 of food management server 4 receives the second record request information (step SJ1).

Next, the information processing section 402 identifies the withdrawal management record HR including the account ID 4111 included in the second record request information received by the server communication control section 401 from the withdrawal food management database 412 (step SJ2).

Then, the server communication control section 401 transmits the withdrawal management record HR specified by the information processing section 402 to the terminal device 3 (step SJ3).

Referring to flow chart FJ, when application execution unit 301 of terminal device 3 receives withdrawal food management record HR from food management server 4 (step SF3), it generates a screen related to vegetable compartment 15 (step SF4).

Step SF4 will be described in detail.
The application executing unit 301 displays a case image in which the upper case image JCG indicated by the upper case image data 4123 included in the same record is superimposed on the lower case image GCG indicated by the lower case image data 4125 included in the withdrawal food management record HR. A vegetable room home screen GM5 having an area HA6 is generated.
Further, when the memo input presence/absence information 4119 indicates that there is a memo, the application execution unit 301 generates the vegetable room home screen GM5 displaying the memo input icon MIC in association with the corresponding case image CG.

Further, step SF4 will be described in detail.
Application execution unit 301 generates food image list screen GM6 in which photographed food images FG indicated by food image data 4116 are arranged for each case CA.

Returning to the description of the flowchart FF, the application execution unit 301 displays a screen related to the vegetable compartment 15 according to the touch operation (step SF5).

Next, the operation of the food management system 1000 when operated on the screen related to the vegetable compartment 15 will be described.

When memo save button B1 is touch-operated, application execution unit 301 transmits information for updating memo input presence/absence information 4119 and memo information 4130 to food management server 4 . This information includes information indicating whether the operation target is the upper case image JCG or the lower case image GCG, memo input presence/absence information 4119 indicating that there is a memo, and memo information 4130 indicating the input memo. The food management server 4 appropriately updates the withdrawal food management record HR based on the received information.

When the memo erase button B2 is touch-operated, the application execution unit 301 transmits information for updating the memo input presence/absence information 4119 and the memo information 4130 to the food management server 4 . This information includes information indicating whether the memo is to be erased from the upper case image JCG or the lower case image GCG, and memo input presence/absence information 4119 indicating that there is no memo. The food management server 4 appropriately updates the withdrawal food management record HR based on the received information.

Note that the vegetable compartment 15 is exemplified as the drawer-type storage compartment described with reference to FIGS. 23-25. However, the screen displays and operations of food management system 1000 shown in FIGS. In this case, the database shown in FIG. 10 is a database that stores various information related to food in the freezer compartment 14 .

[1-3. effect]
As described above, the camera unit 2 includes the main body 201 installed on the upper surface 10B of the refrigerator 1 and the camera 213 extending from the main body 201 to the front of the refrigerator 1 and capturing an image of the refrigerator 1 from the upper front to the lower. and a photographing member 202 provided. The main body 201 includes a position regulating member 205 that contacts the front edge 10A of the refrigerator 1 and regulates the position of the photographing member 202 in the front-rear direction of the refrigerator 1 .

According to this, the position of the photographing member 202 in the front-rear direction of the refrigerator 1 is regulated by the position regulating member 205 , so that the camera 213 is regulated to a position where the refrigerator 1 can be photographed from the front upper side to the lower side. Therefore, it is possible to accurately include the food that is taken in and out of the refrigerator 1 in the photographed result of the camera 213, so that the food that is taken in and out of the refrigerator 1 can be photographed with high accuracy.

The position regulating member 205 is a position regulating piece that extends downward from the bottom surface 206 of the main body 201 in the installed state.

According to this, the position of the camera unit 2 can be regulated by using the gap formed between the main housing 10 and the door 11C. Therefore, since the camera unit 2 can be installed in the existing refrigerator 1, even in the existing refrigerator 1, the food entering and leaving the refrigerator 1 can be photographed with high accuracy.

The position regulating member 205 is formed shorter than the length to the packing provided between the refrigerator 1 and the door 11</b>C of the refrigerator 1 .

According to this, the position of the camera unit 2 can be regulated using the gap formed between the main box body 10 and the door 11C without impairing the cooling effect of the refrigerator compartment 11 . Therefore, the food entering and leaving the refrigerator 1 can be photographed with high accuracy without impairing the cooling effect of the refrigerator compartment 11 .

The position restricting member 205 extends in the width direction of the main body 201 .

According to this, since a large area can be secured to contact the main box body 10, the installation posture of the camera unit 2 can be stabilized. Therefore, it is possible to photograph the food entering and leaving the refrigerator 1 with higher accuracy.

Position regulating member 205 has marks 207 for adjusting the position of camera unit 2 in the width direction of refrigerator 1 .

According to this, the user P can easily install the camera unit 2 in the horizontal direction of the refrigerator 1 by using the mark 207 as a mark. Therefore, the user P can easily install the camera unit 2 so as to accurately photograph the food entering and leaving the refrigerator 1 .

The position regulating member 205 has a mark 207 at the widthwise center of the main body 201 .

According to this, by using the mark 207 as a mark, the user P can easily install the camera unit 2 in the center of the refrigerator 1 in the horizontal direction. Therefore, the user P can easily install the camera unit 2 so that the food entering and leaving the refrigerator 1 can be photographed with higher accuracy.

In addition, as described above, the camera unit 2 includes a camera 213 that captures a moving image of the refrigerator 1 from above the front of the refrigerator 1 and a motion sensor 212 that detects whether or not a person is present around the refrigerator 1. When it is detected that a person is present around the refrigerator 1, a photographing control unit 223 causes the camera 213 to start photographing, and the camera 213 photographs at a predetermined timing after the photographing control unit 223 causes the camera 213 to start photographing. and a recording control unit 224 for starting recording of the result.
Further, the control method of the camera unit 2 is such that when the motion sensor 212 for detecting whether or not a person exists around the refrigerator 1 detects that a person exists around the refrigerator 1, the camera 213 is caused to start photographing. , at a predetermined timing after the camera 213 has started photographing, recording of the photographed result of the camera 213 is started.

According to the camera unit 2 and the control method of the camera unit 2, when the motion sensor 212 detects that a person is present around the refrigerator 1, the camera 213 is caused to start photographing, thereby opening the door 11C or the drawer of the refrigerator 1. It is possible to increase the possibility that the food taken in and out of the refrigerator 1 before the door 16A is opened is included in the photographed result of the camera 213. Therefore, it is possible to accurately photograph the food entering and leaving the refrigerator 1 .

After the imaging control unit 223 causes the camera 213 to start imaging, the recording control unit 224 starts recording the imaging results of the camera 213 when the door 11C or the drawer 16A of the refrigerator 1 is opened.

According to this, since recording is started at a timing when it is highly likely that food will start to be put in and out, it is possible to suppress unnecessary recording in a situation where food is not put in and out of the refrigerator 1.例文帳に追加In addition, since the recording control unit 224 can suppress an increase in the data volume of the recording file 231, it is possible to prevent an increase in the processing targets of the food detection unit 225 and an increase in the processing time and processing load of the food detection unit 225. can be prevented.

The recording control unit 224 stops recording when the open door 11C or the drawer 16A of the refrigerator 1 is closed after recording is started.

According to this, since the recording ends at the timing when it is highly likely that the loading/unloading of the food is finished, unnecessary recording can be suppressed in a situation where the food is not loaded/unloaded to/from the refrigerator 1.例文帳に追加In addition, since the recording control unit 224 can suppress an increase in the data volume of the recording file 231, it is possible to prevent an increase in the processing targets of the food detection unit 225 and an increase in the processing time and processing load of the food detection unit 225. can be prevented.

When the recording is stopped, the recording control unit 224 generates one recording file 231 of the photographed results from the time when the door 11C or the drawer 16A of the refrigerator 1 is opened until it is closed.

According to this, since the photographed result from the time when the door 11C or the drawer 16A of the refrigerator 1 is opened until it is closed is generated as one recorded file 231, an increase in the data capacity of the one recorded file 231 is suppressed. One recording file 231 can be quickly generated. In addition, since one recorded file 231 can be generated quickly while suppressing an increase in the data volume of one recorded file 231, the food detection unit 225 can quickly Processing related to the recorded file 231 can be started.

When the motion sensor 212 detects that there is no person around the refrigerator 1 after the recording control unit 224 has stopped recording, the photography control unit 223 causes the camera 213 to end photography.

According to this, when the human detection sensor 212 detects that there is no person around the refrigerator 1, the camera 213 is allowed to finish photographing, so that the camera 213 photographs in a situation where no food is loaded or unloaded from the refrigerator 1. can be prevented. Therefore, food entering and leaving the refrigerator 1 can be accurately photographed while preventing an increase in power consumption.

The camera unit 2 includes a food detection section 225 that detects food entering and leaving the refrigerator 1 based on the recording result of the recording control section 224 .

According to this, since the food can be detected based on the photographed result of photographing the food entering and leaving the refrigerator 1 with high accuracy, the food entering and leaving the refrigerator 1 can be detected with high accuracy.

Further, as described above, the food management application 311 causes the terminal processor 300 of the terminal device 3 having the touch panel 32 to display the refrigerator interior image RG by the touch panel 32, and in the displayed refrigerator interior image RG, It functions as an application execution unit 301 that displays a photographed food image FG in a divided area BA corresponding to a storage area for food stored in the refrigerator compartment 11 .
In addition, the terminal device 3 displays a refrigerating compartment interior image RG by means of the touch panel 32, and in the displayed refrigerating compartment interior image RG, there are divided areas corresponding to food storage areas stored in the refrigerating compartment 11. The area BA is provided with an application execution unit 301 that displays the photographed food image FG.
In addition, the control method of the terminal device 3 is to display the image RG inside the refrigerator compartment, and in the image RG inside the refrigerator compartment displayed, a divided area BA corresponding to the accommodation area of the food accommodated in the refrigerator compartment 11 is photographed. A food image FG is displayed.

According to the food management application 311, the terminal device 3, and the control method of the terminal device 3, in the image showing the inside of the refrigerator compartment, the image is divided into divided areas BA corresponding to storage areas of food stored in the refrigerator. A food image FG is displayed. Therefore, the user P can intuitively grasp what kind of food is stored in the refrigerator compartment 11 .

Photographed food image FG is an image photographed by refrigerator camera 203 .

According to this, since the photographed image of the food actually stored in the refrigerator 1 is provided to the user P, the user P can more intuitively know what kind of food is stored in the refrigerator compartment 11. can be comprehended.

The refrigerator interior image RG is an image corresponding to the model of the refrigerator 1 used by the user P of the terminal device 3 .

According to this, since the inside image RG corresponding to the refrigerating compartment 11 of the refrigerator 1 actually used is provided to the user P, what kind of food is stored in the refrigerating compartment 11? The user P can more intuitively grasp whether there is any.

The refrigerator interior image RG is divided into a plurality of divided areas BA. The number of divided areas BA included in the refrigerator interior image RG is the number corresponding to the model of the refrigerator 1 used by the user P of the terminal device 3 .

According to this, the user P can grasp what kind of food is stored in the storage area of the refrigerator compartment 11 of the refrigerator 1 that is actually used. Therefore, the user P can more intuitively grasp what kind of food is stored in the refrigerator compartment 11 .

When displaying a plurality of photographed food images FG in one divided area BA, the application execution unit 301 displays the plurality of photographed food images FG side by side.

According to this, since a plurality of photographed food images FG can be displayed in an easy-to-view manner for each divided area BA, the user P can intuitively and easily grasp what kind of food is stored in the refrigerator compartment 11. can.

In one divided area BA, the upper limit number of photographed food images FG that can be displayed simultaneously is set.

According to this, it is possible to prevent deterioration of visibility when there are many photographed food images FG displayed in one divided area BA. Therefore, the user P can intuitively and easily grasp what kind of food is stored in the refrigerator compartment 11 .

When the number of photographed food images FG to be displayed in one divided area BA exceeds the upper limit number, the application execution unit 301 displays the photographed food images FG switchably within the range of the upper limit number.

According to this, when a large number of photographed food images FG are displayed in one divided area BA, the user P can visually recognize photographed food images FG that cannot be displayed while preventing deterioration of visibility.

The application executing unit 301 receives memo input from the user P of the terminal device 3 for each photographed food image FG, and displays a memo input icon MIC indicating that the memo has been received in association with the photographed food image FG.

According to this, it is possible for the user P to grasp for which food the memo has been input. Therefore, the user P can intuitively grasp what kind of food is stored in the refrigerator compartment 11, and can also grasp which food has a memo input.

The food management application 311 is an application program that can be installed on the terminal device 3 .

According to this, by installing the food management application 311, the terminal device 3 that does not have the function of displaying the refrigerator interior image RG and the photographed food image FG can be changed to a terminal device 3 that can display these images. can be done.

In addition, as described above, the food management application 311 causes the terminal processor 300 of the terminal device 3 having the touch panel 32 to display the captured image of the drawer camera 204 that captures the drawer 16A of the refrigerator 1 having the upper case JCA and the lower case GCA. is functioned as an application execution unit 301 that displays . The application execution unit 301 can display the upper case image JCG and the lower case image GCG, and accepts memo input from the user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the user P has input a memo to the upper case image JCG, the application executing unit 301 displays a memo input icon MIC indicating that the memo has been input in association with the upper case image JCG. When a memo input from the user P is accepted for the image GCG, the memo input icon MIC is displayed in association with the lower case image GCG.

The terminal device 3 also includes a touch panel 32 and an application execution unit 301 that displays an image captured by the drawer camera 204 . The application execution unit 301 can display the upper case image JCG and the lower case image GCG, and accepts memo input from the user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the user P has input a memo to the upper case image JCG, the application executing unit 301 displays a memo input icon MIC indicating that the memo has been input in association with the upper case image JCG. When a memo input from the user P is accepted for the image GCG, the memo input icon MIC is displayed in association with the lower case image GCG.

In addition, the control method of the terminal device 3 is capable of displaying the upper case image JCG and the lower case image GCG, and inputting a memo from the user P of the terminal device 3 to each of the upper case image JCG and the lower case image GCG. is received, and when a memo input is received from the user P for the upper case image JCG, a memo input icon MIC indicating that the memo input has been received is displayed in association with the upper case image JCG, and is displayed on the lower case image GCG. On the other hand, when a memo input from the user P is received, the memo input icon MIC is displayed in association with the lower case image GCG.

According to this, the food management application 311, the terminal device 3, and the control method of the terminal device 3 are such that the photographed image of the case CA of the drawer 16A is displayed, and the user P writes a memo on the displayed photographed image. can be entered. Therefore, the user P can intuitively grasp the food stored in the drawer 16A of the refrigerator 1, and convenience for the user P can be improved.

The application executing unit 301 displays the upper case image JCG superimposed on the lower case image GCG, and when receiving an operation to move the upper case image JCG from the user P, moves the first case image to display the lower case image JCG. Display GCG.

According to this, the lower case image GCG can be displayed in such a manner that the actual case CA is moved. Therefore, the user P can more intuitively grasp the food stored in the drawer 16A of the refrigerator 1, and the convenience for the user P can be improved.

When the memo input icon MIC is displayed in association with the upper case image JCG, the application execution unit 301 moves the memo input icon MIC together with the upper case image JCG.

According to this, even when the user P moves the upper case image JCG, the association between the upper case image JCG and the memo input icon MIC can be maintained. Therefore, even when the user P moves the upper case image JCG, the user P can appropriately grasp which image the memo input icon MIC relates to, and the convenience for the user P can be improved.

The application execution unit 301 displays a switch button for switching the captured image to be displayed to the upper case image JCG or the lower case image GCG, and switches the captured image to be displayed to the upper case image JCG or the lower case image GCG according to the operation of the switch button. switch to

According to this, the case image CG to be displayed can be easily switched by operating the switch button, so that the convenience for the user P can be improved.

The upper case image JCG is a photographed image when the upper case JCA is pulled out of the refrigerator 1 by a predetermined threshold or more. The lower case image GCG is a photographed image when the lower case GCA is pulled out of the refrigerator 1 by a predetermined threshold or more.

According to this, the case image CG can be a photographed image that captures as much food as possible, so that the user P can intuitively grasp the food stored in the drawer 16A of the refrigerator 1, and the convenience of the user P is increased. can be improved.

The food management application 311 is an application program that can be installed on the terminal device 3 .

According to this, by installing the food management application 311, the terminal device 3 that does not have the function of displaying the case image CG and the function of receiving the input of the memo for the displayed case image CG can be changed to these functions. can be a terminal device 3 having.

(Embodiment 2)
Next, Embodiment 2 will be described. In the description of the second embodiment, the same components as those of the food management system 1000 of the first embodiment are denoted by the same reference numerals, and detailed descriptions thereof are appropriately omitted.

[2-1. Constitution]
FIG. 26 is a block diagram showing configurations of the terminal device 3 and the food management server 4 according to the second embodiment.
26 and 7, the server processor 400 according to the second embodiment executes the control program stored in the server storage unit 410 to control the server communication control unit 401 and the information processing unit 402. In addition, it functions as the food recognition unit 403 .

The food recognition unit 403 performs food recognition processing based on the photographed food image FG indicated by the food image data 4116 included in each of the first update information and the second update information.

The food recognition process is a process of recognizing food from the photographed food image FG and acquiring from the icon database an icon indicating the same type of food as the recognized food. This icon database stores an icon representing the food for each food. Hereinafter, this icon will be referred to as a food icon. The icon database may be stored in the food management server 4 or may be stored in an external device with which the food management server 4 can communicate.
In the present embodiment, the food icon corresponds to an example of the food image of the present disclosure.

For example, the food recognition unit 403 recognizes food as follows.
The food recognition unit 403 identifies, from the images stored in the food image database, an image that has the highest degree of matching with the photographed food image FG indicated by the food image data 4116 included in the update information. The food image database is a database in which image data is stored for each food. The degree of matching is determined based on feature amounts such as shape and color, for example. Then, the food recognition unit 403 recognizes that the food associated with the specified image is the food indicated by the food image data 4116 included in the update information.

The food recognition unit 403 may recognize the food by AI (Artificial Intelligence). For example, machine learning processing is performed to learn feature amounts such as the color and shape of food from photographed food images FG for teachers, and a learned model is constructed in advance. This trained model is stored in the server storage unit 410 . The food recognition unit 403 uses the food image data 4116 included in the update information as input data, refers to the trained model, and recognizes the food indicated by the photographed food image FG. Here, this learned model may be stored in a device other than the food management server 4 connected to the global network GN, and the food may be recognized by the food management server 4 . In this configuration, the food recognition unit 403 transmits the food image data 4116 included in the update information to the device that stores the learned model, and acquires the food identification result from this device. Then, the food recognition unit 403 recognizes the food indicated by the acquired identification result as the food indicated by the photographed food image FG indicated by the food image data 4116 included in the update information.

After recognizing the food, the food recognition unit 403 acquires the food icon of the recognized food from the icon database. In the second embodiment, the icon data of the food icon acquired by the food recognition unit 403 is appropriately stored instead of or together with the food image data 4116 included in the refrigerator compartment food management record RR. Further, in Embodiment 2, the icon data of the food icon acquired by the food recognition unit 403 is appropriately stored in place of or together with the food image data 4116 of the withdrawn food management record HR.

Accordingly, in the second embodiment, the food icon can be displayed in the application UI 320 instead of the photographed food image FG.

[2-2. effects, etc.]
According to the second embodiment, the same effects as those of the first embodiment are obtained.

(Other embodiments)
As described above, the above embodiments have been described as examples disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments with modifications, replacements, additions, omissions, and the like. Further, it is also possible to combine the constituent elements described in the above embodiments to form a new embodiment.
Therefore, other embodiments will be exemplified below.

In the embodiment described above, the refrigerator 1 was exemplified as the storage. However, the storage of the present disclosure is not limited to the refrigerator 1, and may be, for example, a warehouse installed in an office or the like. In the above-described embodiments, food was exemplified as the article of the present disclosure, but the article of the present disclosure is not limited to food, and may be other articles such as books depending on the storage.

In the embodiment described above, the configuration in which the position regulating member 205 has the mark 207 at the center of the main body 201 in the left-right direction is exemplified. However, the position of the mark 207 is not limited to the center of the main body 201 in the horizontal direction. The position of the mark 207 may be other than the center of the main body 201 in the horizontal direction. In this case, each mark is appropriately recorded on the center position recording paper CY so that the camera unit 2 whose position is adjusted with reference to the mark 207 can appropriately perform various detections.

In the above-described embodiment, the refrigerator interior image RG, which is an image of the refrigerator compartment 11 viewed from the front when the left door 11A and the right door 11B are open, is exemplified as the refrigerator interior image of the present disclosure. . However, the refrigerator interior image of the present disclosure is not limited to this refrigerator interior image RG, and may be an image of a refrigerator viewed from the front when a single door is open, or an image of the interior of one refrigerator. In the case of the refrigerator 1 including a refrigerator compartment and a freezer compartment, an image of the inside of the refrigerator 1 when the door is open may be viewed from the front.

The imaging range of the drawer camera 204 may be a range in which one or a plurality of arbitrary drawer-type storage rooms are combined according to the type of the drawer-type storage room on the screen displayed by the application UI 320 .

The food management system 1000 may manage other elements such as the remaining amount and expiration date in addition to the storage date and time.

The main body 201 and the photographing member 202 included in the camera unit 2 may be configured separately.

The camera unit 2 may be configured to transmit the case image CG to the food management server 4 even when the case image CG is not pulled out by a predetermined threshold. In the case of this configuration, the food management server 4 stores the case image CG in the case where more than a predetermined threshold has not been withdrawn, and provides the terminal device 3 with the case image CG. Then, the application execution unit 301 displays this case image CG on the application UI 320 so as to be replaceable with each of the upper case image JCG and the lower case image GCG.

In the above-described embodiment, the user P installs the camera unit 2 in the refrigerator 1, but the camera unit 2 may be installed in the refrigerator 1 before shipment.

The server processor 400 may function as the food detection section 225 and the camera unit processor 220 may not function as the food detection section 225 . In this configuration, camera unit 2 transmits recorded file 231 to food management server 4 .

In the embodiment described above, the human sensor 212 was exemplified as the sensor of the present disclosure. However, the sensor of the present disclosure is not limited to the motion sensor 212, and may be an illuminance sensor, an electrostatic sensor provided in the grip of the refrigerator 1, or the like. In the case of this electrostatic sensor, refrigerator 1 and camera unit 2 can communicate, and refrigerator 1 transmits the detection value of the electrostatic sensor to camera unit 2 .

The terminal device 3 may store the refrigerator compartment food management record RR and the drawer food management record HR, and the food management application 311 may manage various information related to food detected by the camera unit 2 . In the case of this configuration, the application execution unit 301 appropriately updates the contents of each record stored by the terminal device 3 in the same manner as the food management server 4 described above. As a result, the food management system 1000 does not need to have the food management server 4, and the system configuration is simplified. Also, in this configuration, the application execution unit 301 may function as the food recognition unit 403 .

For example, the types of rooms formed in the main casing 10 of the refrigerator 1 are not limited to the refrigerator compartment 11, the ice making compartment 12, the fresh freezing compartment 13, the freezing compartment 14, and the vegetable compartment 15. Further types of rooms may be formed. Also, the number of doors provided at the front opening of the refrigerator compartment 11 may be one.

For example, the number of cases provided in the drawers 14A and 15A is not limited to two, and may be more. In this case, one case corresponds to the first case of the present disclosure, and the lower case of this certain case corresponds to the second case of the present disclosure.

For example, the camera unit processor 220, terminal processor 300, and server processor 400 may be composed of a single processor or may be composed of multiple processors.

Each unit shown in FIGS. 6, 7, and 26 is an example, and the specific implementation is not particularly limited. That is, it is not always necessary to mount hardware corresponding to each part individually, and it is of course possible to adopt a configuration in which one processor executes a program to realize the function of each part. Further, part of the functions implemented by software in the above-described embodiments may be implemented by hardware, or part of the functions implemented by hardware may be implemented by software. In addition, the specific detailed configurations of other parts of the camera unit 2, the terminal device 3, and the food management server 4 can be arbitrarily changed within the scope of the present disclosure.

13, 14, 22, and 25 are divided according to the main processing content in order to facilitate understanding of the operation of each part of the food management system 1000. The present disclosure is not limited by the division method or names of the processing units.

Note that the above-described embodiment is for illustrating the technology in the present disclosure, and various changes, replacements, additions, omissions, etc. can be made within the scope of the claims or equivalents thereof.

INDUSTRIAL APPLICABILITY As described above, the program, the terminal device, and the control method of the terminal device according to the present invention can be used for displaying images of foods stored in and out of the refrigerator.

1 refrigerator (storage)
2 Camera unit (shooting device)
3 terminal device 4 food management server 10 main box 10A front edge 10B upper surface 11 refrigerator compartment 11A left door 11B right door 11C door 12 ice making compartment (drawer-type storage compartment)
12A, 13A, 14A, 15A, 16A drawer 13 fresh freezing chamber (drawer storage chamber)
14 freezer (withdrawable storage room)
15 Vegetable room (drawer storage room)
32 touch panel (display unit)
201 main body 202 photographing member 203 refrigerator camera 204 drawer camera 205 position regulating member 206 bottom surface 207 mark 212 motion sensor (sensor)
213 camera 223 photography control unit 224 recording control unit 225 food detection unit (goods detection unit)
231 recording file 300 terminal processor (computer)
301 application execution unit (display control unit)
311 food management apps (programs, application programs)
1000 food management system BA division area (area)
FG, FG1 photographed food image (food image)
GCA lower case (second case)
GCG lower case image (2nd case image)
JCA upper case (1st case)
JCG upper case image (1st case image)
MIC memo input icon (memo input image)
P User RG Refrigerator inside image (inside image)

Claims (12)

A computer of a terminal device equipped with a display unit,
The display unit displays an interior image showing the interior of the refrigerator as viewed from the front of the refrigerator, and in the displayed interior image, an area corresponding to a storage area for food stored in the refrigerator. to function as a display control unit that displays a food image showing the food,
program. The food image is an image captured by a camera for a refrigerator compartment that captures food that is loaded and unloaded from the refrigerator.
A program according to claim 1. The food image is an icon corresponding to the type of food,
A program according to claim 1. The refrigerator interior image is an image corresponding to the model of the refrigerator used by the user of the terminal device,
A program according to any one of claims 1 to 3. The inside image is divided into a plurality of areas,
The number of areas included in the refrigerator interior image is a number corresponding to the model of the refrigerator used by the user of the terminal device.
5. A program according to claim 4. The display control unit
When displaying a plurality of the food images in one of the areas, displaying the plurality of food images side by side;
6. A program according to claim 5. In the area of 1, an upper limit number of the food images that can be displayed at the same time is set.
7. A program according to claim 6. The display control unit
When the number of the food images to be displayed in the area of 1 exceeds the upper limit number, the food images are switchably displayed within the range of the upper limit number.
8. A program according to claim 7. The display control unit
Receiving a memo input from the user of the terminal device for each food image, and displaying a memo input image indicating that the memo has been received in association with the food image;
A program according to any one of claims 1 to 8. An application program that can be installed on the terminal device,
A program according to any one of claims 1 to 9. a display unit;
The display unit displays an interior image showing the interior of the refrigerator as viewed from the front of the refrigerator, and in the displayed interior image, an area corresponding to a storage area for food stored in the refrigerator. A display control unit that displays a food image showing the food,
Terminal equipment. displaying an image of the interior of the refrigerator as viewed from the front of the refrigerator;
Displaying a food image showing the food in an area corresponding to an accommodation area of the food stored in the refrigerator in the displayed image of the inside of the refrigerator;
Terminal device control method.

Images