JP7466117B2 - Refrigerator, management system, and program - Google Patents

Description

The present invention relates to a refrigerator.

Conventionally, refrigerators have been proposed that capture images of objects (food, etc.) passing through the opening of the refrigerator compartment at preset time intervals, calculate the direction of travel of the object from the time-series captured images, and determine whether the object is entering or leaving the refrigerator based on the direction of travel (see, for example, Patent Document 1).

JP 2015-81762 A

The configuration described in the above Patent Document 1 requires a certain amount of memory capacity to store multiple time-series captured images.In addition, when calculating the traveling direction of an object, a complex process is performed to extract image portions of the same object from multiple time-series captured images and analyze changes in position, which causes the inconvenience of a large computational burden associated with determining whether the object is entering or leaving the warehouse.
The present invention has been made in consideration of the above-described background, and aims to provide a refrigerator, a management system, and a program that can determine the entry and exit of food through simple processing and store information on the entry and exit of food while suppressing an increase in memory capacity.

In order to solve the above-mentioned conventional problems, the refrigerator of the present disclosure includes a storage chamber having an opening, a passing object detection unit that detects an object passing through the opening, a first camera that captures an image of the inside of the storage chamber, a second camera that is positioned to include the opening in its imaging range, a memory unit, a storage amount recognition unit that recognizes the amount of food stored in the storage chamber by extracting an image portion of the food from an image of the inside of the storage chamber captured by the first camera, a food information acquisition unit that, when an object passing through the opening is detected by the passing object detection unit, captures an image using the second camera and acquires information about the food from the image captured by the second camera, an entry/exit determination unit that, when an object passing through the opening is detected by the passing object detection unit, determines whether food has been stored in the storage chamber or removed from the storage chamber based on a change in the amount of food recognized by the storage amount recognition unit, and determines whether food has been stored in the storage chamber or removed from the storage chamber by the entry/exit determination unit. and a food information management unit that, when it is determined that a food item has been stored in the storage room or that a food item has been removed from the storage room, stores food storage/removal information based on the result of the determination in the storage unit; when an object passing through the opening is detected by the passing object detection unit, and when the storage/removal determination unit determines that a food item has been stored in the storage room or that a food item has been removed from the storage room in response to the detection of the object, associates the determined storage/removal information with the acquired food information and stores it in the storage unit; and, when it is determined that a food item has been stored in the storage room by the storage/removal determination unit and the food information is acquired by the food information acquisition unit, if the coincidence rate between the information of the food item removed within a predetermined time and the information of the food item currently stored is equal to or greater than a predetermined value, updates the information of the food item removed before the predetermined time based on the information of the food item currently stored and stores it in the storage unit.

The management system of the present disclosure is a management system including a refrigerator, a management server that communicates with the refrigerator, and a terminal device that communicates with the refrigerator or the management server, and the refrigerator includes a storage chamber having an opening, a passing object detection unit that detects objects passing through the opening, a first camera that captures an image of the inside of the storage chamber, a second camera that is positioned to include the opening in its imaging range, a memory unit, a storage amount recognition unit that recognizes the amount of food stored in the storage chamber by extracting an image portion of the food from an image of the inside of the storage chamber captured by the first camera, and a passing object detection unit that detects objects passing through the opening. a food information acquisition unit that performs an image capture by the second camera when an object passing through the opening is detected, and acquires information about the food from the image captured by the second camera; an entry/exit determination unit that determines whether food has been entered into the storage chamber or taken out from the storage chamber based on a change in the amount of food recognized by the storage amount recognition unit when an object passing through the opening is detected by the passing object detection unit; and a food entry/exit determination unit that stores food entry/exit information based on a result of the determination when the entry/exit determination unit determines whether food has been entered into the storage chamber or taken out from the storage chamber, and In response to the detection of the object, the entry/exit determination unit determines whether food has been entered into the storage room or whether food has been removed from the storage room, and when the food information acquisition unit acquires information about the food, the determined entry/exit information is associated with the acquired information about the food and stored in the memory unit, and when the entry/exit determination unit determines whether food has been entered into the storage room and the food information acquisition unit acquires the information about the food, if a matching rate between information about the food that was removed within a predetermined time and information about the food that is currently entered is equal to or greater than a predetermined value, the information about the food that was removed before the predetermined time is stored based on information about the food that is currently entered. and a food information management unit that updates the refrigerator based on the stored food information and stores it in the storage unit, and transmits to the management server the food entry or exit information determined by the entry/exit determination unit and the food information acquired by the food information acquisition unit, and the management server receives and stores the data of the food entry list transmitted from the refrigerator, and the terminal device receives and acquires the data of the food entry list from the management server or the refrigerator, and displays information on the food stored in the refrigerator based on the food entry list on the display unit of the terminal device.

The program of the present disclosure also includes a CPU constituting a control unit of a refrigerator including a storage chamber having an opening, a passing object detection unit that detects objects passing through the opening, a first camera that captures an image of the inside of the storage chamber, a memory unit, a second camera that is positioned to include the opening in its imaging range, and a memory unit; a storage amount recognition unit that recognizes the amount of food stored in the storage chamber by extracting an image portion of the food from an image of the inside of the storage chamber captured by the first camera; a food information acquisition unit that, when an object passing through the opening is detected by the passing object detection unit, captures an image with the second camera and acquires information about the food from the image captured by the second camera; an entry/exit determination unit that, when an object passing through the opening is detected by the passing object detection unit, determines whether food has been stored in the storage chamber or removed from the storage chamber based on a change in the amount of food recognized by the storage amount recognition unit; When it is determined that food has been stored in the storage chamber or that food has been removed from the storage chamber, food storage/removal information based on the result of the determination is stored in the storage unit; when an object passing through the opening is detected by the passing object detection unit, and when the storage/removal determination unit determines that food has been stored in the storage chamber or that food has been removed from the storage chamber in response to the detection of the object, and when food information is acquired by the food information acquisition unit, the determined storage/removal information is associated with the acquired food information and stored in the storage unit; and when it is determined that food has been stored in the storage chamber by the storage/removal determination unit and the food information is acquired by the food information acquisition unit, if the coincidence rate between information on food removed within a predetermined time and information on food currently stored is equal to or greater than a predetermined value, the information on food removed before the predetermined time is updated based on information on food currently stored, and stored in the storage unit.

As a result, when an object passing through the opening of the storage chamber is detected by the passing object detection unit, the entry/exit determination unit can determine whether food has entered the storage chamber or been removed from the storage chamber based on the change in the amount of food in the storage chamber recognized by the storage amount recognition unit. In this case, it is only necessary to perform the simple processes of detecting an object passing through the opening of the storage chamber and calculating the change in the amount of food in the storage chamber, and there is no need to store time-series continuous image data in memory. Therefore, it is possible to determine whether food has entered or been removed from the storage chamber through simple processes and store the results in the memory unit while suppressing an increase in memory capacity.

The refrigerator disclosed herein can determine whether food is being stored in or removed from the storage compartment through a simple process of detecting an object passing through an opening using a passing object detection unit and recognizing the amount of food in the storage compartment using a storage amount recognition unit, and can store information on the food being stored and removed.

FIG. 1 is an explanatory diagram of a configuration for checking the status of food stored in a refrigerator installed in a house using a terminal device. Diagram of the refrigerator's appearance An explanatory diagram showing the internal structure of a refrigerator in a cross-sectional view from the right side. An explanatory diagram showing the internal structure of the refrigerator compartment as a cross-sectional view from the front. Control unit configuration diagram 1 is a flowchart showing the overall processing by the control unit. Flowchart of imaging process Flowchart of weight information processing Flowchart of analysis process of images taken at the time of storage Flowchart of analysis process of images at the time of leaving the warehouse Flowchart of weight sensor correction process First flow chart of food management process in warehouse Second Flowchart of In-Store Food Management Process List of food in the warehouse and list of expiration dates

The first invention comprises a storage chamber having an opening on the front, a passing object detection unit that detects objects passing through the opening, a storage capacity recognition unit that recognizes the amount of food stored in the storage chamber, and an entry/exit determination unit that determines the entry of food into the storage chamber and the removal of food from the storage chamber based on a change in the amount of food recognized by the storage capacity recognition unit when an object passing through the opening is detected by the passing object detection unit.
According to the first invention, when the passing object detection unit detects an object passing through the opening of the storage chamber, the entry/exit determination unit determines whether food has been stored or removed based on the change in the amount of food in the storage chamber recognized by the storage amount recognition unit. Therefore, there is no need to capture and store time-series images in memory in order to recognize the direction of food movement and determine whether food has been stored or removed, and it is possible to determine whether food has been stored or removed by simple processing while suppressing an increase in memory capacity.

In a second invention, the storage chamber is divided into a plurality of sub-areas, the storage capacity recognition unit recognizes the amount of food stored in the storage chamber for each of the sub-areas, and the entry/exit determination unit determines the entry of food into each of the sub-areas and the exit of food from each of the sub-areas based on changes in the amount of food in each of the sub-areas recognized by the storage capacity recognition unit.
According to the second aspect of the present invention, it is possible to determine into which sub-area within the storage room the food has been stored, and from which sub-area within the storage room the food has been removed.

The third invention includes a memory unit and a food information management unit that, when the in/out determination unit determines that food has been entered into the storage room or that food has been removed from the storage room, stores food in/out information based on the result of the determination in the memory unit.
According to the third invention, by storing in- and out-of-stock information based on the results of the food in- and out-stock determination by the in- and out-stock determination unit in the memory unit by the food information management unit, it becomes possible to use the food in- and out-stock information at any time.

A fourth invention includes a date and time recognition unit that recognizes the current date and time, and when the entry/exit determination unit determines that food has entered the storage room or been removed from the storage room, the food information management unit includes the date and time recognized by the date and time recognition unit at the time of the determination in the food entry/exit information and stores it in the memory unit.
According to the fourth invention, the food information management unit stores the date and time when food is stored or taken out in the memory unit, making it possible to manage the expiration date, etc. of food stored in the storage unit.

A fifth invention includes a camera positioned to include the opening within its imaging range, and a food information acquisition unit that, when an object passing through the opening is detected by the passing object detection unit, captures an image using the camera and acquires food information from the captured image, and the food information management unit detects an object passing through the opening by the passing object detection unit, and in response to the detection of the object, the entry/exit determination unit determines whether food has been exited from the storage room or whether food has been entered from the storage room, and when food information is acquired by the food information acquisition unit, associates the determined entry/exit information with the acquired food information and stores it in the memory unit.
According to the fifth aspect of the present invention, the food information acquisition unit acquires information about ingredients by performing character recognition (recognizing characters indicating the product name, place of production, ingredients, expiration date, etc.) or by recognizing the shape and color of the image portion of the food in the image captured by the camera. The food management unit then associates the information about ingredients acquired by the food information acquisition unit with the information about the incoming and outgoing food determined by the in/out determination unit, and stores the information in the memory unit. This makes it possible to identify and manage the types of food stored in the storage unit.

In a sixth aspect of the present invention, the passing object detection unit detects an object passing through the opening by extracting an image portion of the object from an image captured by the camera.
According to the sixth aspect of the present invention, there is no need to provide a dedicated configuration for detecting an object passing through the opening, separate from the camera, and therefore the configuration for detecting a passing object can be simplified.

The following describes an embodiment of the present invention with reference to the drawings. Note that the present invention is not limited to this embodiment.

[1. Usage of refrigerator]
1 shows a usage state of a refrigerator according to this embodiment. Refrigerator 1 according to this embodiment is installed in house H, and has a function of communicating with a management server 520 via a gateway 500 and a communication network 510. Refrigerator 1 also has a function of communicating with a terminal device (smartphone, tablet terminal, etc.) 400. Terminal device 400 has a function of communicating with the management server 520 via the gateway 500 and the communication network 510.

The control unit 100 provided in the refrigerator 1 controls the overall operation of the refrigerator 1, and generates an internal food list 131 showing information about the food stored in the storage compartment of the refrigerator 1 and sends it to the management server 520. The refrigerator management application executed on the terminal device 400 acquires the internal food list 131 data from the management server 520, or acquires the internal food list 131 data from the refrigerator 1, and displays information about the food stored in the refrigerator 1 based on the internal food list 131 on the display unit. A user P of the refrigerator 1 can check the status of the food stored in the refrigerator 1 (food name, expiration date, remaining amount, etc.) by visually checking the display on the terminal device 400.

[2. Configuration of the refrigerator]
The configuration of the refrigerator 1 will be described with reference to Figures 2 to 5. Figure 2 is an explanatory diagram showing the external appearance of the refrigerator 1. As shown in Figure 2, the refrigerator 1 has a main box body 2 with an opening on the front side, and inside the main box body 2, a refrigeration chamber 10 (corresponding to a storage chamber of the present invention), an ice-making chamber 30, a switching chamber 20 that is adjacent to the ice-making chamber 30 and allows the temperature inside to be changed, a freezer chamber 40, and a vegetable chamber 50 are formed.

The opening at the front of the refrigerator compartment 10 is provided with a rotating right door 11 and a left door 12. In addition, the switchable compartment 20, ice-making compartment 30, freezer compartment 40, and vegetable compartment 50 are each provided with drawers 21, 31, 41, and 51 for storing food.

Next, FIG. 3 is an explanatory diagram showing the internal configuration of the refrigerator 1 in a cross-sectional view seen from the right side, and FIG. 4 is an explanatory diagram showing the internal configuration of the refrigerator compartment 10 in a cross-sectional view seen from the front. With reference to FIG. 3 and FIG. 4, the refrigerator compartment 10 is provided with an upper shelf 13, a middle shelf 14, and a lower shelf 15 that divide the refrigerator compartment 10 into sub-areas. The refrigerator compartment 10 is also provided with weight sensors 13a and 13b that detect the weight of food placed on the upper shelf 13, weight sensors 14a and 14b that detect the weight of food placed on the middle shelf 14, and weight sensors 15a and 15b that detect the weight of food placed on the lower shelf 15. The refrigerator compartment 10 is also provided with lighting units 16a to 16f that illuminate the interior of the refrigerator compartment 10, illuminance sensors 17a to 17c that detect the illuminance within the refrigerator compartment 10, and a camera 60. The camera 60 corresponds to the second camera of this disclosure.

The camera 60 is positioned near the opening 10a at the top of the refrigerator compartment 10 so that the opening 10a is included in the imaging range, and images the area from the opening 10a to the front ends of the shelves 13, 14, and 15. In addition, a compressor 61, a cooling fan 62, a cooler 63, and a condenser 64, which are auxiliary machines that make up the refrigeration cycle, are provided. Furthermore, a refrigerator compartment duct 70 and cold air outlets 71-73 for circulating cold air are provided at the back of the refrigerator compartment 10.

The refrigerator compartment 10 is provided with an opening/closing sensor 18b that detects whether the right door 11 is open or closed, and an opening/closing sensor 18a that detects whether the left door 12 is open or closed. The opening/closing sensor 18b outputs a closed detection signal when the right door 11 is closed, and outputs an open detection signal when the right door 11 is open. Similarly, the opening/closing sensor 18a outputs a closed detection signal when the left door 12 is closed, and outputs an open detection signal when the left door 12 is open.

The switching chamber 20 is provided with an opening/closing sensor 22 that detects whether the opening 20a is open or closed. The opening/closing sensor 22 outputs a closed detection signal when the drawer 21 is housed in the switching chamber 20, and outputs an open detection signal when the drawer 21 is pulled out of the switching chamber 20.

Similarly, the ice making compartment 30 is provided with an opening/closing sensor 32 that detects whether the opening 30a is opened or closed by the drawer 31. The freezer compartment 40 is provided with an opening/closing sensor 42 that detects whether the opening 40a is opened or closed by the drawer 41, and the vegetable compartment 50 is provided with an opening/closing sensor 52 that detects whether the opening 50a is opened or closed by the drawer 51. The opening/closing sensors 32, 42, and 52 output a closed detection signal and an open detection signal, similar to the opening/closing sensor 22.

Next, FIG. 5 is a configuration diagram of the control unit 100. Referring to FIG. 5, the control unit 100 is an electronic circuit unit that is composed of a CPU (Central Processing Unit) 110, an image processing unit 120, a memory unit 130, a wireless communication unit 140, a date and time recognition unit 141, and an interface circuit (not shown), etc.

The CPU 110 executes the control program for the refrigerator 1 stored in the memory unit 130, thereby functioning as a passing object detection unit 111, a storage capacity recognition unit 112, an entry/exit determination unit 113, a food information acquisition unit 114, and a food information management unit 115.

The control unit 100 is connected to the opening and closing sensors 18a, 18b, 22, 32, 42, 52, the illuminance sensors 17a, 17b, 17c, the weight sensors 13a, 13b, 14a, 14b, 15a, 15b, the camera 60, the lighting units 16a, 16b, 16c, 16d, 16e, 16f, and the refrigeration cycle auxiliary units 61, 62, 63, 64.

The control unit 100 receives detection signals from the open/close sensors 18a, 18b, 22, 32, 42, 52, the illuminance sensors 17a, 17b, 17c, and the weight sensors 13a, 13b, 14a, 14b, 15a, 15b. In addition, the control signals output from the control unit 100 control the operation of the lighting units 16a, 16b, 16c, 16d, 16e, 16f, and the refrigeration cycle auxiliary units 61, 62, 63, 64.

Furthermore, the image capturing by the camera 60 is controlled via the image processing unit 120 by a control signal output from the control unit 100. The image processing unit 120 converts the video signal output from the camera 60 into a digital gradation signal to generate a captured image, and stores the captured image data in the image memory 121. The image processing unit 120 processes the captured image stored in the image memory 121 in response to a control signal output from the CPU 110.

The date and time recognition unit 141 recognizes the current date and time (date and time) by measuring a clock signal output from a clock circuit (not shown) provided in the control unit 100. Note that the current date and time may be recognized by acquiring information on the current date and time from the management server 520 through communication with the management server 520.

In addition to the data of the in-fridge food list 131, the memory unit 130 stores data of incoming food information 132, which records information on foods stored in the refrigerator compartment 10, and outgoing food information 133, which records information on foods that have been taken out of the refrigerator compartment 10. In addition, the memory unit 130 stores data of sample images 134 of various foods, and an expiration date list 135, which shows general expiration dates for various foods.

The passing object detection unit 111 detects objects passing through the opening 10a of the refrigerator compartment 10 based on changes in illuminance inside the refrigerator compartment 10 detected by the illuminance sensors 17a, 17b, and 17c. The storage amount recognition unit 112 recognizes the amount of food stored on the upper shelf 13, middle shelf 14, and lower shelf 15 of the refrigerator compartment 10 by weight based on detection signals from the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b.

The storage amount recognition unit 112 recognizes the weight of food stored on the upper shelf 13 based on the detection signals of the weight sensors 13a and 13b, and recognizes the weight of food stored on the middle shelf 14 based on the detection signals of the weight sensors 14a and 14b. The storage amount recognition unit 112 also detects the weight of food stored on the lower shelf 15 based on the detection signals of the weight sensors 15a and 15b. Each storage space on the upper shelf 13, middle shelf 14, and lower shelf 15 corresponds to a sub-area in the present invention.

When the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10, the entry/exit determination unit 113 determines whether food has been entered into or removed from the refrigerator compartment 10 based on the change in the amount (weight) of food placed on each shelf 13, 14, 15 recognized by the storage capacity recognition unit 112.

The food information acquisition unit 114 uses the camera 60 to capture an image of an object that passes through the opening 10a of the refrigerator compartment 10. Then, when the in/out determination unit 113 determines that food has been stored in or removed from the refrigerator compartment 10, it extracts an image portion of the food from the captured image.

The food information acquisition unit 114 acquires information such as the name, content, place of origin, and expiration date of food that has been stored or shipped by recognizing characters contained in the image portion of the food or by extracting features of the image portion of the food. The food information acquisition unit 114 then records the acquired information in stored food information 132 for food determined to be stored (stored food), and in shipped food information 133 for food determined to be shipped (shipped food).

The food information management unit 115 updates the food information recorded in the in-fridge food list 131 based on the weight of the food in the refrigerator compartment 10 recognized by the storage capacity recognition unit 112, the incoming food information 132 and the outgoing food information 133 recorded by the food information acquisition unit 114, etc.

[3. Processing by the control unit]
Next, the process for managing food stored in the refrigerator compartment 10, which is executed by the control unit 100, will be described with reference to FIGS.

Figure 6 is a flow chart showing an overview of a series of processes executed by the control unit 100. The control unit 100 executes "imaging process" in step S1, "weight information process" in step S2, "image analysis process" in step S3, "weight sensor correction process" in step S4, and "in-storage food management process" in step S5. Each process is described in detail below.

[3-1. Imaging Processing]
The procedure for executing the "imaging process" will be described with reference to the flowchart shown in Fig. 7. The "imaging process" is executed by the passing object detection unit 111 and the food information acquisition unit 114. Steps S10 to S15 in Fig. 7 are processes performed by the passing object detection unit 111. When the passing object detection unit 111 determines in step S10 from the detection signals of the opening/closing sensors 18a, 18b that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have been opened, the process proceeds to step S11.

In step S11, the passing object detection unit 111 detects the illuminance of the upper shelf 13, middle shelf 14, and lower shelf 15 of the refrigerator compartment 10 (the illuminance of each space partitioned by the shelves) using the illuminance sensors 17a, 17b, and 17c. The passing object detection unit 111 then sets the detected illuminance of the upper shelf 13 as the upper reference illuminance, the illuminance of the middle shelf 14 as the middle reference illuminance, and the illuminance of the lower shelf 15 as the lower reference illuminance. The passing object detection unit 111 stores the data of the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance in the memory unit 130.

The upper reference illuminance, the middle reference illuminance, and the lower reference illuminance are the illuminances of the shelves 13, 14, and 15 when one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 are open and before an object (user's hand, food, etc.) passes through the opening 10a of the refrigerator compartment 10. In addition, taking into account errors and fluctuations in the detected illuminance, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance may be set to a slightly lower or higher illuminance than the detected illuminance of each illuminance sensor 17a, 17b, and 17c. In addition, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S13 described later and the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S14 may be set to different values.

In the next step S12, the passing object detection unit 111 detects the illuminance of the upper shelf 13, the middle shelf 14, and the lower shelf 15 using the illuminance sensors 17a, 17b, and 17c, respectively. Then, in the next step S13, the passing object detection unit 111 determines whether there is a shelf whose detected illuminance is lower than the reference illuminance by a predetermined level or more. Specifically, the passing object detection unit 111 determines whether each of the following conditions (1) to (3) is met.

(1) The illuminance detected by the illuminance sensor 17a has changed from the "upper reference illuminance" to "upper reference illuminance - α" or lower.
(2) The illuminance detected by the illuminance sensor 17b has changed from the "medium reference illuminance" to "medium reference illuminance -α" or lower.
(3) The illuminance detected by the illuminance sensor 17c has changed from the "lower reference illuminance" to "lower reference illuminance -α" or less.

where α is a constant for determining the threshold determined by experiments, computer simulations, etc., assuming a drop in illuminance that occurs when an item (food, a user's hand, etc.) enters the upper shelf 13, the middle shelf 14, or the lower shelf 15. α may be a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values individually for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance.

The passing object detection unit 111 then proceeds to step S20 when at least one of the above conditions (1) to (3) is met. When at least one of the above conditions (1) to (3) is met, it is assumed that the user has placed their hand inside the refrigerator compartment 10 to put food into the refrigerator compartment 10 (storage), to take food out of the refrigerator compartment 10 (removal), or to change the position of food in the refrigerator compartment 10. In this case, the passing object detection unit 111 can determine that an object (the user's hand, food being held by the user's hand, etc.) has passed through the opening 10a of the refrigerator compartment 10.

On the other hand, when none of the above conditions (1) to (3) are met, the passing object detection unit 111 proceeds to step S14. In step S14, the passing object detection unit 111 determines whether there is a shelf where the detected illuminance has changed from less than or equal to "reference illuminance - β" to more than or equal to "reference illuminance." Specifically, the passing object detection unit 111 determines whether each of the following conditions (4) to (6) is met.

(4) The illuminance detected by the illuminance sensor 17a has changed from less than the "upper reference illuminance-β" to more than the "upper reference illuminance."
(5) The illuminance detected by the illuminance sensor 17b has changed from less than or equal to the "middle reference illuminance-β" to more than or equal to the "middle reference illuminance."
(6) The illuminance detected by the illuminance sensor 17c has changed from less than the "lower reference illuminance-β" to more than the "lower reference illuminance."

where β is a constant for determining the threshold determined by experiments, computer simulations, etc., assuming an increase in illuminance that occurs when an object (food, a user's hand, etc.) that has entered the upper shelf 13, middle shelf 14, or lower shelf 15 passes through the opening 10a and leaves the refrigerator compartment 10. β may be set to a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values individually for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance.

The passing object detection unit 111 then advances the process to step S22 when at least one of the above conditions (4) to (6) is met. When at least one of the above conditions (4) to (6) is met, it is assumed that the user has removed their hand from the refrigerator compartment 10 to put food into the refrigerator compartment 10 (storage), to take food out of the refrigerator compartment 10 (removal), or to change the position of food contained in the refrigerator compartment 10. In this case, the passing object detection unit 111 can determine that an object (the user's hand, food being held by the user's hand, etc.) has passed through the opening 10a of the refrigerator compartment 10.

On the other hand, when none of the above (4) to (6) is true, the passing object detection unit 111 advances the process to step S15. In step S15, the passing object detection unit 111 determines whether or not both the right door 11 and the left door 12 of the refrigerator compartment 10 are closed, based on the detection signals of the open/close sensors 18a, 18b. Then, when both the right door 11 and the left door 12 are closed, the passing object detection unit 111 advances the process to step S15 and ends the "imaging process". On the other hand, when at least one of the right door 11 and the left door 12 is open, the passing object detection unit 111 advances the process to step S12.

Next, steps S20 to S21 and steps S22 to S23 are processes performed by the food information acquisition unit 114. In step S20, the food information acquisition unit 114 captures an image of an area including the opening 10a of the refrigerator compartment 10 with the camera 60 via the image processing unit 120. In the following step S21, the food information acquisition unit 114 stores the image data captured in step S20 in the memory unit 130 as "entry image" data, and proceeds to step S15. Note that imaging may be performed multiple times, and multiple image data may be stored in the memory unit 130 as "entry image" data.

In step S22, the food information acquisition unit 114 captures an image of an area including the opening 10a of the refrigerator compartment 10 using the camera 60 via the image processing unit 120. In the following step S23, the food information acquisition unit 114 stores the image data captured in step S22 in the memory unit 130 as "image at time of leaving the warehouse" data, and proceeds to step S15. Note that the image may be captured multiple times, and multiple image data may be stored in the memory unit 130 as "image at time of leaving the warehouse" data.

By the "imaging process" shown in FIG. 7, the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 based on the change in illuminance of the upper shelf 13, the middle shelf 14, or the lower shelf 15 detected by the illuminance sensors 17a, 7b, and 17c.

In addition, the passing object detection unit 111 stores in the memory unit 130 an "entry image", which is an image captured near the opening 10a at the time when the passing object detection unit 111 detects an object that has passed through the opening 10a and entered one of the shelves 13, 14, or 15. In addition, the passing object detection unit 111 stores in the memory unit 130 an "exit image", which is an image captured near the opening 10a at the time when the passing object detection unit 111 detects an object that has left one of the shelves 13, 14, or 15 and passed through the opening 10a.

Note that, if an object passing through the opening 10a is detected multiple times in step S13 before it is detected in step S15 of FIG. 7 that the right door 11 and the left door 12 are closed, multiple "entry images" are captured and stored in steps S20 and S21. Then, "image analysis processing" is performed on each of the "entry images" of foods determined to have been entered by the "weight information processing" described below.

Similarly, when an object passing through the opening 10a is detected multiple times in step S14 before it is detected in step S15 of FIG. 7 that the right door 11 and the left door 12 are closed, multiple "images at the time of leaving the warehouse" are captured and stored in steps S22 and S23. Then, "image analysis processing" is performed on each of the "images at the time of leaving the warehouse" of the food that has been determined to have been left the warehouse by the "weight information processing" described below.

[3-2. Weight information processing]
Next, the execution procedure of the "weight processing" will be described with reference to the flowchart shown in Fig. 8. The "weight information processing" is executed by the incoming/outgoing determination unit 113 and the food information acquisition unit 114.

Steps S30 to S33 and S40 in FIG. 8 are processes performed by the entry/exit determination unit 113. In step S30, the entry/exit determination unit 113 recognizes the weight of each shelf 13, 14, 15 using the storage capacity recognition unit 112 and stores the weight in the memory unit 130. In the following step S31, when the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 in step S13 or step S14 in FIG. 7 (when an object enters or leaves the refrigerator compartment 10), the entry/exit determination unit 113 advances the process to step S32.

In step S32, the entry/exit determination unit 113 recognizes the weight of each shelf 13, 14, 15 using the storage capacity recognition unit 112. In the next step S33, the entry/exit determination unit 113 determines whether there is a shelf where the weight recognized in step S32 has increased compared to the weight recognized in step S30. Then, if there is a shelf where the weight has increased (in this case, it can be determined that food has been entered into the refrigerator compartment 10), the entry/exit determination unit 113 proceeds to step S34.

On the other hand, if there is no shelf where the weight has increased, the entry/exit determination unit 113 proceeds from step S33 to step S40, and determines whether there is a shelf where the weight recognized in step S32 has decreased compared to the weight recognized in step S30. If there is a shelf where the weight has decreased (in this case, it can be determined that food has been removed from the refrigerator compartment 10), the entry/exit determination unit 113 proceeds to step S41. If there is no shelf where the weight has increased (in this case, it can be determined that food has not been entered into or removed from the refrigerator compartment 10), the entry/exit determination unit 113 proceeds to step S36 and ends "weight information processing".

Steps S34 to S35 and step S41 are processes performed by the food information acquisition unit 114. In step S34, the food information acquisition unit 114 estimates the storage position of the stored food from the left and right weights of each shelf 13, 14, and 15 recognized by the storage capacity recognition unit 112.

In the next step S35, the food information acquisition unit 114 records the weight and storage location of the stored food in the stored food information 132 (see FIG. 5), proceeds to step S36, and ends the "weight information processing". The stored food information 132 data is temporarily stored in the memory unit 130 in order to add or update information to the in-warehouse food list 131.

In addition, in step S41, the food information acquisition unit 114 records the weight information of the outgoing food in the outgoing food information 133 (see FIG. 5) and proceeds to step S36, terminating the "weight information processing." The data of the outgoing food information 133 is temporarily stored in the memory unit 130 in order to add or update food information to the in-warehouse food list 131.

[3-3. Image analysis processing]
Next, the execution procedure of the "image analysis process" will be described with reference to the flowcharts shown in Figures 9 and 10. The "image analysis process" is executed by the food information acquisition unit 114. Figure 9 shows a process for acquiring information from an "image at the time of entry" for food stored in the refrigerator compartment 10. Figure 10 shows a process for acquiring information from an "image at the time of exit" for food removed from the refrigerator compartment 10.

First, the "image analysis process" for the "image at time of storage" will be described with reference to FIG. 9. In step S50 of FIG. 9, the food information acquisition unit 114 executes character extraction process on the "image at time of storage" of the stored food by the image processing unit 120. Here, stored food is food that is determined to have been stored in the refrigerator compartment 10 by the storage/removal determination unit 113 in step S33 of FIG. 8. In addition, the "image at time of storage" of stored food is the "image at time of storage" captured and stored by the food information acquisition unit 114 in steps S20 and S21 of FIG. 7 for the food determined to have been stored in step S33 of FIG. 8.

In the next step S51, the food information acquisition unit 114 determines whether or not characters have been extracted from the "stored image" by the image processing unit 120. If characters have been extracted, the food information acquisition unit 114 proceeds to step S52, and if characters have not been extracted, the food information acquisition unit 114 proceeds to step S60.

In step S52, the food information acquisition unit 114 analyzes the text information extracted by the image processing unit 120. Then, in the following step S53, the food information acquisition unit 114 acquires information about the stored food, such as the food name, expiration date, place of origin, and content volume, from the text information. In the next step S54, the food information acquisition unit 114 records the acquired information about the stored food in the stored food information 132, and ends the "image analysis process."

In addition, in step S60, the food information acquisition unit 114 extracts an image portion of the stored food from the "stored food image" by the image processing unit 120. In the following step S61, the food information acquisition unit 114 calculates the degree of match between the extracted image portion of the stored food and sample images 134 (see FIG. 5) of various foods stored in advance in the memory unit 130.

In the next step S62, the food information acquisition unit 114 estimates and acquires the food name of the sample image 134 with the highest matching rate as the food name of the stored food, and proceeds to step S54. By processing steps S60 to S62, even if characters cannot be extracted from the "image at the time of storage" of the stored food, it is possible to estimate the food name of the stored food and record it in the stored food information 132.

Next, the "image analysis process" for the "image at time of leaving the warehouse" will be described with reference to FIG. 10. In step S70 of FIG. 10, the food information acquisition unit 114 executes character extraction process on the "image at time of leaving the warehouse" of the leaving food by the image processing unit 120. Here, the leaving food is the food that is determined to have been left from the refrigerator compartment 10 by the entry/exit determination unit 113 in step S40 of FIG. 8. In addition, the "image at time of leaving the warehouse" of the leaving food is the "image at time of leaving the warehouse" captured and stored by the food information acquisition unit 114 in steps S22 and S23 of FIG. 7 for the food that is determined to have been left in step S40 of FIG. 8.

In the next step S71, the food information acquisition unit 114 determines whether or not characters have been extracted from the "image at the time of leaving the warehouse" by the image processing unit 120. If characters have been extracted, the food information acquisition unit 114 proceeds to step S72, and if characters have not been extracted, the food information acquisition unit 114 proceeds to step S80.

In step S72, the food information acquisition unit 114 analyzes the text information extracted by the image processing unit 120. Then, in the following step S73, the food information acquisition unit 114 acquires information about the shipped food, such as the food name, expiration date, place of origin, and content volume, from the text information. In the next step S74, the food information acquisition unit 114 records the acquired information about the shipped food in the shipped food information 133, and ends the "image analysis process."

In step S80, the food information acquisition unit 114 extracts an image portion of the shipped food from the "image at time of shipment" by the image processing unit 120. In the following step S81, the food information acquisition unit 114 calculates the degree of match between the extracted image portion of the shipped food and the sample images 134 of various foods stored in advance in the memory unit 130.

The image analysis process in FIG. 9 is performed on the image at the time of entry when entry into the refrigerator compartment 10 is determined, and the image analysis process in FIG. 10 is performed on the image at the time of exit when exit from the refrigerator compartment 10 is determined. Therefore, when the right door 11 or the left door 12 is detected by the passing object detection unit 111 instead of food, it is not determined that an entry or exit has occurred, so the image analysis process is not performed (the captured image is invalid). This makes it possible to prevent erroneous food information from being obtained from the image portion of the right door 11 or the left door 12.

In the next step S82, the food information acquisition unit 114 estimates and acquires the food name of the sample image 134 with the highest matching rate as the food name of the shipped food, and proceeds to step S74. By processing steps S80 to S82, even if characters cannot be extracted from the "image at time of shipment" of the shipped food, it is possible to estimate the food name of the shipped food and record it in the shipped food information 133.

The image analysis process in FIG. 9 is performed on the image at the time of entry when entry into the refrigerator compartment 10 is determined, and the image analysis process in FIG. 10 is performed on the image at the time of exit when exit from the refrigerator compartment 10 is determined. Therefore, when the right door 11 or the left door 12 is detected by the passing object detection unit 111 instead of food, it is not determined that an entry or exit has occurred, so the image analysis process is not performed (the captured image is invalid). This makes it possible to prevent erroneous food information from being obtained from the image portion of the right door 11 or the left door 12.

[3-4. Weight sensor correction process]
The execution procedure of the "weight sensor calibration process" for correcting the detected weights of the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b will be described with reference to the flowchart shown in Fig. 11. The "weight sensor correction process" is executed by the capacity recognition unit 112. The "weight sensor correction process" is a process for correcting (calibrating) the detection errors of the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b that may occur due to changes over time or the like.

In step S90 of FIG. 11, when the in/out determination unit 113 determines that food has been stored in the refrigerator compartment (YES in step S33 of FIG. 8), the storage capacity recognition unit 112 advances the process to step S91. In step S91, the storage capacity recognition unit 112 determines whether or not information on the content volume of the stored food is recorded in the stored food information 132 (processing of step S54 of FIG. 9). Then, when information on the content volume of the stored food (the content volume of the food obtained in step S53 of FIG. 9) is recorded in the stored food information 132, the storage capacity recognition unit 112 advances the process to step S92.

In step S92, the storage amount recognition unit 112 calculates the ratio between the weight and content amount of the stored food recorded in the stored food information 132. The storage amount recognition unit 112 then sets the calculated ratio as the correction coefficient for the weight sensor used to detect the weight of the stored food. For example, when the stored food is stored on the upper shelf 13 and its weight is detected by the weight sensors 13a and 13b, the storage amount recognition unit 112 sets the calculated ratio as the correction coefficient for the detected weight of the weight sensors 13a and 13b.

In the next step S93, the storage amount recognition unit 112 stores the correction coefficient data in the memory unit 130, and proceeds to step S94 to end the "weight sensor correction process". Thereafter, the storage amount recognition unit 112 treats the detected weight of the weight sensors 13a and 13b multiplied by the correction coefficient as the detected weight. Similarly, when the stored food is stored on the middle shelf 14, the storage amount recognition unit 112 sets the correction coefficient for the weight sensors 14a and 14b, and when the stored food is stored on the lower shelf 15, the storage amount recognition unit 112 sets the correction coefficient for the weight sensors 15a and 15b.

On the other hand, if no content volume information is recorded in the stored food information 132 in step S91, the content volume recognition unit 112 proceeds to step S94 and ends the "weight sensor correction process." In this case, the correction coefficient is not calculated or set.

[3-5. Food management process inside the warehouse]
12 and 13, a procedure for executing the "in-fridge food management process" for managing information on food stored in the refrigerator compartment 10 will be described. The "in-fridge food management process" is executed by the food information management unit 115.

The food information management unit 115 manages information on the foods stored in the refrigerator compartment 10 using the refrigerator food list 131 (see FIG. 5). As shown in FIG. 14, the refrigerator food list 131 records the numbers (1, 2, 3, ...) assigned to the foods stored in the refrigerator compartment 10, and information on each numbered stored food, such as the food name, weight (first, current, consumed amount), storage/removal date/time (first storage, latest storage, latest storage), stock (1: available, 0: not available), expiration date, and place of origin. The food information management unit 115 generates and updates the refrigerator food list 131 based on the stored food information 132 and the stored food information 133 recorded by the food information acquisition unit 114.

In step S100 of FIG. 12, the food information management unit 115 determines whether the entry/exit determination unit 113 has determined that food has been entered into or removed from the refrigerator compartment 10, and when entry or exit has been determined, the process proceeds to step S101. In step S101, the food information management unit 115 determines whether entry has been determined, and when entry has been determined, the process proceeds to step S102. On the other hand, when entry has not been determined (when exit has been determined), the food information management unit 115 proceeds to step S110.

In step S102, the food information management unit 115 determines whether or not food has been removed from the refrigerator compartment 10 within a predetermined time. If food has been removed from the refrigerator compartment 10 within the predetermined time, the process proceeds to step S103. If food has not been removed from the refrigerator compartment 10 within the predetermined time, the process proceeds to step S111. Note that the fact that food has been removed from the refrigerator compartment 10 within the predetermined time can be recognized from the food in-storage list 131.

In step S103, the food information management unit 115 determines whether the matching rate between the food previously sent out and the food currently stored is equal to or greater than a predetermined value. Here, the food information management unit 115 compares the information on the previously sent out product recorded in the in-store food list 131 with the information on the currently stored product recorded in the stored food information 132, and calculates the matching rate between the previously sent out food and the food currently stored based on the matching status of the food name, expiration date, place of origin, etc.

Then, if the matching rate between the food previously shipped and the food currently stocked is equal to or greater than a predetermined value, the food information management unit 115 proceeds to step S104, and if the matching rate between the food previously shipped and the food currently stocked is less than the predetermined value, the food information management unit 115 proceeds to step S112.

Step S104 is the process carried out when the matching rate between the food previously sent out and the food currently stored is equal to or greater than a predetermined value, and it can be determined that the food previously sent out and the food currently stored are the same. In step S104, the food information management unit 115 updates the current weight, most recent stored date and time, and amount consumed (difference between the weight at the time of first storing and the current weight) of the food in the stored food list 131 that was sent out before a predetermined time based on the information recorded in the stored food information 132, and proceeds to step S105 in FIG. 13.

Step S112 is a process carried out when the matching rate between the food that was taken out and the food that was currently stored is less than a predetermined value, and it can be determined that the food that was taken out last time is different from the food that was currently stored. In step S112, the food information management unit 115 newly registers the stored food in the in-warehouse food list 131 based on the information recorded in the stored food information 132, and proceeds to step S105 in FIG. 12.

Step S111 is the process to be performed when no food is removed from the warehouse within a predetermined time. In step S111, the food information management unit 115 newly registers the stored food in the warehouse food list 131 based on the information recorded in the stored food information 132, and proceeds to step S105 in FIG. 12.

Step S110 is the process when it is determined that food has been removed from the refrigerator compartment 10. In step S110, the food information management unit 115 updates the current weight of the food (removed food) in the stored food list 131 to 0. The food information management unit 115 also updates the amount consumed of the food in the stored food list 131, updates the latest removal time based on the information recorded in the removed food information 133, and proceeds to step 105 in FIG. 13.

In step S105 of FIG. 13, the food information management unit 115 determines whether there is any food whose current weight is 0 grams as a result of the processing in step S110. If there is any food whose current weight is 0 grams, the food information management unit 115 proceeds to step S105, and if there is no food whose current weight is 0 grams, the food information management unit 115 proceeds to step S120.

In step S106, the food information management unit 115 updates the information of the corresponding food in the in-fridge food list 131 (food determined to have been taken out this time) to 'out of stock' (stock 0), and proceeds to step S107. In addition, in step S120, the food information management unit 115 updates the information of the corresponding food in the in-fridge food list 131 (food determined to have been received this time) to 'in stock' (stock 1), and proceeds to step S107.

In step S107, the food information management unit 115 determines whether or not expiration date information based on character recognition is recorded in the stored food information 132. If expiration date information is recorded in the stored food information 132, the food information management unit 115 proceeds to step S108, and if expiration date information is not recorded, the food information management unit 115 proceeds to step S121.

In step S108, the food information management unit 115 registers the expiration date of the food currently stored in the in-warehouse food list 131 based on the expiration date information recorded in the stored food information 132, and proceeds to step S109. In addition, in step S121, the food information management unit 115 refers to the expiration date list 135 stored in the memory unit 130 to search for the general expiration date of the relevant food.

As shown in FIG. 14, the expiration date list 135 is a list that associates food names with the expiration dates of the foods, and the food information management unit 115 searches for the expiration dates associated with the food names of the currently stored foods. In the following step S122, the food information management unit 115 registers the expiration date in the data of the corresponding food in the in-store food list 131, and proceeds to step S109.

In step S109, the food information management unit 115 transmits the updated in-storage food list 131 data to the management server 520 via the communication network 510 using the wireless communication unit 140, and ends the "in-storage food management process."

The management server 520 stores the data of the food list 131 received from the refrigerator 1 in a storage device (not shown) and transmits the consumption status of each product recognized from the food list 131, the number of days remaining until the expiration date, etc. to the user's terminal device 400. This allows the user to check the status of the products stored in the refrigerator 1 using the terminal device 400 even when they are away from home, etc.

4. Other embodiments
In the above embodiment, the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 based on the change in illuminance inside the refrigerator compartment 10 detected by the illuminance sensors 17a, 17b, and 17c. As another configuration of the passing object detection unit 111, an object passing through the opening 10a may be detected by extracting an image portion of the object near the opening 10a from the image captured by the camera 60.

Alternatively, as shown in Figures 3 to 5, an object sensor 80 (reflective or transmissive optical sensor, ultrasonic sensor, capacitance sensor, etc.) may be used that includes the opening 10a in its detection range and directly detects objects passing through the opening 10a. In this case, the detection range of the object sensor 80 may be set in front of the imaging range of the camera 60 in the direction from the opening 10a toward the inside of the refrigerator compartment 10. This makes it possible to prevent failure to capture an image of the food being stored due to the delay time between the detection of an object passing through the opening 10a and the execution of imaging by the camera 60.

When the opening/closing sensor 18a detects that the left door 12 is open, and when the opening/closing sensor 18b detects that the right door 11 is open, power may be started to be supplied to the camera 60, enabling the camera 60 to take an image. With this configuration, when the left door 12 and the right door are closed, power to the camera 60 can be cut off to save electricity. Also, by starting power to the camera 60 when the right door or the left door 12 is opened, it is possible to prevent failure to capture an image of food being stored due to a delay in starting the camera 60.

Furthermore, the food information acquisition unit 114 may be configured to invalidate image data captured by the camera 60 when the opening/closing sensor 18a detects that the left door 12 has switched from an open state to a closed state, or when the opening/closing sensor 18b detects that the right door 11 has switched from an open state to a closed state, within a predetermined time from when the passing object detection unit 111 detects an object passing through the opening 10a. With this configuration, when the left door 12 or the right door 11 is closed, when the left door 12 or the right door 11 is detected by the passing object detection unit 111 and imaged by the camera 60, it is possible to prevent erroneous food information from being extracted from the image of the left door 12 or the right door 11.

In the above embodiment, the storage amount recognition unit 112 recognized the amount of food stored on each shelf 13, 14, 15 of the refrigerator compartment 10 based on the weight detected by the weight sensors 13a, 13b, 14a, 14b, 15a, 15b. As another configuration of the storage amount recognition unit 112, the amount of food stored on each shelf 13, 14, 15 of the refrigerator compartment 10 may be recognized based on the illuminance detected by the illuminance sensors 17a, 17b, 17c with the lighting units 16a, 16b, 16c, 16d, 16e, 16f turned on. Alternatively, the amount of food stored on each shelf 13, 14, 15 of the refrigerator compartment 10 may be recognized by extracting an image portion of the food from an image of the inside of the refrigerator compartment 10 captured by the camera 60 or a separately placed camera. In this case, the camera 60 or a separately placed camera corresponds to the first camera of the present disclosure.

In the above embodiment, the image captured by the camera 60 was processed by the image processing unit 120, but the functions of the image processing unit 120 may be provided in the CPU 110, and the captured image may be processed by the CPU 110.

In the above embodiment, the data of the sample image 134 and the expiration date list 135 are stored in the memory unit 130, but they may also be stored in the management server 520 and downloaded from the management server 520 to the control unit 100 of the refrigerator 1 when used.

Although sample image 134 is stored in memory unit 130 or management server 520, an image recognition program that has previously learned the characteristics of each food or stored item, such as color, shape, and size, may be implemented in CPU 110, and the image portion of the stored food may be processed by the image recognition program to calculate a match rate and obtain an estimated food name of the stored food. This makes sample image 134 unnecessary, making it possible to reduce the storage capacity stored in memory unit 130 or management server 520 and also speeding up the recognition processing time for estimating the food name of the stored food. The food name estimation for out-of-stock foods can also be processed in a similar manner to the above.

In the above embodiment, the in/out determination unit 113 determines whether food is in or out of the refrigerator compartment 10 of the refrigerator 1, the food information acquisition unit 114 acquires information on the in/out food, and the food information management unit 115 manages information on the food stored in the refrigerator compartment 10. The switching compartment 20, freezer compartment 40, and vegetable compartment 50 are also provided with a passing object detection unit that detects objects passing through the openings 20a, 40a, and 50a, and a storage amount recognition unit that recognizes the amount of stored food, so that the in/out of food can be determined. In addition, a camera that captures an image of an area including the openings 20a, 40a, and 50a can be provided to acquire information on the in/out food from the captured image, and information on the food stored in the switching compartment 20, freezer compartment 40, and vegetable compartment 50 can be acquired and food can be managed.

As described above, the refrigerator according to the present invention can distinguish between the entry of food into the storage compartment and the removal of food from the storage compartment with a simple configuration while minimizing an increase in memory capacity, and can therefore be used to manage the food stored in the refrigerator.

REFRIGERATOR 10 Refrigerating compartment 10a Opening 11 Right door 12 Left door 13 Upper shelf 13 Shelf 13a, 13b Weight sensor 14 Middle shelf 14a, 14b Weight sensor 15 Lower shelf 15a, 15b Weight sensor 16a, 16b, 16c, 16d, 16r, 16f Illumination unit 17a, 17b, 17c Illuminance sensor 18a, 18b Opening/closing sensor 20 Switching compartment 20a Opening 22 Opening/closing sensor 30 Ice making compartment 30a Opening 32 Opening/closing sensor 32 Step 32 Opening/closing sensor 40 Step 40 Freezer compartment 40a Opening 42 Opening/closing sensor 50 Vegetable compartment 50a Opening 52 Opening/closing sensor 60 Camera 61 Compressor 62 Cooling fan 63 Cooler 64 Condenser 100 Control unit 110 CPU
111 Passing object detection unit 112 Storage volume recognition unit 113 Incoming/outgoing determination unit 114 Food information acquisition unit 115 Food information management unit 120 Image processing unit 121 Image memory 130 Storage unit 131 In-store food list 132 In-store food information 133 Out-store food information 134 Sample image 135 Expiration date list 140 Wireless communication unit 141 Date and time recognition unit 400 Terminal device 500 Gateway 510 Communication network 520 Management server

Claims (6)

a storage chamber having an opening;
a passing object detection unit that detects an object passing through the opening;
A first camera that captures an image of the inside of the storage chamber;
A second camera disposed so as to include the opening in its imaging range;
A storage unit;
a storage amount recognition unit that recognizes the amount of food stored in the storage chamber by extracting an image portion of the food from an image of the inside of the storage chamber captured by the first camera;
a food information acquisition unit that performs image capture using the second camera when an object passing through the opening is detected by the passing object detection unit, and acquires information about the food from the captured image of the second camera;
an entry/exit determination unit that determines whether food has been entered into or removed from the storage chamber based on a change in the amount of food recognized by the storage amount recognition unit when an object passing through the opening is detected by the passing object detection unit;
a food information management unit which, when the entry/exit determination unit determines that a food has been entered into or removed from the storage chamber, stores food entry/exit information based on the result of the determination in the memory unit; when the passing object detection unit detects an object passing through the opening, and when the entry/exit determination unit determines that a food has been entered into or removed from the storage chamber in response to the detection of the object and the food information acquisition unit acquires food information, associates the determined entry or removal information with the acquired food information and stores it in the memory unit; and, when the entry/exit determination unit determines that a food has been entered into the storage chamber and the food information acquisition unit acquires the food information, if a matching rate between information on a food that was removed within a predetermined time and information on a food that has been currently entered is equal to or greater than a predetermined value, updates information on a food that was removed before the predetermined time based on information on a food that has been currently entered, and stores it in the memory unit;
A refrigerator equipped with A time recognition unit that recognizes the current time and date,
The refrigerator according to claim 1, wherein when the input/output determination unit determines that food has been input to or output from the storage chamber, the food information management unit stores in the memory unit a date and time recognized by the date and time recognition unit at the time of the determination, which is included in the food input/output information. The refrigerator according to claim 1 or 2, wherein the passing object detection unit detects an object passing through the opening by extracting an image portion of the object from the image captured by the second camera. A wireless communication unit is provided.
4. The refrigerator according to claim 1, wherein the food information management unit transmits, to a management server via the wireless communication unit, information on the incoming or outgoing food determined by the incoming/outgoing food determination unit and data on a food list in the refrigerator based on the food information acquired by the food information acquisition unit. A management system including a refrigerator, a management server that communicates with the refrigerator, and a terminal device that communicates with the refrigerator or the management server,
The refrigerator includes:
a storage chamber having an opening;
a passing object detection unit that detects an object passing through the opening;
A first camera that captures an image of the inside of the storage chamber;
A second camera disposed so as to include the opening in its imaging range;
A storage unit;
a storage amount recognition unit that recognizes the amount of food stored in the storage chamber by extracting an image portion of the food from an image of the inside of the storage chamber captured by the first camera;
a food information acquisition unit that performs image capture using the second camera when an object passing through the opening is detected by the passing object detection unit, and acquires information about the food from the captured image of the second camera;
an entry/exit determination unit that determines whether food has been entered into or removed from the storage chamber based on a change in the amount of food recognized by the storage amount recognition unit when an object passing through the opening is detected by the passing object detection unit;
When the entry/exit determination unit determines that food has been entered into or removed from the storage chamber, food entry/exit information based on the result of the determination is stored in the memory unit, and when the passing object detection unit detects an object passing through the opening, the entry/exit determination unit determines that food has been entered into or removed from the storage chamber in response to the detection of the object, and when food information is acquired by the food information acquisition unit, the determined entry or exit information is associated with the acquired food information and stored in the memory unit, and the entry/exit determination unit a food information management unit that, when it is determined that a food has been stored in the storage room by the food information acquisition unit and a matching rate between information of the food that has been removed within a predetermined time and information of the food that has been stored this time is equal to or greater than a predetermined value, updates information of the food that has been removed before the predetermined time based on information of the food that has been stored this time and stores the updated information in the storage unit, and transmits to the management server information on the food that has been stored or removed determined by the storage/return determination unit and data on a list of food items stored in the storage unit based on information on the food that has been acquired by the food information acquisition unit,
The management server receives and stores the data of the food list transmitted from the refrigerator,
The terminal device receives and acquires the data of the stored food list from the management server or the refrigerator, and displays information about the foods stored in the refrigerator on a display unit of the terminal device based on the stored food list. a storage chamber having an opening;
a passing object detection unit that detects an object passing through the opening;
A first camera that captures an image of the inside of the storage chamber;
A storage unit;
A second camera disposed so as to include the opening in its imaging range;
A storage unit;
A CPU constituting a control unit of a refrigerator comprising:
a storage amount recognition unit that recognizes the amount of food stored in the storage chamber by extracting an image portion of the food from an image of the inside of the storage chamber captured by the first camera;
a food information acquisition unit that performs image capture using the second camera when an object passing through the opening is detected by the passing object detection unit, and acquires information about the food from the captured image of the second camera;
an entry/exit determination unit that determines whether food has been entered into or removed from the storage chamber based on a change in the amount of food recognized by the storage amount recognition unit when an object passing through the opening is detected by the passing object detection unit;
a food information management unit which, when the entry/exit determination unit determines that a food has been entered into or removed from the storage chamber, stores food entry/exit information based on the result of the determination in the memory unit; when the passing object detection unit detects an object passing through the opening, and when the entry/exit determination unit determines that a food has been entered into or removed from the storage chamber in response to the detection of the object and the food information acquisition unit acquires food information, associates the determined entry or removal information with the acquired food information and stores it in the memory unit; and, when the entry/exit determination unit determines that a food has been entered into the storage chamber and the food information acquisition unit acquires the food information, if a matching rate between information on a food that was removed within a predetermined time and information on a food that has been currently entered is equal to or greater than a predetermined value, updates information on a food that was removed before the predetermined time based on information on a food that has been currently entered, and stores it in the memory unit;
A program that makes it work.

Images