CN114704997A

CN114704997A - Refrigerator with a door

Info

Publication number: CN114704997A
Application number: CN202210344474.2A
Authority: CN
Inventors: 村上和宏
Original assignee: Panasonic Intellectual Property Management Co Ltd
Current assignee: Panasonic Intellectual Property Management Co Ltd
Priority date: 2017-10-06
Filing date: 2018-09-30
Publication date: 2022-07-05
Also published as: JP2019070478A; JP7012263B2; CN109631485B; CN109631485A

Abstract

The refrigerator of the present invention performs camera shooting at the 1 st shooting time, and stores the shot image at the 1 st shooting time in a storage unit as a storage-time image. The refrigerator according to the present invention captures an image by the camera at the 2 nd capturing time, and stores the captured image at the 2 nd capturing time in the storage unit as an outgoing image. The refrigerator according to the present invention acquires information on the food items stored in the storage room and the food items removed from the storage room based on the storage-time image and the removal-time image.

Description

Refrigerator with a door

The application is a divisional application of patent applications with application date of 2018, 9 and 30 months, application number of 201811156794.5 and invention name of 'refrigerator'.

Technical Field

The present invention relates to a refrigerator.

Background

Japanese patent application laid-open No. 2016 and 57022 (hereinafter, referred to as document 1) proposes a refrigerator in which an in-compartment camera is provided at the center of a food storage compartment, the inside of the storage compartment is photographed by the in-compartment camera when a door of the storage compartment is opened and closed, and an image at the time of storing food and a current image are output side by side from time-series photographed images. In the refrigerator described in document 1, depending on the arrangement state of the food in the storage room, the food may overlap in the imaging direction of the in-refrigerator camera and some of the food may not be recognized from the captured image.

Therefore, japanese patent application laid-open No. 2015-81762 (hereinafter, referred to as document 2) proposes a refrigerator provided with a switch for detecting opening and closing of a door of a storage chamber; and a camera having an opening of the storage chamber as a photographing range, and photographing foods put in or out of the storage chamber through the opening when the door is opened, instead of photographing foods stored in the storage chamber.

Disclosure of Invention

In the refrigerator described in the above-mentioned document 2, when it is detected that the door of the storage chamber is opened, the food passing through the opening of the storage chamber is photographed. Therefore, depending on the timing from the opening of the door to the storage or retrieval of the food, the user may not be able to photograph the food. Further, when a part of a door pocket (door pocket) or the like is photographed instead of food, there is a possibility that an article other than food may be erroneously recognized as food.

In view of the above-described technical background, an object of the present invention is to provide a refrigerator capable of more reliably obtaining information on food items stored in a storage chamber and food items taken out of the storage chamber.

In order to solve the conventional problems, a refrigerator according to the present invention includes: a storage chamber having an opening; an opening/closing part for opening/closing the opening; an opening/closing sensor for detecting an opening/closing state of the opening/closing section; a camera capable of capturing an image of an imaging range including the opening; a storage unit; and a food information acquisition unit for detecting a period from when the open/close sensor detects that the open/close unit is switched from the closed state to the open state to when the open/close sensor detects that the open/close unit is switched from the open state to the closed state, shooting by the camera at the 1 st shooting time set by the time when the food stored in the storage chamber passes through the opening, storing the shot image at the 1 st shooting time in the storage part as a storage-time image, the camera is used for shooting at the 2 nd shooting time set by the time of the food delivered from the storage chamber passing through the opening, the shot image at the 2 nd shooting time is taken as the image at the time of delivery and stored in the storage part, and acquiring information of the food put in the storage chamber and the food taken out from the storage chamber based on the image at the time of putting in the storage and the image at the time of taking out.

Thus, the food information acquiring unit stores the warehousing-time image captured at the 1 st imaging time when the food warehoused in the storage chamber is assumed to pass through the opening of the storage chamber and the delivery-time image captured at the 2 nd imaging time when the food delivered from the storage chamber is assumed to pass through the opening in the storage chamber in the storage unit. The food information acquiring unit can more reliably acquire information on the food put in the storage chamber and the food taken out of the storage chamber based on the image at the time of storage and the image at the time of delivery, which have a high possibility of including the image portion of the food.

According to the refrigerator of the present invention, by assuming the timing at which the food put in and out of the storage chamber passes through the opening of the storage chamber and photographing the vicinity of the opening with the camera, it is possible to more reliably acquire information on the food put in and out of the storage chamber.

Drawings

Fig. 1 is an explanatory diagram of a configuration in which the status of food items stored in a refrigerator provided in a house is confirmed by a terminal device.

Fig. 2 is an explanatory view of an external appearance of the refrigerator.

Fig. 3 is an explanatory diagram showing an internal structure of the refrigerator in a cross-sectional view from the right side.

Fig. 4 is an explanatory view showing an internal structure of the refrigerating compartment in a sectional view from the front.

Fig. 5 is a structural diagram of the control unit.

Fig. 6 is a flowchart showing the overall processing of the control unit.

Fig. 7 is a flowchart of the image pickup processing.

Fig. 8 is a flowchart of the weight information processing.

Fig. 9 is a flowchart of the analysis processing of the image at the time of warehousing.

Fig. 10 is a flowchart of the image analysis processing at the time of shipment.

Fig. 11 is a flowchart of the weight sensor correction process.

Fig. 12 is a flowchart of the in-house food management process 1.

Fig. 13 is a flow chart of the in-house food management process 2.

Fig. 14 is an explanatory view of a list of foods in a warehouse and a reference list of shelf lives.

Fig. 15 is a flowchart of another embodiment of the image pickup process.

Detailed Description

The refrigerator of claim 1 includes: a storage chamber having an opening; an opening/closing part for opening/closing the opening; an opening/closing sensor for detecting an opening/closing state of the opening/closing section; a camera capable of capturing an image of an imaging range including the opening; a storage unit; and a food information acquisition unit for detecting a period from when the open/close sensor detects that the open/close unit is switched from the closed state to the open state to when the open/close sensor detects that the open/close unit is switched from the open state to the closed state, shooting by the camera at the 1 st shooting time set by the time when the food stored in the storage chamber passes through the opening, storing the shot image at the 1 st shooting time in the storage part as a storage-time image, shooting by the camera at the 2 nd shooting time set by the time when the food taken out of the storage chamber passes through the opening is assumed, storing the shot image at the 2 nd shooting time in the storage part as an image during delivery, and acquiring information of the food put in the storage chamber and the food taken out from the storage chamber based on the image at the time of putting in the storage and the image at the time of taking out.

According to the invention of claim 1, the food information acquiring unit stores the warehousing-time image captured at the 1 st imaging time when the food warehoused in the storage chamber is assumed to pass through the opening of the storage chamber and the delivery-time image captured at the 2 nd imaging time when the food delivered from the storage chamber is assumed to pass through the opening in the storage chamber in the storage unit. The food information acquiring unit can more reliably acquire information on the food put in the storage chamber and the food taken out of the storage chamber based on the image at the time of storage and the image at the time of delivery, which have a high possibility of including the image portion of the food.

The refrigerator of claim 2 includes: an illuminance sensor for detecting illuminance in the storage chamber; and an imaging time setting unit that sets a 1 st reference illuminance and a 2 nd reference illuminance based on the illuminance detected by the illuminance sensor when the opening/closing unit is switched from the closed state to the open state by the opening/closing sensor, sets the 1 st imaging time based on a time when the illuminance detected by the illuminance sensor becomes equal to or less than the 1 st reference illuminance, and sets the 2 nd imaging time based on a time when the illuminance detected by the illuminance sensor becomes equal to or more than the 2 nd reference illuminance after the illuminance detected by the illuminance sensor becomes equal to or less than the 1 st reference illuminance.

According to the invention of claim 2, the illuminance in the storage chamber changes according to the presence of an object (food, a user's hand, or the like) in the storage chamber. Therefore, the 1 st imaging time at which the food put into the storage chamber is assumed to pass through the opening and the 2 nd imaging time at which the food taken out of the storage chamber is assumed to pass through the opening can be appropriately set according to the change in the illuminance in the storage chamber detected by the illuminance sensor.

The refrigerator according to claim 3 includes an imaging time setting unit that sets, as the 1 st imaging time, a time at which a 1 st predetermined time elapses from a time at which the opening/closing sensor detects that the opening/closing unit is switched from the closed state to the open state, and sets, as the 2 nd imaging time, a time earlier than a 2 nd predetermined time by which the opening/closing sensor detects that the opening/closing unit is switched from the open state to the closed state.

According to the invention of claim 3, since the time from when the user opens the opening/closing unit of the storage chamber to when the food is put in storage and the time from when the user takes the food out of the storage chamber to when the opening/closing unit is closed are maintained in the cold-insulated state in the storage chamber, a short fixed time can be expected. Therefore, the 1 st and 2 nd photographing times can be set by a simple configuration in which the 1 st and 2 nd predetermined times are set.

In the refrigerator according to claim 4, the food information acquiring unit repeatedly performs shooting by the camera and stores a shot image in the storage unit during a period from when the 1 st shooting time elapses until the open/close sensor detects that the open/close unit is switched from the open state to the closed state, and when the open/close sensor detects that the open/close unit is switched from the open state to the closed state, stores a shot image, which is shot at a time earlier than a time at which the open/close sensor detects that the open/close unit is switched from the open state to the closed state by the 2 nd predetermined time and stored in the storage unit, as the delivery-time image.

According to the 4 th aspect of the present invention, the food information acquiring unit repeatedly performs shooting by the camera and stores a shot image in the storage unit during a period from when the 1 st shooting time elapses to when the open/close sensor detects that the open/close unit is switched from the open state to the closed state. This makes it possible to store, as the image at the time of shipment, the captured image at a time earlier than the time at which the open/close sensor detects that the open/close unit has been switched from the open state to the closed state by the 2 nd predetermined time.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the embodiment.

[1. mode of use of refrigerator ]

Fig. 1 shows a usage of the refrigerator according to the present embodiment. The refrigerator 1 of the present embodiment is installed in a house H and has a function of communicating with the management server 520 through the gateway 500 and the communication network 510. Further, the refrigerator 1 has a function of communicating with a terminal device (smartphone, tablet terminal, or the like) 400. The terminal apparatus 400 has a function of performing communication with the management server 520 through the gateway 500 and the communication network 510.

The control unit 100 provided in the refrigerator 1 controls the operation of the entire refrigerator 1, and generates and transmits an in-refrigerator food list 131 indicating food information stored in the storage chamber of the refrigerator 1 to the management server 520. The management application of the refrigerator running in the terminal device 400 acquires data of the in-refrigerator food list 131 from the management server 520 or acquires data of the in-refrigerator food list 131 from the refrigerator 1. Then, the management application displays the food information stored in the refrigerator 1 on the display unit based on the in-house food list 131. The user P of the refrigerator 1 can confirm the status (food name, expiration date, remaining amount, and the like) of the food stored in the refrigerator 1 by visually recognizing the display of the terminal device 400.

[2. Structure of refrigerator ]

The structure of the refrigerator 1 will be described with reference to fig. 2 to 5. Fig. 2 is an explanatory diagram showing an external appearance of the refrigerator 1. As shown in fig. 2, the refrigerator 1 includes a main body 2 opened at a front surface. In main casing 2, refrigerating room 10 (corresponding to a storage room of the present invention), ice making room 30, switching room 20 provided side by side with ice making room 30 and capable of changing the temperature in the interior, freezing room 40, and vegetable room 50 are formed.

A rotary right door 11 and a rotary left door 12 (corresponding to an opening/closing portion of the present invention) are provided at an opening portion in the front of the refrigerating compartment 10.

Drawers

21, 31, 41, and 51 for storing foods are provided in the switching compartment 20, the ice making compartment 30, the freezing compartment 40, and the vegetable compartment 50, respectively.

Next, fig. 3 is an explanatory diagram illustrating an internal configuration of the refrigerator 1 in a sectional view from the right side, and fig. 4 is an explanatory diagram illustrating an internal configuration of the refrigerating room 10 in a sectional view from the front side. Referring to fig. 3 and 4, in the refrigerating compartment 10, an upper shelf 13, a middle shelf 14, and a lower shelf 15 are disposed to divide the refrigerating compartment 10 into sub-sections. In addition,

weight sensors

13a, 13b that detect the weight of food placed on the upper shelf 13 are also provided in the refrigerating compartment 10;

weight sensors

14a, 14b that detect the weight of food placed on the middle shelf 14; and

weight sensors

15a, 15b that detect the weight of food placed on the lower shelf 15. Further, in refrigerating room 10, illumination units 16a to 16f for illuminating the inside of refrigerating room 10, illuminance sensors 17a to 17c for detecting the illuminance in refrigerating room 10, and camera 60 are provided.

The camera 60 is disposed near the opening 10a at the upper part of the refrigerating room 10 so that the opening 10a is included in the imaging range, and images the range from the opening 10a to the vicinity of the front end of each

shelf

13, 14, 15. Further, a compressor 61, a cooling fan 62, a cooler 63, and a condenser 64, which are auxiliary devices constituting the refrigeration cycle, are provided in the refrigerator 1. Refrigerating room duct 70 and cold air discharge ports 71 to 73 for allowing cold air to flow therethrough are also disposed on the rear surface of refrigerating room 10.

In refrigerating room 10, opening/closing sensor 18b for detecting opening/closing of right door 11 and opening/closing sensor 18a for detecting opening/closing of left door 12 are provided. The opening/closing sensor 18b outputs a closing detection signal when the right door 11 is closed, and outputs an opening detection signal when the right door 11 is opened. Similarly, the open-close sensor 18a outputs a close detection signal when the left door 12 is closed, and outputs an open detection signal when the left door 12 is opened.

An opening/closing sensor 22 for detecting opening/closing of the opening 20a is provided in the switching chamber 20. The open/close sensor 22 outputs a close detection signal when the drawer 21 is stored in the switching chamber 20, and outputs an open detection signal when the drawer 21 is not stored in the switching chamber 20.

Similarly, the ice making compartment 30 is provided with an opening/closing sensor 32 for detecting opening/closing of the opening 30a by the drawer 31. In addition, an opening/closing sensor 42 for detecting opening/closing of drawer 41 to opening 40a is provided in freezing room 40, and an opening/closing sensor 52 for detecting opening/closing of drawer 51 to opening 50a is provided in vegetable room 50. The open/

close sensors

32, 42, 52 output a close detection signal and an open detection signal, as with the open/close sensor 22.

Next, fig. 5 is a configuration diagram of the control unit 100. Referring to fig. 5, the control Unit 100 is an electronic circuit Unit including a CPU (Central Processing Unit) 110, an image Processing Unit 120, a storage Unit 130, a wireless communication Unit 140, a date and time recognition Unit 141, an interface circuit (not shown), and the like.

The CPU110 functions as a passage object detection unit 111, a storage amount recognition unit 112, an entry/exit determination unit 113, a food information acquisition unit 114, a food information management unit 115, and an imaging time setting unit 116 by running a control program of the refrigerator 1 stored in the storage unit 130.

The control unit 100 is connected to the opening/

closing sensors

18a, 18b, 22, 32, 42, 52, the

illuminance sensors

17a, 17b, 17c, the

weight sensors

13a, 13b, 14a, 14b, 15a, 15b, the camera 60, the

illumination units

16a, 16b, 16c, 16d, 16e, 16f, and the refrigeration cycle

auxiliary machines

61, 62, 63, 64.

Detection signals of the opening/

closing sensors

18a, 18b, 22, 32, 42, 52, the

illuminance sensors

17a, 17b, 17c, the

weight sensors

13a, 13b, 14a, 14b, 15a, 15b, and the like are input to the control unit 100. Further, the operations of the

illumination units

16a, 16b, 16c, 16d, 16e, and 16f, the refrigeration cycle

auxiliary machines

61, 62, 63, and 64, and the like are controlled based on control signals output from the control unit 100.

Further, the image processing unit 120 controls the image capturing by the camera 60 according to a control signal output from the control unit 100. The image processing unit 120 converts the video signal output from the camera 60 into a digital tuning signal to generate a captured image, and stores data of the captured image in the image memory 121. The image processing unit 120 processes the captured image stored in the image memory 121 based on a control signal output from the CPU 110.

The date and time identifying unit 141 counts a clock signal output from a clock circuit (not shown) provided in the control unit 100 to identify the current date and time (date and time). Further, the current date and time may be identified by acquiring current date and time information from the management server 520 through communication with the management server 520.

The storage unit 130 stores data of a food list 131 in the refrigerator, and also stores data of stored food information 132 in which food information stored in the refrigerating compartment 10 is recorded and data of delivered food information 133 in which food information delivered from the refrigerating compartment 10 is recorded. In addition, the storage unit 130 also stores data of sample images 134 of various foods and a standard shelf life list 135 indicating the general shelf lives of the various foods.

The object detection unit 111 detects an object passing through the opening 10a of the refrigerating compartment 10 based on the change in illuminance in the refrigerating compartment 10 detected by the

illuminance sensors

17a, 17b, and 17 c. The storage amount recognition part 112 recognizes the amount of food items stored on the upper shelf 13, the middle shelf 14, and the lower shelf 15 of the refrigerating compartment 10 by weight based on the detection signals of the

weight sensors

13a, 13b, 14a, 14b, 15a, and 15 b.

The storage amount recognition unit 112 recognizes the weight of the food stored on the upper shelf 13 based on the detection signals of the

weight sensors

13a and 13b, and recognizes the weight of the food stored on the middle shelf 14 based on the detection signals of the

weight sensors

14a and 14 b. The storage amount recognition unit 112 detects the weight of the food stored on the lower shelf 15 based on the detection signals of the

weight sensors

15a and 15 b.

When the object passing through the opening 10a of the refrigerating compartment 10 is detected by the object detection section 111, the in-out determination section 113 determines whether food is put in the refrigerating compartment 10 or taken out of the refrigerating compartment 10 based on the change in the amount (weight) of the food placed on each

shelf

13, 14, 15 recognized by the storage amount recognition section 112.

The food information acquiring unit 114 photographs an object passing through the opening 10a of the refrigerating compartment 10 with the camera 60. Then, when the in-out determination unit 113 determines that food is put into the refrigerating compartment 10 or food is taken out of the refrigerating compartment 10, an image portion of the food is extracted from the photographed image.

The food information acquisition part 114 recognizes characters included in the image portion of the food, or extracts the characteristics of the image portion of the food, thereby acquiring information such as the name, net content, origin, and expiration date of the food put in or out of stock. Then, for the food determined as being in stock (stock food), the food information acquisition unit 114 records the acquired information in the stock food information 132, and for the food determined as being out of stock (out-of-stock food), the food information acquisition unit 114 records the acquired information in the out-of-stock food information 133.

The food information management unit 115 updates the food information recorded in the in-storage food list 131 based on the weight of the food in the refrigerating room 10 recognized by the storage amount recognition unit 112, the stored food information 132 and the delivered food information 133 recorded by the food information acquisition unit 114, and the like.

The imaging time setting unit 116 sets the time at which the camera 60 captures an image until the right door 11 and the left door 12 are both closed when the open/

close sensors

18a and 18b detect that the right door 11 or the left door 12 is switched from the closed state to the open state. Imaging time setting unit 116 sets the 1 st imaging time at which the food stored in refrigerating room 10 is assumed to pass through opening 10a, and the 2 nd imaging time at which the food taken out of refrigerating room 10 is assumed to pass through opening 10 a.

[3. treatment by the control Unit ]

Next, with reference to fig. 6 to 14, a process performed by the control unit 100 for managing food items stored in the refrigerating compartment 10 will be described.

Fig. 6 is a flowchart showing an outline of a series of processing executed by the control unit 100. The control unit 100 executes the "image capturing process" in step S1, the "weight information process" in step S2, the "image analysis process" in step S3, the "weight sensor correction process" in step S4, and the "in-house food management process" in step S5, respectively. Next, the details of each process will be described.

[3-1. image pickup processing ]

The execution procedure of the "image pickup processing" will be described with reference to the flowchart shown in fig. 7. The "imaging process" is performed by the object detection unit 111, the food information acquisition unit 114, and the imaging time setting unit 116. When it is recognized in step S10 that one or both of right door 11 and left door 12 of refrigerating room 10 has switched from the closed state to the open state based on the detection signals of opening/

closing sensors

18a and 18b, food information acquiring unit 114 performs shooting with camera 60 at the 1 st shooting time and the 2 nd shooting time until it is recognized in step S15 that both of right door 11 and left door 12 have switched to the closed state.

When recognizing that one or both of right door 11 and left door 12 of refrigerator compartment 10 have been switched from the closed state to the open state in step S10, food item information acquisition unit 114 proceeds to the process in step S10. Steps S11 to S14 are processes performed by the object detector 111 and the imaging time setting unit 116. In the present embodiment, the imaging time setting unit of the present invention is configured by a combination of the object detection unit 111 and the imaging time setting unit 116.

In step S11, the object detection unit 111 detects the illuminance of each of the upper shelf 13, the middle shelf 14, and the lower shelf 15 of the refrigerating compartment 10 (the illuminance of each space defined by the shelves) by the

illuminance sensors

17a, 17b, and 17 c. Then, the object detection unit 111 sets the detected illuminance on the upper shelf 13 as the upper reference illuminance, sets the illuminance on the middle shelf 14 as the middle reference illuminance, and sets the illuminance on the lower shelf 15 as the lower reference illuminance. The data of the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance are stored in the storage unit 130 by the object detection unit 111.

The upper reference illuminance, the middle reference illuminance, and the lower reference illuminance refer to the illuminance of each

shelf

13, 14, 15 in a state where one or both of the right door 11 and the left door 12 of the refrigerating compartment 10 are opened and an object (a user's hand, food, or the like) passes through the opening 10a of the refrigerating compartment 10. In addition, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance may be set to be slightly lower or higher than the illuminance detected by each of the

illuminance sensors

17a, 17b, and 17c, in consideration of the error and variation in the detected illuminance.

The upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, which are used in step S13 described later, may be set to different values from the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, which are used in step S14. In this case, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S13 correspond to the 1 st reference illuminance of the present invention, and the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S14 correspond to the 2 nd reference illuminance of the present invention.

In the next step S12, the object detection unit 111 detects the illuminance of each of the upper shelf 13, the middle shelf 14, and the lower shelf 15 by the

illuminance sensors

17a, 17b, and 17 c. Then, in the next step S13, the object detection unit 111 determines whether or not there is a shelf whose detected illuminance is lower than the reference illuminance by a predetermined level or more. Specifically, the object detection unit 111 determines whether or not the following conditions (1) to (3) are satisfied.

(1) The illuminance detected by the illuminance sensor 17a is changed from the "upper reference illuminance" to the "upper reference illuminance- α" or less.

(2) The illuminance detected by the illuminance sensor 17b is changed from "middle reference illuminance" to "middle reference illuminance- α" or less.

(3) The detected illuminance by the illuminance sensor 17c changes from the "lower reference illuminance" to the "lower reference illuminance- α" or less.

Wherein, α: the threshold value setting constant is determined by experiments, computer simulations, or the like, assuming that the illuminance decreases when an object (food, a user's hand, or the like) enters the upper shelf 13, the middle shelf 14, or the lower shelf 15. Further, α may be a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance individually.

When at least one of the above-described (1) to (3) is satisfied, the object detection unit 111 determines that the object has passed through the opening 10 a. Here, when at least one of the above-described items (1) to (3) is satisfied, it is assumed that the user inserts his or her hand into refrigerating room 10 (enters) or takes out food from refrigerating room 10 (leaves), or performs a change in the position of food in refrigerating room 10, for example. In this case, it can be determined by object detection unit 111 that an object (a user's hand, food in the user's hand, or the like) has passed through opening 10a of refrigerating room 10.

Therefore, in step S13, the imaging time setting unit 116 sets the time at which at least one of the above (1) to (3) is determined to be satisfied by the object detection unit 111 as the 1 st imaging time, and the process proceeds to step S20. The 1 st photographing time is a time set on the assumption that the food put in the refrigerating compartment 10 passes through the opening 10 a. In step S13, the time at which a predetermined time has elapsed since the object detection unit 111 determined that at least one of the above (1) to (3) is established may be set as the 1 st imaging time.

On the other hand, when none of the above (1) to (3) is satisfied, the imaging time setting unit 116 proceeds to the process of step S14. Step S14 is a process performed by the object detection unit 111 and the imaging time setting unit 116. The object detection unit 111 determines whether or not there is a shelf where the detected illuminance is changed from "reference illuminance- β" or lower to "reference illuminance" or higher. Specifically, the object detection unit 111 determines whether or not the following conditions (4) to (6) are satisfied.

(4) The illuminance detected by the illuminance sensor 17a is changed from "upper reference illuminance- β" or lower to "upper reference illuminance" or higher.

(5) The illuminance detected by the illuminance sensor 17b is changed from "the middle reference illuminance- β" or lower to "the middle reference illuminance" or higher.

(6) The illuminance detected by the illuminance sensor 17c changes from "lower reference illuminance- β" or lower to "lower reference illuminance" or higher.

Wherein, the ratio of beta: the threshold value determination constant is determined by experiments, computer simulations, or the like, assuming that the illuminance increases when an object (food, a user's hand, or the like) entering the upper shelf 13, the middle shelf 14, or the lower shelf 15 passes through the opening 10a and comes out of the refrigerating compartment 10. Note that β may be a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance.

When at least one of the above-described items (4) to (6) is satisfied, object detection unit 111 determines that an object (food, a user's hand, or the like) present in refrigerator compartment 10 has been taken out of refrigerator compartment 10 and passed through opening 10 a. Here, when at least one of the above-described items (4) to (6) is satisfied, it is assumed that the user retracts the hand that has been inserted into refrigerating room 10 in order to put food into refrigerating room 10 (put in storage), to take food out of refrigerating room 10 (put out of storage), to change the position of food stored in refrigerating room 10, or the like. In this case, it can be determined by object detection unit 111 that an object (a user's hand, food in the user's hand, or the like) has passed through opening 10a of refrigerating room 10.

Then, in step S14, the image capturing time setting unit 116 sets the time determined by the object detection unit 111 that at least one of the above-described (4) to (6) is satisfied as the 2 nd image capturing time, and the process proceeds to step S22. The 2 nd photographing time is a time set on the assumption that the food taken out of the refrigerating compartment 10 passes through the opening 10 a. In step S14, the 2 nd shooting time may be set to a time when a predetermined time has elapsed from the time when the object detection unit 111 determines that at least one of the above (4) to (6) is established.

On the other hand, when none of the above (4) to (6) is satisfied, the object detection unit 111 proceeds to the process of step S15. In step S15, object detection unit 111 determines whether or not both right door 11 and left door 12 of refrigerating compartment 10 are closed, based on the detection signals of opening/

closing sensors

18a and 18 b. When both the right door 11 and the left door 12 are closed by the object detection unit 111, the "image pickup processing" is ended. On the other hand, when at least one of the right door 11 and the left door 12 is opened, the object detection unit 111 proceeds to the process of step S12.

Next, steps S20 to S21 and steps S22 to S23 are processes performed by the food information acquisition unit 114. In step S20, food information acquiring unit 114 captures an image of a range including opening 10a of refrigerating room 10 with camera 60 by image processing unit 120. In the next step S21, the food information acquisition unit 114 stores the image data captured in step S20 in the storage unit 130 as "image at the time of warehousing" data, and the process proceeds to step S15. Further, it is also possible to perform imaging a plurality of times and store a plurality of image data as "image at storing time" data in the storage unit 130.

In step S22, food information acquiring unit 114 uses image processing unit 120 to capture an image of a region including opening 10a of refrigerating compartment 10 with camera 60. In the next step S23, the food information acquisition unit 114 stores the image data captured in step S22 in the storage unit 130 as "images at the time of shipment" data, and the process proceeds to step S15. Further, it is also possible to perform shooting a plurality of times and store a plurality of image data in the storage unit 130 as "image at the time of shipment".

Based on the "image pickup processing" shown in fig. 7, the object detection unit 111 detects an object passing through the opening 10a of the refrigerating compartment 10 based on the change in illuminance of the upper shelf 13, the middle shelf 14, or the lower shelf 15 detected by the

illuminance sensors

17a, 17b, and 17 c.

The storage unit 130 stores a "storage-time image" as a captured image of the vicinity of the opening 10a at the time when the passage of the object through the opening 10a and the entry of the object into one of the

shelves

13, 14, and 15 is detected by the object detection unit 111. The "image at the time of shipment" as a captured image of the vicinity of the opening 10a at the time when the object detection unit 111 detects the object passing through the opening 10a after being taken out from any of the

shelves

13, 14, and 15 is stored in the storage unit 130.

In addition, in step S15 of fig. 7, when an object passing through the opening 10a is detected a plurality of times in step S13 before closing of the right door 11 and the left door 12 is detected, a plurality of "images at the time of entering the garage" are captured and stored in steps S20 to S21. The "image analysis processing" is executed for each "warehousing-time image" of the food determined to be warehoused according to the "weight information processing" described later.

Similarly, in step S15 of fig. 7, when an object passing through the opening 10a is detected a plurality of times in step S14 before the closing of the right door 11 and the left door 12 is detected, a plurality of "images at the time of delivery" are captured and stored in steps S22 to S23. The "image analysis processing" is executed for each "outgoing image" of the food determined to be outgoing from the warehouse based on the "weight information processing" described later.

[3-2. weight information processing ]

Next, the execution procedure of the "weight processing" will be described based on the flowchart shown in fig. 8. The "weight information processing" is executed by the storage/retrieval determining unit 113 and the food information acquiring unit 114.

Steps S30 to S33 and S40 in fig. 8 are processes performed by the in-and-out determination unit 113. In step S30, the storage/retrieval determination unit 113 recognizes the weight of each

shelf

13, 14, 15 by the storage amount recognition unit 112 and stores the weight in the storage unit 130. In the next step S31, when the object passing through the opening 10a of the refrigerating room 10 is detected by the object detection section 111 in step S13 or step S14 in fig. 7 (when an object is put into or taken out of the refrigerating room 10), the in-and-out storage determination section 113 proceeds to the process in step S32.

In step S32, the storage/retrieval determination unit 113 identifies the weight of each

shelf

13, 14, 15 by the storage amount identification unit 112. In the next step S33, the storage/retrieval determination unit 113 determines whether or not there is a shelf whose weight is increased as identified in step S32, for the weight identified in step S30. If there is a shelf with an increased weight (in this case, it can be determined that the food has been put into the refrigerating compartment 10), the in-out determination section 113 proceeds to the process of step S34.

On the other hand, when there is no shelf with a weight increase, the storage/retrieval determination unit 113 proceeds from step S33 to step S40, and determines whether or not there is a shelf with a weight decrease recognized in step S32, with respect to the weight recognized in step S30. If there is a shelf with a reduced weight (in this case, it can be determined that the food has been taken out of the refrigerating compartment 10), the in-out determination unit 113 proceeds to the process of step S41. In addition, when there is no shelf with a weight increase (in this case, it can be determined that there is no food put into the refrigerating compartment 10 or food is taken out from the refrigerating compartment 10), the in-out compartment determining section 113 proceeds to the processing of step S36, and ends the "weight information processing".

Steps S34 to S35 and step S41 are processes performed by the food information acquisition unit 114. In step S34, the food information acquisition unit 114 estimates the storage position of the warehoused food from the weight of each

shelf

13, 14, 15 identified by the storage amount identification unit 112.

In the next step S35, the food information acquiring unit 114 records the weight of the warehoused food and the information on the storage position in the warehouse food information 132 (see fig. 5), and proceeds to the processing in step S36 to end the "weight information processing". In order to add or update information to or from the in-store food list 131, data of the stored food information 132 is temporarily stored in the storage unit 130.

In step S41, the food information acquiring unit 114 records the weight information of the shipped food in the shipped food information 133 (see fig. 5), and proceeds to the processing in step S36 to end the "weight information processing". In order to add or update food information to or from the in-store food list 131, data of the out-of-store food information 133 is temporarily stored in the storage unit 130.

[3-3. image analysis processing ]

Next, the execution procedure of the "image analysis processing" will be described based on the flowcharts shown in fig. 9 and 10. The "image analysis processing" is executed by the food information acquisition unit 114. Fig. 9 is a process of acquiring information from the "image at the time of warehousing" for the food warehoused in the refrigerating compartment 10. Fig. 10 shows a process of acquiring information from the "images at the time of shipment" for food items shipped from the refrigerating room 10.

First, with reference to fig. 9, "image analysis processing" for "image at the time of warehousing" will be described. In step S50 of fig. 9, the food information acquisition section 114 executes character extraction processing by the image processing section 120 for the "warehousing-time image" of the warehoused food. Here, the "put-in food" refers to the food judged to have been put in the refrigerating compartment 10 by the in-out judgment section 113 in step S33 of fig. 8. The "warehousing-time image" of the warehoused food is the "warehousing-time image" which is captured and stored by the food information acquisition unit 114 in steps S20 and S21 in fig. 7, with respect to the food determined to be warehoused in step S33 in fig. 8.

In the next step S51, the food information acquisition unit 114 determines whether or not a character is extracted from the "put-in-store image" by the image processing unit 120. Then, when the character is extracted, the food information acquisition unit 114 proceeds to the process of step S52, and when the character is not extracted, proceeds to the process of step S60.

In step S52, the food information acquisition unit 114 analyzes the character information extracted by the image processing unit 120. Then, in the next step S53, the food information acquisition unit 114 acquires information on the warehoused food, such as the food name, the expiration date, the production place, and the net content, from the character information. In the next step S54, the food information acquisition unit 114 records the acquired information of the warehoused food in the warehoused food information 132, and ends the "image analysis processing".

In step S60, the food information acquisition unit 114 extracts the image portion of the warehoused food from the "image at the time of warehousing" by the image processing unit 120. In the next step S61, the food information acquisition unit 114 calculates the matching rate between the extracted image portion of the warehoused food and the various food sample images 134 (see fig. 5) stored in the storage unit 130 in advance.

In the next step S62, the food information acquiring unit 114 estimates that the food name of the sample image 134 having the highest matching rate is the food name of the food put in storage, and proceeds to the processing in step S54. According to the processing of steps S60 to S62, even in the case where the character cannot be extracted from the "warehousing-time image" of the warehoused food, the food name of the warehoused food can be estimated and recorded in the warehoused food information 132.

Next, with reference to fig. 10, "image analysis processing" for the "images at the time of shipment" will be described. In step S70 of fig. 10, the food information acquisition unit 114 executes the character extraction process by the image processing unit 120 for the "shipment-time image" of the shipment food. Here, the food delivered from the refrigerator compartment 10 is determined to have been delivered from the refrigerator compartment 10 by the delivery/retrieval determination section 113 in step S40 in fig. 8. The "outgoing image" of the outgoing food is the "outgoing image" captured and stored by the food information acquisition unit 114 in steps S22 and S23 in fig. 7, with respect to the food determined to be outgoing in step S40 in fig. 8.

In the next step S71, the food information acquisition unit 114 determines whether or not the image processing unit 120 has extracted a character from the "images at the time of shipment". Then, when the character is extracted, the food information acquisition unit 114 proceeds to the process of step S72, and when the character is not extracted, proceeds to the process of step S80.

In step S72, the food information acquisition unit 114 analyzes the information of the character extracted by the image processing unit 120. In the next step S73, the food information acquiring unit 114 acquires information on the shipment of the food, such as the name of the food, the expiration date, the place of manufacture, and the net content, from the character information. In the next step S74, the food information acquiring unit 114 records the acquired information of the shipped food in the shipped food information 133, and ends the "image analysis processing".

In step S80, the food information acquiring unit 114 extracts the image portion of the picked-up food from the "picked-up image" by the image processing unit 120. In the next step S81, the food information acquiring unit 114 calculates the matching rate between the extracted image portion of the shipped food and each of the food sample images 134 stored in the storage unit 130 in advance.

In the next step S82, the food information acquiring unit 114 estimates that the food name of the sample image 134 having the highest matching rate is the food name of the delivered food, and proceeds to the processing in step S74. According to the processing of steps S80 to S82, even when the character cannot be extracted from the "image at the time of shipment" of the shipment food, the food name of the shipment food can be estimated and recorded in the shipment food information 133.

The image analysis processing of fig. 9 is performed for determining an image at the time of entering into the refrigerating room 10, and the image analysis processing of fig. 10 is performed for determining an image at the time of leaving from the refrigerating room 10. Therefore, when the right door 11 or the left door 12 is detected by the object detection unit 111 instead of the food, it cannot be determined as warehousing or ex-warehousing, and therefore the image analysis processing is not performed (the captured image is invalid). Thereby, it is possible to prevent erroneous food information from being acquired from the image portion of the right door 11 or the left door 12.

[3-4. weight sensor correction processing ]

The procedure of executing the "weight sensor correction process" for correcting the detected weights of the

weight sensors

13a, 13b, 14a, 14b, 15a, and 15b will be described with reference to the flowchart shown in fig. 11. The "weight sensor correction processing" is executed by the storage amount recognition unit 112. The "weight sensor correction processing" is processing for correcting (correcting) a detection error portion of the

weight sensors

13a, 13b, 14a, 14b, 15a, 15b that may occur due to a temporal change or the like.

In step S90 of fig. 11, when the storage determination unit 113 determines that there is food entering the refrigerating compartment (YES in step S33 of fig. 8), the storage amount recognition unit 112 proceeds to the process of step S91. In step S91, the storage amount recognition unit 112 determines whether or not the information on the net content of the warehoused food is recorded in the warehouse food information 132 (the processing of step S54 in fig. 9). When the information of the net content of the warehoused food (the net content of the food acquired in step S53 of fig. 9) is recorded in the warehoused food information 132, the storage amount identifying unit 112 proceeds to the process of step S92.

In step S92, the storage amount recognition unit 112 calculates the ratio of the weight to the net content of the warehoused food recorded in the warehoused food information 132. Then, the storage amount recognition unit 112 sets the calculated ratio as a correction coefficient of a weight sensor for detecting the weight of the warehoused food. For example, when the warehoused food is warehoused on the upper shelf 13 and the

weight sensors

13a and 13b detect the weight, the storage amount identifying unit 112 sets the calculated ratio as a correction coefficient of the detected weight of the

weight sensors

13a and 13 b.

In the next step S93, the storage amount recognition unit 112 stores the data of the correction coefficient in the storage unit 130, and proceeds to the process in step S94 to end the "weight sensor correction process". Then, the storage amount recognition unit 112 multiplies the detected weight of the

weight sensors

13a and 13b by a correction coefficient to obtain a detected weight. Similarly, the storage amount recognition unit 112 sets the correction coefficients of the

weight sensors

14a and 14b when the warehoused food is stored on the middle shelf 14, and sets the correction coefficients of the

weight sensors

15a and 15b when the warehoused food is stored on the lower shelf 15.

On the other hand, in step S91, if the net content information is not recorded in the warehoused food information 132, the storage amount identifying part 112 proceeds to the process in step S94 and ends the "weight sensor correction process". In this case, the calculation and setting of the correction coefficient are not performed.

[3-5. management treatment of foods in warehouse ]

The execution procedure of the "in-house food management process" for managing the food information stored in the refrigerating compartment 10 will be described with reference to the flowcharts shown in fig. 12 and 13. The "in-library food management processing" is executed by the food information management unit 115.

The food information management unit 115 manages food information stored in the refrigerating compartment 10 based on an in-compartment food list 131 (see fig. 5). As shown in fig. 14, the in-house food list 131 records: the food numbers (1, 2, 3, …) put in the cold storage chamber 10 and the information of each item of food put in storage with numbers include the name, weight (first time, current time, amount of eating), date and time of putting in storage and out of storage (first time putting in storage, latest time putting out of storage, latest time putting in storage), stock (1: present, 0: absent), shelf life and production place. The food information management unit 115 generates and updates the in-warehouse food list 131 based on the in-warehouse food information 132 and the out-warehouse food information 133 recorded by the food information acquisition unit 114.

In step S100 of fig. 12, the food information management unit 115 determines whether or not it is determined that food is put into the refrigerator compartment 10 or food is taken out from the refrigerator compartment 10 by the put-in/out determination unit 113, and proceeds to the processing of step S101 when it is determined that food is put into the refrigerator compartment 10 or food is taken out. In step S101, the food information management unit 115 determines whether or not the warehousing determination is made, and when the warehousing determination is made, the process proceeds to step S102. On the other hand, when the warehousing determination is not made (when the ex-warehouse determination is made), the food information management unit 115 proceeds to the processing in step S110.

In step S102, food information management unit 115 determines whether or not the food is taken out of refrigerating room 10 within a predetermined time. If the refrigerator compartment 10 is taken out of the refrigerator compartment within the predetermined time, the process proceeds to step S103, and if the refrigerator compartment 10 is not taken out of the refrigerator compartment within the predetermined time, the process proceeds to step S111. Further, it can be recognized from the in-house food list 131 that the food is taken out of the refrigerating compartment 10 within a predetermined time. In addition, the predetermined time is assumed to be a time when the user returns (restocks) a part of the food taken out of the refrigerating room 10 to the refrigerating room 10 after using the part, and is set to, for example, about 10 minutes.

In step S103, the food information management unit 115 determines whether or not the matching rate between the food that was delivered last time and the food that was put in the warehouse this time is equal to or higher than a predetermined value. Here, the food information management unit 115 compares the product information of the last shipment recorded in the in-stock food list 131 with the product information of the present-time stock recorded in the stock food information 132, and calculates the matching rate between the food of the last shipment and the food of the present-time stock based on the matching state of the food name, the expiration date, the place of production, and the like.

Then, when the matching rate between the food that was delivered last time and the food that was put in the warehouse this time is equal to or greater than the predetermined value, the food information management unit 115 proceeds to the processing in step S104, and when the matching rate between the food that was delivered last time and the food that was put in the warehouse this time is smaller than the predetermined value, the processing proceeds to step S112.

Step S104 is a process performed when it can be determined that the matching rate between the food delivered last time and the food put in the warehouse this time is equal to or greater than a predetermined value and the food delivered last time and the food put in the warehouse this time are the same item. In step S104, the food information management unit 115 updates the current weight, the latest warehousing date and time, and the amount of eating (the difference between the initial warehousing time and the current weight) of the food delivered before the predetermined time in the in-warehouse food list 131 based on the information recorded in the warehoused food information 132, and proceeds to the processing in step S105 of fig. 13.

Step S112 is a process performed when it can be determined that the matching rate between the food delivered last time and the food put in the warehouse this time is smaller than a predetermined value and the food delivered last time and the food put in the warehouse this time are different items. In step S112, in the in-store food list 131, the food information management unit 115 newly registers the stored food based on the information recorded in the stored food information 132, and the process proceeds to step S105 in fig. 13.

Step S111 is a process when the shipment is not made within a predetermined time. In step S111, the food information management unit 115 newly registers the warehoused food in the in-warehouse food list 131 based on the information recorded in the warehouse food information 132, and proceeds to the processing in step S105 of fig. 13.

Step S110 is a process when it is determined that the food has been taken out of the refrigerating compartment 10. In step S110, the food information management unit 115 updates the current weight of the corresponding food (ex-warehouse food) to 0 in the in-warehouse food list 131. In addition, the food information management unit 115 is configured to store a list of food items in the library

131, the eating amount of the corresponding food is updated based on the information recorded in the ex-warehouse food information 133

The latest delivery time is updated, and the process proceeds to step S105 in fig. 13.

In step S105 of fig. 13, the food information management unit 115 determines whether or not there is a food whose current weight is 0 g, based on the processing in step S110. The food information management unit 115 proceeds to the processing in step S106 when there is a food whose current weight becomes 0 gram, and proceeds to the processing in step S120 when there is no food whose current weight becomes 0 gram.

In step S106, the food information management unit 115 updates the information on the corresponding food items in the in-store food item list 131 (food items determined to be currently delivered) to "out-of-stock" (stock is 0), and proceeds to the processing in step S107. In step S120, the food information management unit 115 updates the information on the corresponding food (food determined to be put in the warehouse this time) in the in-warehouse food list 131 to "stored" (stored as 1), and proceeds to the processing in step S107.

In step S107, the food information management unit 115 determines whether or not the information on the expiration date recognized by the character is recorded in the put-in food information 132. When the shelf life information is recorded in the stored food information 132, the food information management unit 115 proceeds to the processing in step S108, and when the shelf life information is not recorded, the processing proceeds to step S121.

In step S108, the food information management unit 115 registers the quality guarantee period of the currently warehoused food in the in-warehouse food list 131 based on the quality guarantee period information recorded in the warehoused food information 132, and proceeds to the process in step S109. In addition, in step S121, the food information management part 115 refers to the quality guarantee period reference list 135 stored in the storage part 130, and searches for a general quality guarantee period of the corresponding food.

As shown in fig. 14, the quality guarantee period reference list 135 is a list in which the food names and the quality guarantee periods of the foods are associated with each other, and the food information management unit 115 searches for the quality guarantee period associated with the food name of the current warehousing food. In the next step S122, the food information management unit 115 registers the expiration date in the corresponding food data in the in-store food list 131, and proceeds to the process in step S109.

In step S109, the food information management unit 115 transmits the updated data of the in-house food list 131 to the management server 520 via the communication network 510 by using the wireless communication unit 140, and terminates the "in-house food management processing".

The management server 520 stores the data of the in-house food list 131 received from the refrigerator 1 in a storage device (not shown), and transmits the eating status, the number of days of expiration date remaining, and the like of each product identified from the in-house food list 131 to the terminal device 400 of the user. Thus, even when the user is out, the terminal device 400 can check the status of the product stored in the refrigerator 1.

[4 ] Another embodiment of the image pickup processing ]

Next, another embodiment of the "image pickup processing" will be described based on the flowchart shown in fig. 15. The flowchart shown in fig. 15 is implemented by the food information acquiring unit 114 and the imaging time setting unit 116. The imaging time setting unit 116 sets the time when the 1 st predetermined time has elapsed since the right door 11 or the left door 12 of the refrigerator compartment was opened as the 1 st imaging time, and sets the time 2 nd predetermined time earlier than the time when the right door 11 or the left door 12 of the refrigerator compartment 10 was closed as the 2 nd imaging time.

When it is recognized from the detection signals of opening/

closing sensors

18a and 18b that one or both of right door 11 and left door 12 of refrigerating room 10 has been switched from the closed state to the open state in step S130, food information acquisition unit 114 performs imaging by camera 60 at the 1 st imaging time and the 2 nd imaging time until it is recognized in step S137 that both of right door 11 and left door 12 have been switched from the open state to the closed state.

When recognizing that one or both of right door 11 and left door 12 of refrigerating room 10 have been switched from the closed state to the open state in step S130, food information acquisition unit 114 proceeds to the process in step S131. Steps S131 to S132 are processes performed by the imaging time setting unit 116.

In step S131, the imaging time setting unit 116 starts a timer having the 1 st predetermined time as the counted time from the last time (time up). In the next step S132, when the timer time is up (becomes the 1 st shooting time), the process proceeds to step S133.

Steps S133 to S138 are processes performed by the food information acquiring unit 114. The food information acquisition unit 114 performs shooting by the camera 60 a plurality of times in step S133, and stores the data of the shot image in the storage unit 130 as data of the "image at the time of warehousing" in the next step S134.

In the following loop of steps S135 to S137, the food information acquiring unit 114 repeatedly executes the processing of steps S135 and S136 until the opening/

closing sensors

18a and 18b detect that the right door 11 and the left door 12 are closed in step S137. The food information acquisition unit 114 performs shooting by the camera 60 and records the shooting time in the storage unit 130 in step S135, and stores the data of the shot image in the storage unit 130 as the outgoing image candidate data in step S136. This sequentially stores the data of the time-series captured images in the storage unit 130.

When the opening/

closing sensors

18a and 18b detect that the right door 11 and the left door 12 are closed in step S137, the food information acquiring unit 114 proceeds to the process of step S138. In step S138, the food information acquisition unit 114 stores, in the storage unit 130, image data captured at a time 2 nd predetermined time earlier than the time at which both the right door 11 and the left door 12 are detected to be in the closed state in step S137, out of the shipment-time image candidate data stored in the storage unit 130, as a shipment-time image.

As another configuration in which the imaging time setting unit 116 sets the 1 st imaging time and the 2 nd imaging time, the 1 st imaging time and the 2 nd imaging time may be set based on the object detection status of the object sensor 80 (see fig. 3 to 5) that detects an object passing through the opening 10 a. In this case, the imaging time setting unit 116 sets, for example, the time when a predetermined time has elapsed from the time when the object sensor 80 is switched from the state in which no object is detected to the state in which an object is detected, to the 1 st imaging time. The imaging time setting unit 116 then sets the time, which is earlier than the time by a predetermined time, as the 2 nd imaging time, for example, based on the time when the state in which the object sensor 80 detects the object is switched to the state in which the object is not detected.

[5 ] other embodiments ]

In the above embodiment, the object detection unit 111 detects an object passing through the opening 10a of the refrigerating compartment 10 based on the change in illuminance in the refrigerating compartment 10 detected by the

illuminance sensors

17a, 17b, and 17 c. As another configuration of the passage object detection unit 111, as shown in fig. 3 to 5, an object sensor 80 (a reflection-type or transmission-type optical sensor, an ultrasonic sensor, a capacitance sensor, or the like) that includes the opening 10a in a detection range and directly detects an object passing through the opening 10a may be used. In this case, the detection range of object sensor 80 may be set to be on the front side of the imaging range of camera 60 in the direction from opening 10a into refrigerating room 10. This prevents failure in the imaging of the food put in storage due to a delay time from the time when the object passing through the opening 10a is detected to the time when the camera 60 performs imaging.

Further, when the open/close sensor 18a detects that the left door 12 is opened and when the open/close sensor 18b detects that the right door 11 is opened, the camera 60 may start to be energized to enable the camera 60 to take an image. With this configuration, when the left door 12 and the right door 11 are closed, the power supply to the camera 60 is cut off to save power. Further, by starting energization of the camera 60 at the timing when the right door 11 or the left door 12 is opened, it is possible to prevent failure of shooting of the warehoused food due to delay in activation of the camera 60.

The food information acquiring unit 114 may invalidate the image data captured by the camera 60 when the open-close sensor 18a detects that the left door 12 is switched from the open state to the closed state, or when the open-close sensor 18b detects that the right door 11 is switched from the open state to the closed state, within a predetermined time period after the object passing through the opening 10a is detected by the object detecting unit 111. According to this configuration, when the left door 12 or the right door 11 is closed, the object detection unit 111 detects the left door 12 or the right door 11 and the camera 60 captures the detected object, thereby preventing erroneous food information from being extracted from the image of the left door 12 or the right door 11.

In the above embodiment, the storage amount recognition portion 112 recognizes the amount of food items stored on each of the

shelves

13, 14, and 15 of the refrigerating compartment 10 based on the detected weight of the

weight sensors

13a, 13b, 14a, 14b, 15a, and 15 b. As another configuration of the storage amount recognition unit 112, the amount of food stored in each

shelf

13, 14, 15 of the refrigerating compartment 10 is recognized based on the illuminance detected by the

illuminance sensor

17a, 17b, 17c in a state where the

illumination units

16a, 16b, 16c, 16d, 16e, 16f are turned on. Alternatively, the amount of food items stored on the

shelves

13, 14, and 15 of the refrigerating compartment 10 may be identified by extracting image portions of the food items from the images of the inside of the refrigerating compartment 10 captured by the camera 60 or a camera separately provided.

Although the image processing unit 120 performs the process of the captured image by the camera 60 in the above embodiment, the CPU110 may have the function of the image processing unit 120 and the CPU110 may perform the process of the captured image.

In the above embodiment, the data of the sample image 134 and the expiration date reference list 135 are stored in the storage unit 130, but may be stored in the management server 520 and downloaded from the management server 520 to the control unit 100 of the refrigerator 1 during use.

Although the sample image 134 is stored in the storage unit 130 or the management server 520, an image recognition program in which the characteristics of various foods and stored materials such as color, shape, size, and the like are learned in advance may be installed in the CPU110, and the image portion of the warehoused food may be processed by the image recognition program to calculate the matching rate and estimate the matching rate as the food name of the warehoused food. This eliminates the need for the sample image 134, and can reduce the storage capacity stored in the storage unit 130 or the management server 520, thereby making it possible to increase the processing time for identifying the food name for which the warehousing food is estimated. Estimation of the food name of the delivered food can be handled in the same manner as described above.

In the above embodiment, the entering and exiting of food items in the refrigerating compartment 10 of the refrigerator 1 are determined by the entering/exiting determination section 113, information on the entered food items and the exiting food items is acquired by the food information acquisition section 114, and information on the food items stored in the refrigerating compartment 10 is managed by the food information management section 115. The switching chamber 20, the freezing chamber 40, and the vegetable chamber 50 may be provided with a passage object detection unit that detects objects passing through the

openings

20a, 40a, and 50a, and a storage amount recognition unit that recognizes the amount of food stored, thereby determining whether food is stored or removed. Further, it is also possible to manage the food by providing a camera that photographs a range including the

openings

20a, 40a, and 50a, acquiring information of the warehoused food and the shipped food from the photographed images (the warehousing image and the shipped image), and acquiring information of the food stored in the switching chamber 20, the freezing chamber 40, and the vegetable chamber 50.

As described above, the refrigerator according to the present invention can more reliably acquire information on foods put in the storage chamber and foods taken out of the storage chamber by imaging the storage chamber at the time when the foods put in the storage chamber and the foods taken out of the storage chamber pass through the storage chamber, and thus can be applied to the use of managing the foods stored in the refrigerator.

Claims

1. A refrigerator, comprising:

a storage chamber having an opening;

a door provided in the storage chamber;

a camera capable of capturing an image of an imaging range including the opening;

an opening/closing sensor for detecting an opening/closing state of the door;

a storage unit; and

and a food information acquisition unit that performs imaging by the camera a plurality of times during a period from when the open-close sensor detects that the door is switched from the closed state to the open state to when the open-close sensor detects that the door is switched from the open state to the closed state, stores an image captured at a 1 st imaging time at which a 1 st predetermined time has elapsed since the open-close sensor detected that the door is switched from the closed state to the open state in the storage unit as an image at the time of warehousing, and stores an image captured at a 2 nd imaging time later than the 1 st imaging time and before the time when the open-close sensor detects that the door is switched from the open state to the closed state in the storage unit as an image at the time of warehousing.

2. A refrigerator as claimed in claim 1, wherein:

the food information acquiring unit repeats shooting by the camera until the open/close sensor detects that the door is switched from the open state to the closed state after the 1 st shooting time has elapsed, stores a plurality of shot images in the storage unit as the delivery-time image candidates together with the shooting time, and stores the delivery-time image candidate, which is shot at the 2 nd shooting time, among the plurality of delivery-time image candidates in the storage unit as the delivery-time image when the open/close sensor detects that the door is switched from the open state to the closed state.

3. A refrigerator as claimed in claim 1 or 2, characterized in that:

the food information acquisition unit acquires information on the food put in the storage chamber and the food taken out of the storage chamber based on the image at the time of storage and the image at the time of delivery.

4. A refrigerator as claimed in claim 1, wherein:

the storage chamber is provided at the uppermost layer of the refrigerator,

the camera is disposed at an upper portion of the receiving chamber.

5. A refrigerator as claimed in claim 1, wherein:

the receiving chamber is a refrigerating chamber.