JP7012263B2 - refrigerator - Google Patents

Description

The present invention relates to a refrigerator.

Conventionally, an internal camera was installed in the center of the food storage room, and when the door of the storage room was opened and closed, the storage room was imaged by the internal camera, and when the food was stored from the time-series captured images. A refrigerator has been proposed in which the image of the above and the current image are output side by side (see, for example, Patent Document 1). In the refrigerator described in Patent Document 1, depending on the arrangement of foods in the containment chamber, the foods may overlap in the image pickup direction of the camera in the refrigerator, and some foods may not be recognized from the captured image.
Therefore, we provided a switch to detect the opening and closing of the door of the containment chamber and a camera whose imaging range is the opening of the containment chamber. Alternatively, a refrigerator has been proposed in which the food to be delivered is imaged (see, for example, Patent Document 2).

Japanese Unexamined Patent Publication No. 2016-57022 Japanese Unexamined Patent Publication No. 2015-81762

An object of the present invention is to have a camera take an image based on the detection result of the illuminance sensor in a refrigerator provided with an illuminance sensor in the accommodation room .

The refrigerator of the present invention has a storage room having an opening, a door provided in the storage room, a camera that captures an imaging range including the opening, an illuminance sensor provided in the storage room, and a door closed. The illuminance detected by the illuminance sensor between the time when the door is switched to the open state and the time when the door is closed is equal to or more than the first predetermined value from the illuminance detected when the door is switched from the closed state to the open state. It is characterized by being provided with a first control means for causing a camera to take an image when the image is lowered .

According to the present invention, in a refrigerator provided with an illuminance sensor in a storage room, a camera can image an image based on the detection result of the illuminance sensor .

Explanatory diagram of the configuration to check the status of food stored in the refrigerator installed in the house with a terminal device Explanatory drawing of the appearance of the refrigerator Explanatory drawing showing the structure of the inside of the refrigerator by the cross-sectional view seen from the right side. Explanatory view showing the internal structure of the refrigerator compartment by a cross-sectional view seen from the front. Configuration diagram of the control unit Flowchart showing the overall processing by the control unit Flow chart of imaging process Flow chart of weight information processing Flowchart of image analysis process at the time of warehousing Flowchart of image analysis process at the time of delivery Flow chart of weight sensor correction process First flowchart of food management process in the warehouse Second flowchart of food management processing in the warehouse Explanatory drawing of the food list in the warehouse and the best-by date standard list Flowchart of another embodiment of the imaging process

The first invention comprises a storage chamber having an opening, an opening / closing portion for opening / closing the opening, an opening / closing sensor for detecting the open / closed state of the opening / closing portion, a camera for capturing an image pickup range including the opening, and storage. From the time when the open / close sensor detects that the open / close portion has switched from the closed state to the open state, until the open / close sensor detects that the open / close portion has switched from the open state to the closed state. In the meantime, the camera takes an image at the first imaging time set assuming the time when the food to be stored in the storage chamber passes through the opening, and the captured image at the first imaging time is stored. An image is stored in the storage unit, and an image is taken by the camera at a second imaging time set assuming a time point when the food delivered from the storage chamber passes through the opening, and the second imaging time point is performed. The captured image in the above is stored in the storage unit as a delivery image, and information on the food stored in the storage chamber and the food delivered from the storage chamber is acquired based on the storage image and the delivery image. It has a food information acquisition department.
According to the first invention, the food information acquisition unit has an image at the time of warehousing taken at the time of the first imaging assuming a time point when the food to be stored in the storage unit passes through the opening of the storage unit, and a warehousing from the storage room. The image at the time of delivery taken at the time of the second imaging assuming the time when the food to be processed passes through the opening is stored in the storage unit. Then, the food information acquisition unit obtains information on the food to be stored in the storage room and the food to be delivered from the storage room based on the image at the time of warehousing and the image at the time of warehousing, which are likely to include the image portion of the food. You can definitely get it.

The second invention is based on an illuminance sensor that detects the illuminance in the accommodation room and the detected illuminance of the illuminance sensor at the time when the opening / closing sensor detects that the opening / closing portion has been switched from the closed state to the open state. , The first reference illuminance and the second reference illuminance are set, the first imaging time point is set based on the time point when the detection illuminance of the illuminance sensor becomes equal to or less than the first reference illuminance, and the detection illuminance of the illuminance sensor is set. It is provided with an imaging time point setting unit that sets the second imaging time point based on the time point when the illuminance becomes equal to or higher than the second reference illuminance after becoming equal to or lower than the first reference illuminance.
According to the second invention, the illuminance in the containment chamber changes according to the existence state of an object (food, a user's hand, etc.) in the containment chamber. Therefore, based on the change in the illuminance in the containment chamber detected by the illuminance sensor, the food to be stored in the containment chamber is opened at the first imaging time assuming the time when the food is to pass through the opening, and the food to be delivered from the containment chamber is opened. It is possible to appropriately set the second imaging time point assuming the time point when passing through the unit.

In the third aspect of the present invention, the time point at which the first predetermined time has elapsed from the time when the opening / closing sensor detects that the opening / closing portion has switched from the closed state to the open state is set as the first imaging time, and the opening / closing is performed. It is provided with an imaging time point setting unit that sets a time point before a second predetermined time as a time point before the time when it is detected by the sensor that the opening / closing part is switched from the open state to the closed state as the second imaging time point.
According to the third invention, the time from the user opening the opening / closing part of the storage part to the storage of food and the time from the user leaving the food from the storage room to closing the opening / closing part are the times in the storage room. It can be expected that it will be a relatively short period of time in order to maintain the cold storage state. Therefore, the first imaging time point and the second imaging time point can be set by a simple configuration of setting the first predetermined time and the second predetermined time.

In the fourth aspect of the invention, the food information acquisition unit is described from the time when the first imaging time has elapsed until the opening / closing sensor detects that the opening / closing unit has switched from the open state to the closed state. The image taken by the camera is repeatedly executed to store the captured image in the storage unit, and when the opening / closing sensor detects that the opening / closing unit has switched from the open state to the closed state, the opening / closing sensor causes the opening / closing unit to move. The captured image captured and stored in the storage unit at a time before the second predetermined time from the time when it is detected that the open state has been switched to the closed state is stored as the image at the time of delivery.
According to the fourth invention, the food information acquisition unit captures images with a camera from the time when the first imaging time elapses until the opening / closing sensor detects that the opening / closing unit has switched from the open state to the closed state. Is repeatedly executed to save the captured image in the storage unit. As a result, the captured image at the time before the second predetermined time before the time when the opening / closing sensor detects that the opening / closing portion has been switched from the open state to the closed state can be saved as the image at the time of delivery.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to the present embodiment.

[1. How to use the refrigerator]
FIG. 1 shows a usage mode of the refrigerator of the present embodiment. The refrigerator 1 of the present embodiment is installed in the house H and has a function of communicating with the management server 520 via the gateway 500 and the communication network 510. Further, the refrigerator 1 has a function of communicating with a terminal device (smartphone, tablet terminal, etc.) 400. The terminal device 400 has a function of communicating with the management server 520 via the gateway 500 and the communication network 510.

The control unit 100 provided in the refrigerator 1 controls the overall operation of the refrigerator 1 and generates an in-fridge food list 131 showing information on foods stored in the storage room of the refrigerator 1 to generate a management server 520. Send to. The refrigerator management application executed in the terminal device 400 is obtained by acquiring the data of the food list 131 in the refrigerator from the management server 520, or by acquiring the data of the food list 131 in the refrigerator from the refrigerator 1. Based on the food list 131, information on the food stored in the refrigerator 1 is displayed on the display unit. The user P of the refrigerator 1 can confirm the status (food name, expiration date, remaining amount, etc.) of the food contained in the refrigerator 1 by visually recognizing the display of the terminal device 400.

[2. Refrigerator configuration]
The configuration of the refrigerator 1 will be described with reference to FIGS. 2 to 5. FIG. 2 is an explanatory diagram showing the appearance of the refrigerator 1. As shown in FIG. 2, the refrigerator 1 includes a main box body 2 opened in the front surface, and inside the main box body 2, a refrigerating chamber 10 (corresponding to a storage chamber of the present invention), an ice making chamber 30, and an ice making chamber are provided. A switching chamber 20, a freezer compartment 40, and a vegetable compartment 50, which are attached to the 30 and can change the temperature inside the refrigerator, are formed.

A rotary right door 11 and a left door 12 (corresponding to the opening / closing portion of the present invention) are provided in the opening on the front surface of the refrigerating chamber 10. Further, the switching chamber 20, the ice making chamber 30, the freezing chamber 40, and the vegetable compartment 50 are provided with drawers 21, 31, 41, and 51 for accommodating food, respectively.

Next, FIG. 3 is an explanatory view showing the configuration of the inside of the refrigerator 1 by a cross-sectional view seen from the right side, and FIG. 4 is an explanatory view showing the internal configuration of the refrigerator compartment 10 by a cross-sectional view seen from the front. be. With reference to FIGS. 3 and 4, the refrigerating chamber 10 is provided with an upper shelf 13, a middle shelf 14, and a lower shelf 15 that divide the refrigerating chamber 10 into sub-areas. Further, in the refrigerating chamber 10, weight sensors 13a and 13b for detecting the weight of food placed on the upper shelf 13, weight sensors 14a and 14b for detecting the weight of food placed on the middle shelf 14, and a lower shelf 15 are used. Weight sensors 15a and 15b for detecting the weight of the placed food are provided. Further, the refrigerating chamber 10 is provided with lighting units 16a to 16f for illuminating the inside of the refrigerating chamber 10, illuminance sensors 17a to 17c for detecting the illuminance in the refrigerating chamber 10, and a camera 60.

The camera 60 is arranged near the opening 10a at the upper part of the refrigerating chamber 10 so that the opening 10a is included in the imaging range, and images the range from the opening 10a to the vicinity of the front ends of the shelves 13, 14, 15. .. Further, a compressor 61, a cooling fan 62, a cooler 63, and a condenser 64, which are auxiliary machines constituting the refrigeration cycle, are provided. Further, on the back surface of the refrigerating chamber 10, a refrigerating chamber duct 70 for circulating cold air and cold air discharge ports 71 to 73 are arranged.

The refrigerating chamber 10 is provided with an opening / closing sensor 18b for detecting the opening / closing of the right door 11 and an opening / closing sensor 18a for detecting the opening / closing of the left door 12. The open / close sensor 18b outputs a close detection signal when the right door 11 is closed, and outputs an open detection signal when the right door 11 is open. Similarly, the open / close sensor 18a outputs a close detection signal when the left door 12 is closed, and outputs an open detection signal when the left door 12 is open.

The switching chamber 20 is provided with an opening / closing sensor 22 for detecting the opening / closing of the opening 20a. The open / close sensor 22 outputs a close detection signal when the drawer 21 is housed in the switching chamber 20, and outputs an open detection signal when the drawer 21 is pulled out from the switching chamber 20.

Similarly, the ice making chamber 30 is provided with an opening / closing sensor 32 for detecting the opening / closing of the opening 30a by the drawer 31. Further, the freezing chamber 40 is provided with an opening / closing sensor 42 for detecting the opening / closing of the opening 40a by the drawer 41, and the vegetable compartment 50 is provided with an opening / closing sensor 52 for detecting the opening / closing of the opening 50a by the drawer 51. The open / close sensors 32, 42, and 52 output the close detection signal and the open detection signal, similarly to the open / close sensor 22.

Next, FIG. 5 is a configuration diagram of the control unit 100. With reference to FIG. 5, the control unit 100 includes an electron configured by a CPU (Central Processing Unit) 110, an image processing unit 120, a storage unit 130, a wireless communication unit 140, a date / time recognition unit 141, an interface circuit (not shown), and the like. It is a circuit unit.

By executing the control program of the refrigerator 1 stored in the storage unit 130, the CPU 110 executes a passing object detection unit 111, a storage capacity recognition unit 112, a warehousing / delivery determination unit 113, a food information acquisition unit 114, and a food information management unit. It functions as the 115 and the imaging time point setting unit 116.

The control unit 100 includes open / close sensors 18a, 18b, 22, 32, 42, 52, illuminance sensors 17a, 17b, 17c, weight sensors 13a, 13b, 14a, 14b, 15a, 15b, a camera 60, lighting units 16a, 16b, It is connected to 16c, 16d, 16e, 16f, and refrigeration cycle accessories 61, 62, 63, 64.

Detection signals of open / close sensors 18a, 18b, 22, 32, 42, 52, illuminance sensors 17a, 17b, 17c, weight sensors 13a, 13b, 14a, 14b, 15a, 15b and the like are input to the control unit 100. .. Further, the operation of the lighting units 16a, 16b, 16c, 16d, 16e, 16f, the refrigerating cycle auxiliary machines 61, 62, 63, 64 and the like is controlled by the control signal output from the control unit 100.

Further, the control signal output from the control unit 100 controls the image pickup by the camera 60 via the image processing unit 120. The image processing unit 120 converts the video signal output from the camera 60 into a digital tone signal to generate a captured image, and stores the captured image data in the image memory 121. The image processing unit 120 processes the captured image stored in the image memory 121 according to the control signal output from the CPU 110.

The date / time recognition unit 141 recognizes the current date / time (date and time) by clocking a clock signal output from a clock circuit (not shown) provided in the control unit 100. In addition, the information of the current date and time may be acquired from the management server 520 by communication with the management server 520, and the current date and time may be recognized.

In the storage unit 130, in addition to the data of the food list 131 in the refrigerator, the stored food information 132 in which the information of the food stored in the refrigerating chamber 10 is recorded and the information of the food discharged from the refrigerating chamber 10 are recorded. The data of the delivered food information 133 is stored. Further, the storage unit 130 stores data of sample images 134 of various foods and a best-by-date list 135 showing general best-by-dates of various foods.

The passing object detection unit 111 detects an object passing through the opening 10a of the refrigerating chamber 10 based on the change in the illuminance in the refrigerating chamber 10 detected by the illuminance sensors 17a, 17b, 17c. The storage amount recognition unit 112 determines the amount of food stored in the upper shelf 13, the middle shelf 14, and the lower shelf 15 of the refrigerating chamber 10 based on the detection signals of the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b. And recognize by weight.

The storage amount recognition unit 112 recognizes the weight of the food stored in the upper shelf 13 based on the detection signals of the weight sensors 13a and 13b, and determines the weight of the food stored in the middle shelf 14 based on the detection signals of the weight sensors 14a and 13b. Recognize based on the detection signal of 14b. Further, the storage capacity recognition unit 112 detects the weight of the food stored in the lower shelf 15 based on the detection signals of the weight sensors 15a and 15b.

The warehousing / delivery determination unit 113 is placed on the shelves 13, 14, 15 recognized by the capacity recognition unit 112 when an object passing through the opening 10a of the refrigerating chamber 10 is detected by the passing object detection unit 111. Based on the change in the amount (weight) of the food, it is determined whether the food is stored in the refrigerating chamber 10 and the food is discharged from the refrigerating chamber 10.

The food information acquisition unit 114 takes an image of an object that has passed through the opening 10a of the refrigerating chamber 10 by the camera 60. Then, when the warehousing / delivery determination unit 113 determines that the food is warehousing in the refrigerating chamber 10 or the food is warehousing from the refrigerating chamber 10, the image portion of the food is extracted from the captured image.

The food information acquisition unit 114 recognizes the characters contained in the image portion of the food, or extracts the characteristics of the image portion of the food, so that the food name, content, place of origin, and taste of the food received or delivered can be obtained. Get information such as deadlines. Then, the food information acquisition unit 114 records the acquired information in the warehousing food information 132 for the food determined to be warehousing (stocked food), and the warehousing food information 133 for the food determined to be warehousing (delivered food). Record in.

The food information management unit 115 is based on the weight of the food in the refrigerating room 10 recognized by the storage capacity recognition unit 112, the in-stock food information 132 and the out-of-delivery food information 133 recorded by the food information acquisition unit 114, and the like. The food information recorded in the food list 131 is updated.

The image pickup time setting unit 116 detects that both the right door 11 and the left door 12 are in the closed state from the time when the right door 11 or the left door 12 is switched from the closed state to the open state by the open / close sensors 18a and 18b. The time point at which the image is taken by the camera 60 is set until the image is taken. The imaging time point setting unit 116 assumes a first imaging time assuming a time point when the food stored in the refrigerating chamber 10 passes through the opening 10a, and a time point when the food discharged from the refrigerating room 10 passes through the opening 10a. The time point of the second imaging is set.

[3. Processing by control unit]
Next, with reference to FIGS. 6 to 14, a process relating to the management of the food contained in the refrigerating chamber 10 executed by the control unit 100 will be described.

FIG. 6 is a flowchart showing an outline of a series of processes executed by the control unit 100. The control unit 100 has "imaging processing" in step S1, "weight information processing" in step S2, "image analysis processing" in step S3, "weight sensor correction processing" in step S4, and "food management processing in the refrigerator" in step S5. "Is executed respectively. The details of each process will be described below.

[3-1. Imaging processing]
The execution procedure of the "imaging process" will be described according to the flowchart shown in FIG. The "imaging process" is executed by the passing object detection unit 111, the food information acquisition unit 114, and the imaging time point setting unit 116. In step S10, the food information acquisition unit 114 recognized that one or both of the right door 11 and the left door 12 of the refrigerating chamber 10 was switched from the closed state to the open state by the detection signals of the open / close sensors 18a and 18b. From time to time, in step S15, until it is recognized that both the right door 11 and the left door 12 are closed, imaging by the camera 60 is performed at the first imaging time and the second imaging time.

When the food information acquisition unit 114 recognizes in step S10 that one or both of the right door 11 and the left door 12 of the refrigerating chamber 10 has been switched from the closed state to the open state, the process proceeds to step S10. Steps S11 to S14 are processes by the passing object detection unit 111 and the imaging time point setting unit 116. In the present embodiment, the image pickup time point setting unit of the present invention is configured by the configuration in which the passing object detection unit 111 and the image pickup time point setting unit 116 are combined.

In step S11, the passing object detection unit 111 detects each illuminance (illuminance of each space partitioned by the shelves) of the upper shelf 13, the middle shelf 14, and the lower shelf 15 of the refrigerating chamber 10 by the illuminance sensors 17a, 17b, 17c. .. Then, the passing object detection unit 111 uses the detected illuminance of the upper shelf 13 as the upper reference illuminance, the illuminance of the middle shelf 14 as the middle reference illuminance, and the illuminance of the lower shelf 15 as the lower reference illuminance. The passing object detection unit 111 stores the data of the passing object in the storage unit 130.

For the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, one or both of the right door 11 and the left door 12 of the refrigerating chamber 10 are opened, and an object (user's hand, food, etc.) is an opening 10a of the refrigerating chamber 10. It is the illuminance of each shelf 13, 14, 15 in the state before passing through. In consideration of the error and fluctuation of the detected illuminance, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance are set to illuminance slightly lower than or slightly higher than the detected illuminance of each illuminance sensor 17a, 17b, 17c. You may.

Further, even if the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S13, which will be described later, and the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S14 are set to different values. good. In this case, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S13 correspond to the first reference illuminance of the present invention, and the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S14. Corresponds to the second reference illuminance of the present invention.

In the following step S12, the passing object detection unit 111 detects the illuminance of the upper shelf 13, the middle shelf 14, and the lower shelf 15 by the illuminance sensors 17a, 17b, and 17c, respectively. Then, in the next step S13, the passing object detection unit 111 determines whether or not there is a shelf whose detected illuminance is lower than the reference illuminance by a predetermined level or more. Specifically, the passing object detection unit 111 determines the success or failure of each of the following conditions (1) to (3).

(1) The detected illuminance of the illuminance sensor 17a changed from "upper reference illuminance" to "upper reference illuminance-α" or less.
(2) The detected illuminance of the illuminance sensor 17b changed from "medium reference illuminance" to "medium reference illuminance-α" or less.
(3) The detected illuminance of the illuminance sensor 17c changed from "lower reference illuminance" to "lower reference illuminance-α" or less.

However, α: Determined by experiments, computer simulations, etc., assuming a decrease in illuminance that occurs when an object (food, user's hand, etc.) enters the upper shelf 13, the middle shelf 14, or the lower shelf 15. A constant for setting the threshold. In addition, α may be a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, and may be set to different values individually for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance. May be good.

The passing object detection unit 111 determines that the object has passed through the opening 10a when at least one of the above (1) to (3) is satisfied. Here, when at least one of the above (1) to (3) is satisfied, the user can put the food in the refrigerator compartment 10 (warehousing) or take out the food from the refrigerator compartment 10 (delivery). Alternatively, it is presumed that the inside of the refrigerating room 10 has been modified in order to change the position of the food in the refrigerating room 10. Then, in this case, the passing object detection unit 111 can determine that the object (user's hand, food held by the user's hand, etc.) has passed through the opening 10a of the refrigerating chamber 10.

Therefore, the imaging time point setting unit 116 sets the time point when it is determined by the passing object detection unit 111 that at least one of the above (1) to (3) is satisfied in step S13 as the first imaging time point. The process proceeds to step S20. The first imaging time is a time set assuming a time when the food stored in the refrigerating chamber 10 passes through the opening 10a. Even if the time point after a predetermined time from the time when at least one of the above (1) to (3) is determined to be satisfied by the passing object detection unit 111 in step S13 is set as the first imaging time point. good.

On the other hand, when none of the above (1) to (3) is satisfied, the imaging time point setting unit 116 proceeds to the process in step S14. Step S14 is a process by the passing object detection unit 111 and the imaging time point setting unit 116. The passing object detection unit 111 determines whether or not there is a shelf whose detected illuminance has changed from "reference illuminance-β" or less to "reference illuminance" or more. Specifically, the passing object detection unit 111 determines the success or failure of each of the following conditions (4) to (6).

(4) The detected illuminance of the illuminance sensor 17a changed from "upper reference illuminance-β" or less to "upper reference illuminance" or more.
(5) The detected illuminance of the illuminance sensor 17b changed from "medium reference illuminance-β" or less to "medium reference illuminance" or more.
(6) The detected illuminance of the illuminance sensor 17c changed from "lower reference illuminance-β" or less to "lower reference illuminance" or more.

However, β: the illuminance generated when an object (food, user's hand, etc.) that has entered the upper shelf 13, the middle shelf 14, or the lower shelf 15 passes through the opening 10a and goes out of the refrigerating chamber 10. A constant for determining the threshold value determined by experiments, computer simulations, etc., assuming an increase in. Note that β may be set in common for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance. You may.

In the passing object detection unit 111, when at least one of the above (4) to (6) is satisfied, the object (food, user's hand, etc.) existing in the refrigerating chamber 10 is removed from the refrigerating chamber 10. It is determined that the vehicle has exited and passed through the opening 10a. Here, when at least one of the above (4) to (6) is satisfied, the user can put the food in the refrigerator compartment 10 (warehousing), take out the food from the refrigerator compartment 10 (delivery), or It is presumed that the position of the food stored in the refrigerating room 10 was changed, and the hand placed in the refrigerating room 10 was removed. Then, in this case, the passing object detection unit 111 can determine that the object (user's hand, food held by the user's hand, etc.) has passed through the opening 10a of the refrigerating chamber 10.

Therefore, the imaging time point setting unit 116 sets the time point at which at least one of the above (4) to (6) is determined by the passing object detection unit 111 in step S14 as the second imaging time point. The process proceeds to step S22. The second imaging time point is a time point set assuming a time point when the food discharged from the refrigerating chamber 10 passes through the opening 10a. In step S14, the time after a predetermined time from the time when at least one of the above (4) to (6) is determined by the passing object detection unit 111 in step S14 is defined as the second imaging time. It may be set.

On the other hand, when none of the above (4) to (6) is satisfied, the passing object detection unit 111 proceeds to the process in step S15. In step S15, the passing object detection unit 111 determines from the detection signals of the open / close sensors 18a and 18b whether or not both the right door 11 and the left door 12 of the refrigerating chamber 10 are closed. Then, when the right door 11 and the left door 12 are both closed, the passing object detection unit 111 proceeds to step S15 and ends the “imaging process”. On the other hand, when at least one of the right door 11 and the left door 12 is open, the passing object detection unit 111 proceeds to the process in step S12.

Next, steps S20 to S21 and steps S22 to S23 are processes by the food information acquisition unit 114. In step S20, the food information acquisition unit 114 captures a range including the opening 10a of the refrigerating chamber 10 by the camera 60 via the image processing unit 120. In the following step S21, the food information acquisition unit 114 stores the data of the image captured in step S20 as the data of the "image at the time of warehousing" in the storage unit 130, and proceeds to the process in step S15. It should be noted that the imaging may be performed a plurality of times and the data of the plurality of images may be stored in the storage unit 130 as the data of the “image at the time of warehousing”.

Further, in step S22, the food information acquisition unit 114 captures a range including the opening 10a of the refrigerating chamber 10 by the camera 60 via the image processing unit 120. In the following step S23, the food information acquisition unit 114 stores the data of the image captured in step S22 in the storage unit 130 as the data of the “image at the time of delivery”, and proceeds to the process in step S15. It should be noted that the imaging may be performed a plurality of times and the data of the plurality of images may be stored in the storage unit 130 as the data of the “image at the time of delivery”.

The refrigerating room is based on the change in the illuminance of the upper shelf 13, the middle shelf 14, or the lower shelf 15 detected by the illuminance sensors 17a, 7b, 17c by the passing object detection unit 111 by the "imaging process" shown in FIG. An object passing through the opening 10a of 10 is detected.

Further, the “image at the time of warehousing” is an image taken in the vicinity of the opening 10a at the time when the passing object detection unit 111 detects an object that has passed through the opening 10a and entered any of the shelves 13, 14, 15. Is stored in the storage unit 130. Further, the image taken at the vicinity of the opening 10a at the time when the passing object detection unit 111 detects an object that has passed through the opening 10a from any of the shelves 13, 14, 15 is referred to as a “delivery image”. Is stored in the storage unit 130.

In step S15 of FIG. 7, when an object passing through the opening 10a is detected a plurality of times in step 13 before it is detected that the right door 11 and the left door 12 are closed, steps S20 to S21. As a result, a plurality of "images at the time of warehousing" are captured and saved. Then, the "image analysis process" is executed for each of the "images at the time of warehousing" of the foods whose warehousing is determined by the "weight information processing" described later.

Similarly, in step S15 of FIG. 7, when an object passing through the opening 10a is detected a plurality of times in step S14 before it is detected that the right door 11 and the left door 12 are closed, steps S22 to S22 to By S23, a plurality of "images at the time of delivery" are captured and stored. Then, an "image analysis process" is executed for each of the "images at the time of delivery" of the food whose delivery has been determined by the "weight information processing" described later.

[3-2. Weight information processing]
Next, the execution procedure of the "weight processing" will be described according to the flowchart shown in FIG. "Weight information processing" is executed by the warehousing / delivery determination unit 113 and the food information acquisition unit 114.

Steps S30 to S33 and S40 in FIG. 8 are processes by the warehousing / delivery determination unit 113. In step S30, the warehousing / delivery determination unit 113 recognizes the weights of the shelves 13, 14 and 15 by the storage capacity recognition unit 112 and stores them in the storage unit 130. In the following step S31, when an object passing through the opening 10a of the refrigerating chamber 10 is detected by the passing object detection unit 111 in step S13 or step S14 of FIG. 7 (the object has entered and exited the refrigerating chamber 10). When), the warehousing / delivery determination unit 113 proceeds to the process in step S32.

In step S32, the warehousing / delivery determination unit 113 recognizes the weights of the shelves 13, 14 and 15 by the storage capacity recognition unit 112. In the next step S33, the warehousing / delivery determination unit 113 determines whether or not there is a shelf in which the weight recognized in step S32 has increased with respect to the weight recognized in step S30. Then, when there is a shelf whose weight has increased (in this case, it can be determined that the food has been stored in the refrigerating chamber 10), the warehousing / delivery determination unit 113 proceeds to the process in step S34.

On the other hand, when there is no shelf whose weight has increased, the warehousing / delivery determination unit 113 proceeds from step 33 to step S40, and there is a shelf in which the weight recognized in step S32 is reduced with respect to the weight recognized in step S30. Judge whether or not. Then, when there is a shelf whose weight has been reduced (in this case, it can be determined that the food has been delivered from the refrigerating chamber 10), the warehousing / delivery determination unit 113 proceeds to the process in step S41. Further, when there is no shelf whose weight has increased (in this case, it can be determined that the food has not been warehousing in the refrigerating room 10 and the food has not been warehousing from the refrigerating room 10), the warehousing / delivery determination unit 113 may perform step S36. Proceed to process and end "weight information processing".

Steps S34 to S35 and steps S41 are processes by the food information acquisition unit 114. In step S34, the food information acquisition unit 114 estimates the storage position of the stored food from the left and right weights of the shelves 13, 14, and 15 recognized by the storage capacity recognition unit 112.

In the next step S35, the food information acquisition unit 114 records information on the weight and storage position of the stored food in the stored food information 132 (see FIG. 5), proceeds to the process in step S36, and "weight information processing". To finish. The data of the in-stock food information 132 is temporarily stored in the storage unit 130 in order to add or update the information to the in-store food list 131.

Further, in step S41, the food information acquisition unit 114 records the weight information of the delivered food in the delivered food information 133 (see FIG. 5), proceeds to the process in step S36, and ends the "weight information processing". The data of the delivered food information 133 is temporarily stored in the storage unit 130 in order to add or update the food information to the in-store food list 131.

[3-3. Image analysis processing]
Next, the execution procedure of the "image analysis process" will be described according to the flowcharts shown in FIGS. 9 and 10. The "image analysis process" is executed by the food information acquisition unit 114. FIG. 9 is a process of acquiring information from the “image at the time of warehousing” of the food warehousing in the refrigerating room 10. Further, FIG. 10 is a process of acquiring information from the “image at the time of delivery” of the food delivered from the refrigerating room 10.

First, with reference to FIG. 9, the "image analysis process" for the "image at the time of warehousing" will be described. In step S50 of FIG. 9, the food information acquisition unit 114 executes character extraction processing by the image processing unit 120 for the “image at the time of warehousing” of the warehousing food. Here, the warehousing food is the food that is determined to have been warehousing in the refrigerating chamber 10 by the warehousing / delivery determination unit 113 in step S33 of FIG. Further, the “image at the time of warehousing” of the warehousing food is a “image at the time of warehousing” captured and stored by the food information acquisition unit 114 in steps S20 and S21 of FIG. 7 for the food determined to be warehousing in step S33 of FIG. ".

In the following step S51, the food information acquisition unit 114 determines whether or not the character has been extracted from the "image at the time of warehousing" by the image processing unit 120. Then, the food information acquisition unit 114 proceeds to the process in step S52 when the character is extracted, and proceeds to the process in step S60 when the character is not extracted.

In step S52, the food information acquisition unit 114 analyzes the character information extracted by the image processing unit 120. Then, in the following step S53, the food information acquisition unit 114 acquires information on the stored food such as the food name, the expiration date, the production area, and the content from the text information. In the next step S54, the food information acquisition unit 114 records the acquired information on the stored food in the stored food information 132, and ends the “image analysis process”.

Further, in step S60, the food information acquisition unit 114 extracts the image portion of the stored food from the “image at the time of storage” by the image processing unit 120. In the following step S61, the food information acquisition unit 114 calculates the matching rate between the extracted image portion of the stored food and the sample images 134 (see FIG. 5) of various foods previously stored in the storage unit 130.

In the next step S62, the food information acquisition unit 114 presumes and acquires the food name of the sample image 134 having the highest matching rate as the food name of the stored food, and proceeds to the process in step S54. By the processing of steps S60 to S62, even when characters cannot be extracted from the "image at the time of warehousing" of the warehousing food, the food name of the warehousing food can be estimated and recorded in the warehousing food information 132.

Next, with reference to FIG. 10, the "image analysis process" for the "image at the time of delivery" will be described. In step S70 of FIG. 10, the food information acquisition unit 114 executes a character extraction process by the image processing unit 120 for the “image at the time of delivery” of the food to be delivered. Here, the delivered food is the food determined to have been delivered from the refrigerating chamber 10 by the warehousing / delivery determining unit 113 in step S40 of FIG. Further, the “image at the time of delivery” of the food to be delivered is a “image at the time of delivery” captured and stored by the food information acquisition unit 114 in steps S22 and S23 of FIG. 7 for the food determined to be delivered in step S40 of FIG. ".

In the following step S71, the food information acquisition unit 114 determines whether or not the character has been extracted from the “image at the time of delivery” by the image processing unit 120. Then, the food information acquisition unit 114 proceeds to the process in step S72 when the character is extracted, and proceeds to the process in step S80 when the character is not extracted.

In step S72, the food information acquisition unit 114 analyzes the character information extracted by the image processing unit 120. Then, in the following step S73, the food information acquisition unit 114 acquires information on the delivered food such as the food name, the expiration date, the production area, and the content from the text information. In the next step S74, the food information acquisition unit 114 records the acquired food information in the food information 133, and ends the “image analysis process”.

Further, in step S80, the food information acquisition unit 114 extracts the image portion of the food to be delivered from the “image at the time of delivery” by the image processing unit 120. In the following step S81, the food information acquisition unit 114 calculates the matching rate between the extracted image portion of the delivered food and the sample image 134 of various foods previously stored in the storage unit 130.

In the next step S82, the food information acquisition unit 114 presumes and acquires the food name of the sample image 134 having the highest match rate as the food name of the delivered food, and proceeds to the process in step S74. By the processing of steps S80 to S82, even when characters cannot be extracted from the "image at the time of delivery" of the food to be delivered, the food name of the food to be delivered can be estimated and recorded in the food information 133 to be delivered.

The image analysis process of FIG. 9 is performed on the image at the time of warehousing when the warehousing to the refrigerating room 10 is determined, and the image analysis processing of FIG. 10 is the warehousing when the warehousing from the refrigerating room 10 is determined. It is performed for the time image. Therefore, when the right door 11 or the left door 12 is detected by the passing object detection unit 111 instead of the food, the image analysis process is not performed because it is not determined to be warehousing or warehousing (the captured image becomes invalid). This makes it possible to prevent erroneous food information from being acquired from the image portion of the right door 11 or the left door 12.

[3-4. Weight sensor correction process]
According to the flowchart shown in FIG. 11, the execution procedure of the “weight sensor calibration process” for correcting the detected weight of the weight sensors 13a, 13b, 14a, 14b, 15a, 15b will be described. The "weight sensor correction process" is executed by the accommodation capacity recognition unit 112. The "weight sensor correction process" is a process for correcting (calibrating) the detection error of the weight sensors 13a, 13b, 14a, 14b, 15a, 15b that may occur due to changes over time.

In step S90 of FIG. 11, when the storage capacity recognition unit 112 determines that the food has been stored in the refrigerating room by the warehousing / delivery determination unit 113 (YES in step S33 of FIG. 8), the processing proceeds to step S91. .. In step S91, the storage capacity recognition unit 112 determines whether or not the information on the content of the stored food is recorded in the stored food information 132 (processing in step S54 in FIG. 9). Then, when the storage amount recognition unit 112 records the information of the content of the stored food (the content of the food acquired in step S53 of FIG. 9) in the stored food information 132, the processing proceeds to step S92.

In step S92, the storage capacity recognition unit 112 calculates the ratio between the weight of the stored food and the content recorded in the stored food information 132. Then, the accommodation capacity recognition unit 112 sets the calculated ratio as the correction coefficient of the weight sensor used for detecting the weight of the stored food. For example, when the stored food is stored in the upper shelf 13 and the weight is detected by the weight sensors 13a and 13b, the storage capacity recognition unit 112 sets the calculated ratio as the correction coefficient of the detected weight of the weight sensors 13a and 13b. do.

In the following step S93, the accommodation capacity recognition unit 112 stores the correction coefficient data in the storage unit 130, proceeds to the process in step S94, and ends the "weight sensor correction process". Hereinafter, the accommodation capacity recognition unit 112 treats the value obtained by multiplying the detected weights of the weight sensors 13a and 13b by the correction coefficient as the detected weight. Similarly, the storage capacity recognition unit 112 sets the correction coefficients of the weight sensors 14a and 14b when the stored food is stored in the middle shelf 14, and when the stored food is stored in the lower shelf 15, the weight is set. The correction coefficients of the sensors 15a and 15b are set.

On the other hand, if the content information is not recorded in the in-stock food information 132 in step S91, the accommodation capacity recognition unit 112 proceeds to step S94 and ends the “weight sensor correction process”. In this case, the correction coefficient is not calculated or set.

[3-5. Food management processing in the warehouse]
According to the flowcharts shown in FIGS. 12 and 13, the execution procedure of the “food management process in the refrigerator” for managing the information of the food stored in the refrigerating chamber 10 will be described. The “food management process in the refrigerator” is executed by the food information management unit 115.

The food information management unit 115 manages the information of the food stored in the refrigerating room 10 by the food list 131 in the refrigerator (see FIG. 5). As shown in FIG. 14, in the food list 131 in the refrigerator, the numbers (1, 2, 3, ...) Attached to the foods stored in the refrigerating room 10 and the information of the foods stored in the refrigerators are numbered. The food name, weight (first time, current, amount of food), warehousing / delivery date / time (first warehousing, latest warehousing, latest warehousing) inventory (1: yes, 0: none), expiration date, and place of origin are recorded. .. The food information management unit 115 generates and updates the in-store food list 131 based on the in-stock food information 132 and the outbound food information 133 recorded by the food information acquisition unit 114.

In step S100 of FIG. 12, the food information management unit 115 determines whether or not the warehousing / delivery determination unit 113 determines whether or not the food is warehousing in the refrigerating room 10 or the food is warehousing from the refrigerating room 10. When the delivery is determined, the process proceeds to step S101. In step S101, the food information management unit 115 determines whether or not the warehousing is determined, and if the warehousing is determined, proceeds to step S102. On the other hand, when the warehousing is not determined (when the warehousing is determined), the food information management unit 115 proceeds to step S110.

In step S102, the food information management unit 115 determines whether or not there has been a delivery from the refrigerating room 10 within a predetermined time. Then, if there is a warehousing from the refrigerating room 10 within a predetermined time, the process proceeds to step S103, and if there is no warehousing from the refrigerating room 10 within a predetermined time, the process proceeds to step S111. It can be recognized from the food list 131 in the refrigerator that the refrigerator has been delivered from the refrigerator within a predetermined time. Further, the predetermined time is set to, for example, about 10 minutes, assuming that the user uses a part of the food discharged from the refrigerating room 10 and returns it to the refrigerating room 10 (restocking).

In step S103, the food information management unit 115 determines whether or not the matching rate between the food delivered last time and the food received this time is equal to or higher than a predetermined value. Here, the food information management unit 115 compares the information of the previously delivered product recorded in the food list 131 in the warehouse with the product and information recorded this time in the stored food information 132, and foods. From the matching status of the name, expiration date, production area, etc., the matching rate between the food that was delivered last time and the food that was received this time is calculated.

Then, when the matching rate between the food delivered last time and the food received this time is equal to or higher than the predetermined value, the food information management unit 115 proceeds to step S104, and the food delivered last time and the food received this time are received. When the concordance rate with is less than a predetermined value, the process proceeds to step S112.

Step S104 is a process in which the matching rate between the food delivered last time and the food received this time is equal to or higher than a predetermined value, and it can be determined that the food delivered last time and the food received this time are the same. Is. In step S104, the food information management unit 115 sets the current weight of the food delivered before the predetermined time, the latest warehousing date and time, and the amount of food (difference between the first warehousing and the current weight) in the food list 131 in the warehousing. It is updated based on the information recorded in the information 132, and the process proceeds to step S105 in FIG.

Step S112 is a process in which the matching rate between the food delivered this time and the food delivered this time is less than a predetermined value, and it can be determined that the food delivered last time and the food received this time are different. In step S112, the food information management unit 115 newly registers the stored food based on the information recorded in the stored food information 132 in the stored food list 131, and proceeds to the process in step S105 of FIG.

Step S111 is a process when there is no delivery within a predetermined time. In step S111, the food information management unit 115 newly registers the stored food based on the information recorded in the stored food information 132 in the stored food list 131, and proceeds to the process in step S105 of FIG.

Step S110 is a process when it is determined that the food has been delivered from the refrigerating chamber 10. In step S110, the food information management unit 115 updates the current weight of the corresponding food (delivered food) to 0 in the in-store food list 131. Further, the food information management unit 115 updates the amount of food consumed in the food list 131 in the warehouse, updates the latest delivery time based on the information recorded in the food information 133, and steps 105 in FIG. Proceed to the process.

In step S105 of FIG. 13, the food information management unit 115 determines whether or not there is a food whose weight is currently 0 gram by the processing of step S110. Then, the food information management unit 115 proceeds to step S105 when there is a food having a weight of 0 g, and proceeds to step S120 when there is no food having a weight of 0 g.

In step S106, the food information management unit 115 updates the information of the corresponding food (food determined to have been delivered this time) in the food list 131 in the warehouse to'out of stock'(stock 0), and proceeds to step S107. Proceed with processing. Further, in step S120, the food information management unit 115 updates the information of the corresponding food (food determined to have been received this time) in the food list 131 in the warehouse to'in stock'(inventory 1), and steps. Proceed to S107.

In step S107, the food information management unit 115 determines whether or not the information on the expiration date by character recognition is recorded in the in-stock food information 132. Then, the food information management unit 115 proceeds to the process in step S108 when the information on the expiration date is recorded in the stored food information 132, and proceeds to the process in step S121 when the information on the expiration date is not recorded.

In step S108, the food information management unit 115 registers the expiration date of the food received this time in the in-store food list 131 based on the expiration date information recorded in the stored food information 132, and processes in step S109. Proceed. Further, in step S121, the food information management unit 115 searches for a general expiration date of the corresponding food with reference to the expiration date list 135 stored in the storage unit 130.

As shown in FIG. 14, the best-by-date list 135 is a list in which the food name and the best-by-date of the food are associated with each other, and the food information management unit 115 has the best-by-date associated with the food name of the food in stock this time. Search for deadlines. In the following step S122, the food information management unit 115 registers the expiration date in the data of the corresponding food in the food list 131 in the refrigerator, and proceeds to the process in step S109.

In step S109, the food information management unit 115 transmits the updated data of the food list 131 in the refrigerator to the management server 520 via the communication network 510 by the wireless communication unit 140, and ends the “food management process in the refrigerator”. do.

The management server 520 stores the data of the food list 131 in the refrigerator received from the refrigerator 1 in a storage device (not shown), and the eating status of each product recognized from the food list 131 in the refrigerator, the number of days remaining until the expiration date, etc. Is transmitted to the user's terminal device 400. As a result, the user can check the status of the products stored in the refrigerator 1 by the terminal device 400 even when he / she is out.

[4. Another Embodiment of Imaging Processing]
Next, another embodiment of the "imaging process" will be described according to the flowchart shown in FIG. The flowchart shown in FIG. 15 is executed by the food information acquisition unit 114 and the imaging time point setting unit 116. The imaging time point setting unit 116 sets the time point at which the first predetermined time has elapsed since the right door 11 or the left door 12 of the refrigerating chamber 10 was opened as the first imaging time point, and the right door 11 and the left door of the refrigerating chamber 10 are set. The time point before the second predetermined time from the time when 12 is closed is set as the second imaging time point.

When the food information acquisition unit 114 recognizes in step S130 that one or both of the right door 11 and the left door 12 of the refrigerating chamber 10 is switched from the closed state to the open state from the detection signals of the open / close sensors 18a and 18b. Then, in step S137, until it is recognized that both the right door 11 and the left door 12 have switched from the closed state to the open state, the camera 60 performs imaging at the first imaging time and the second imaging time.

When the food information acquisition unit 114 recognizes in step S130 that one or both of the right door 11 and the left door 12 of the refrigerating chamber 10 has been switched from the closed state to the open state, the process proceeds to step S131. Steps S131 to S132 are processes by the imaging time point setting unit 116.

In step S131, the imaging time point setting unit 116 starts a timer in which the first predetermined time is set as the time counting time until the time is up. Then, in the following step S132, when the timer times out (when the first imaging time is reached), the process proceeds to step S133.

Steps S133 to S138 are processes by the food information acquisition unit 114. The food information acquisition unit 114 executes image capture by the camera 60 a plurality of times in step S133, and stores the image data of the image captured in the storage unit 130 as data of the “image at the time of warehousing” in the subsequent step S134.

Following the loop of steps S135 to S137, the food information acquisition unit 114 in steps S135 and S136 until it is detected in step S137 that the right door 11 and the left door 12 are closed by the open / close sensors 18a and 18b. Repeat the process. In step S135, the food information acquisition unit 114 performs image pickup by the camera 60 and records the image pickup time in the storage unit 130, and in step S136, the image data of the image captured is used as image candidate data at the time of delivery in the storage unit 130. save. As a result, the data of the captured images in the time series are sequentially stored in the storage unit 130.

When the food information acquisition unit 114 detects in step S137 that the right door 11 and the left door 12 are closed by the open / close sensors 18a and 18b, the food information acquisition unit 114 proceeds to the process in step S138. In step S138, the food information acquisition unit 114 detected that both the right door 11 and the left door were closed in step S137 from the data of the image candidate at the time of delivery stored in the storage unit 130. The image data captured before the second predetermined time from the time point is stored in the storage unit 130 as an image at the time of delivery.

As another configuration for setting the first imaging time point and the second imaging time point by the image pickup time point setting unit 116, the object detection status by the objective sensor 80 (see FIGS. 3 to 5) for detecting the object passing through the opening 10a is set. Based on this, the first imaging time point and the second imaging time point may be set. In this case, the imaging time point setting unit 116 sets the time point at which a predetermined time has elapsed from this time point as the first image pickup time point, based on the time point at which the object is switched from the undetected state to the detected state by the objective sensor 80. Set. Further, the imaging time point setting unit 116 is based on the time when the state in which the object is detected by the objective sensor 80 is subsequently switched to the state in which the object is not detected, for example, a time point before a predetermined time. Is set at the time of the second imaging.

[5. Other embodiments]
In the above embodiment, the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerating chamber 10 by the change of the illuminance in the refrigerating chamber 10 detected by the illuminance sensors 17a, 17b, 17c. As another configuration of the passing object detection unit 111, as shown in FIGS. 3 to 5, the objective sensor 80 (reflection type or transmission type) that includes the opening 10a in the detection range and directly detects an object passing through the opening 10a. A type of optical sensor, ultrasonic sensor, capacitance sensor, etc.) may be used. In this case, the detection range of the objective sensor 80 may be set toward the front side of the image pickup range of the camera 60 in the direction from the opening 10a toward the inside of the refrigerating chamber 10. As a result, it is possible to prevent the image of the food to be stored from failing due to the delay time from the time when the object passing through the opening 10a is detected to the time when the image is taken by the camera 60.

Further, when the opening / closing sensor 18a detects that the left door 12 has been opened, and the opening / closing sensor 18b detects that the right door 11 has been opened, the camera 60 is energized to start energizing the camera. Imaging by 60 may be possible. According to this configuration, when the left door 12 and the right door are closed, the power supply to the camera 60 can be cut off to save power. Further, by starting the energization of the camera 60 at the timing when the right door or the left door 12 is opened, it is possible to prevent the image of the food to be stored from failing due to the delay in starting the camera 60.

Further, in the food information acquisition unit 114, the left door 12 is switched from the open state to the closed state by the open / close sensor 18a within a predetermined time from the time when the passing object detection unit 111 detects the object passing through the opening 10a. When it is detected that the right door 11 is switched from the open state to the closed state by the open / close sensor 18b, the image data captured by the camera 60 may be invalidated. According to this configuration, when the left door 12 or the right door 11 is closed, when the left door 12 or the right door 11 is detected by the passing object detection unit 111 and imaged by the camera 60, the left door 12 or the right It is possible to prevent erroneous food information from being extracted from the image of the door 11.

In the above embodiment, the storage capacity recognition unit 112 is the food stored in the shelves 13, 14, 15 of the refrigerating chamber 10 based on the weight detected by the weight sensors 13a, 13b, 14a, 14b, 15a, 15b. Recognized the amount. As another configuration of the accommodation capacity recognition unit 112, each of the refrigerating chambers 10 is based on the detected illuminance by the illuminance sensors 17a, 17b, 17c in the state where the lighting units 16a, 16b, 16c, 16d, 16e, 16f are lit. The amount of food contained in the shelves 13, 14 and 15 may be recognized. Alternatively, the amount of food contained in the shelves 13, 14, 15 of the refrigerating chamber 10 can be determined by extracting an image portion of the food from the image captured inside the refrigerating chamber 10 by the camera 60 or a separately arranged camera. You may recognize it.

In the above embodiment, the image processing unit 120 performs the processing on the captured image of the camera 60, but the CPU 110 may be provided with the function of the image processing unit 120 and the CPU 110 may perform the processing on the captured image.

In the above embodiment, the data of the sample image 134 and the expiration date list 135 are stored in the storage unit 130, but they are stored in the management server 520 and downloaded from the management server 520 to the control unit 100 of the refrigerator 1 at the time of use. You may.

Further, although the sample image 134 is stored in the storage unit 130 or the management server 520, an image recognition program in which the characteristics of each food or stored item such as color, shape, and size are learned in advance is mounted on the CPU 110. , The image portion of the stored food may be processed by an image recognition program to calculate the matching rate and presumed to be the food name of the stored food. As a result, the sample image 134 becomes unnecessary, the storage capacity stored in the storage unit 130 or the management server 520 can be reduced, and the recognition processing time for estimating the food name of the stored food can be processed quickly. The estimation of the food name of the delivered food can be processed in the same manner as described above.

In the above embodiment, for the refrigerating room 10 of the refrigerator 1, the warehousing / delivery determination unit 113 determines the warehousing / delivery of food, the food information acquisition unit 114 acquires information on the warehousing food and the warehousing food, and the food information management unit. Information on the food contained in the refrigerating room 10 was managed by 115. The switching chamber 20, the freezing chamber 40, and the vegetable compartment 50 also have a passing object detection unit that detects an object passing through the openings 20a, 40a, and 50a, and a storage capacity recognition unit that recognizes the amount of food contained. Thereby, it is possible to determine the receipt and delivery of food. Further, by providing a camera that captures a range including the openings 20a, 40a, and 50a and acquiring information on the in-stock food and the out-of-delivery food from the captured images (image at the time of warehousing and images at the time of warehousing), the switching chamber 20 is frozen. Information on the food contained in the room 40 and the vegetable room 50 can be acquired to manage the food.

As described above, the refrigerator according to the present invention is stored in the storage room by taking an image of the food stored in the storage room and the food discharged from the storage room assuming the time when the food passes through the storage room. Since information on food and food delivered from the storage room can be obtained more reliably, it can be applied to the use of managing food stored in the refrigerator.

1 Refrigerator 10 Refrigerator room 10a Opening 11 Right door 12 Left door 13 Upper shelf 13 Shelf 13a, 13b Weight sensor 14 Middle shelf 14a, 14b Weight sensor 15 Lower shelf 15a, 15b Weight sensor 16a, 16b, 16c, 16d, 16r, 16f Illumination unit 17a, 17b, 17c Illumination sensor 18a, 18b Open / close sensor 20 Switching room 20a Opening / closing sensor 30 Ice making room 30a Opening / closing sensor 32 Step 32 Opening / closing sensor 40 steps 40 Freezing room 40a Opening / closing sensor 50 Vegetables Room 50a Opening 52 Open / close sensor 60 Camera 61 Compressor 62 Cooling fan 63 Cooler 64 Condenser 100 Control unit 110 CPU
111 Passing object detection unit 112 Storage capacity recognition unit 113 Entry / exit judgment unit 114 Food information acquisition unit 115 Food information management unit 116 Imaging time setting unit 120 Image processing unit 121 Image memory 130 Storage unit 131 Food list in storage 132 Food information in warehousing 133 Outgoing food information 134 Sample image 135 Expiration date list 140 Wireless communication unit 141 Date and time recognition unit 400 Terminal equipment 500 Gateway 510 Communication network 520 Management server

Claims (8)

It ’s a refrigerator,
A containment room with an opening and
The door provided in the containment chamber and
A camera that captures the imaging range including the opening, and
The illuminance sensor provided in the containment chamber and
The illuminance detected by the illuminance sensor between the time when the door is switched from the closed state to the open state and the time when the door is closed is the illuminance detected when the door is switched from the closed state to the open state. A refrigerator provided with a first control means for causing the camera to take an image when the value drops by the first predetermined value or more . The door consists of a right door and a left door.
The illuminance detected by the illuminance sensor is measured between the time when one of the right door and the left door is switched from the closed state to the open state and the time when both the right door and the left door are closed. When the illuminance detected at the time when one of the right door and the left door is switched from the closed state to the open state is lowered by the first predetermined value or more , the first control means causes the camera to take an image. The refrigerator according to claim 1, characterized in that. The door consists of a right door and a left door.
The illuminance detected by the illuminance sensor between the time when both the right door and the left door are switched from the closed state to the open state and the time when both the right door and the left door are closed is the right. A claim characterized in that the first control means causes the camera to take an image when the illuminance detected at the time when both the door and the left door are switched from the closed state to the open state is lowered by the first predetermined value or more. Item 1 The refrigerator according to item 1. A second control means for causing the camera to take an image based on the detection result of the illuminance sensor is further provided after the camera is made to take an image by the first control means until the door is closed . The refrigerator according to any one of claims 1 to 3, wherein the refrigerator is characterized by the above. The refrigerator according to claim 4, wherein when the illuminance detected by the illuminance sensor rises by a second predetermined value or more , the second control means causes the camera to take an image. The containment chamber is separated by multiple shelves and is separated by multiple shelves.
When the illuminance detected by the illuminance sensor provided on each of the plurality of shelves drops by the first predetermined value or more from the illuminance detected when the door is switched from the closed state to the open state. The refrigerator according to any one of claims 1 to 5, wherein the first control means causes the camera to take an image. The storage room is provided at the top of the refrigerator.
The refrigerator according to any one of claims 1 to 6 , wherein the camera is provided on the top surface of the storage chamber. The refrigerator according to any one of claims 1 to 7 , wherein the storage room is a refrigerating room.

Images