JP2019070478A - refrigerator - Google Patents

Abstract

To provide a refrigerator capable of acquiring information of food brought in a storage chamber and taken out of the storage chamber more surely.SOLUTION: A food information acquisition part performs imaging by a camera 60 in a first imaging time point which is set by assuming the time point when food to be brought in passes an opening 10a between the time when a right door 11 or a left door is switched from a closed state to an open state and the time when both of the right door 11 and the left door are brought into the closed state, and stores the photographed image in a storage part as a brought-in-time image, and it performs imaging by the camera 60 in a second imaging time point which is set by assuming the time point when food to be taken out passes the opening 10a and stores the photographed image in the storage part as a taken-out-time image, and based on the brought-in-time image and the taken-out-time image, it acquires the information of food which is brought into a refrigeration chamber 10 and food which is taken out of the refrigeration chamber.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a refrigerator.

Conventionally, an in-chamber camera is installed in the central portion of the accommodation chamber for food, and when the door of the accommodation chamber is opened and closed, the accommodation chamber is imaged with the in-chamber camera, and when food is accommodated from captured images in time series A refrigerator has been proposed in which the image of the image and the current image are arranged and output (for example, see Patent Document 1). In the refrigerator described in Patent Document 1, depending on the arrangement state of the food in the storage chamber, the food may overlap in the imaging direction of the in-house camera, and some food may not be recognized from the captured image.
Therefore, a switch that detects the opening and closing of the storage chamber door and a camera that takes the opening of the storage chamber as the imaging range are provided, and the food is not stored in the storage chamber, but is stored from the opening when the door is opened. Alternatively, there has been proposed a refrigerator configured to image food to be delivered (see, for example, Patent Document 2).

JP, 2016-57022, A JP, 2015-81762, A

In the refrigerator described in the patent document 2, when the opening of the door of the storage chamber is detected, the food which passes through the opening of the storage chamber is imaged. Therefore, depending on the timing from when the user opens the door to when the food is put in or out, the food may not be imaged. In addition, not a food but a part of a door pocket or the like may be imaged, and an object other than the food may be erroneously recognized as the food.
This invention is made in view of this background, and it aims at providing the refrigerator which can acquire more reliably the information of the foodstuff stored in a storage room, and the food discharged from a storage room. .

  In order to solve the above-mentioned conventional problems, the refrigerator according to the present invention includes a storage chamber having an opening, an opening and closing unit opening and closing the opening, an opening and closing sensor detecting the opening and closing state of the opening and closing unit, and the opening The open / close unit is opened from the open state by the open / close sensor when it is detected that the open / close unit is switched from the closed state to the open state by the camera for capturing an imaging range including the storage unit and the open / close sensor Before it is detected that the food has been switched to the camera, the camera performs imaging at the first imaging time which is set on the assumption that the food stored in the storage room passes through the opening, The captured image at the first imaging time is stored as the storage image in the storage unit as a storage image, and the second imaging time is set by the camera on the assumption that the food to be discharged from the storage room passes the opening. Imaging Then, the captured image at the second imaging time is stored in the storage unit as the leaving image, and the food and the storage room are stored in the storing chamber based on the storing image and the leaving image. And a food information acquisition unit for acquiring information of the food to be delivered.

  Thereby, the food information acquisition unit opens the image at the time of warehousing captured at the first imaging time assuming that the food stored in the storage unit passes through the opening of the storage unit, and the food discharged from the storage chamber is opened The leaving-time image captured at the second imaging time point assuming the time point of passing through the unit is stored in the storage unit. Then, the food information acquisition unit further determines the information on the food to be stored in the storage room and the food to be output from the storage room based on the storage time image and the storage time image that are likely to include the image portion of the food. It can be acquired surely.

  According to the refrigerator of the present invention, the food stored in the storage room and the food discharged from the storage room are stored by imaging the vicinity of the opening with a camera on the assumption that the food passes through the opening of the storage room. It is possible to more reliably obtain information on the food that is stored in the room and the food that is output from the storage room.

Explanatory drawing of the structure which confirms the condition of the food accommodated in the refrigerator installed in the house by a terminal device Illustration of the appearance of the refrigerator Explanatory drawing which showed the structure in the inside of the refrigerator of a refrigerator by sectional drawing seen from the right side Explanatory drawing which showed the internal structure of the refrigerator compartment by sectional drawing seen from the front Control unit configuration diagram Flow chart showing overall processing by control unit Flow chart of imaging process Flowchart of weight information processing Flow chart of analysis process of image at storage Flow chart of analysis processing at the time of delivery Flow chart of weight sensor correction process First flow chart of in-house food management processing Second flow chart of in-house food management processing Explanatory diagram of in-house food list and shelf life criteria list Flowchart of another embodiment of imaging processing

According to a first aspect of the present invention, there is provided a storage chamber having an opening, an opening and closing unit for opening and closing the opening, an opening and closing sensor for detecting the opening and closing state of the opening and closing unit, a camera for imaging an imaging range including the opening. And when the open / close sensor detects that the open / close unit has switched from the open state to the closed state since the open / close sensor detects that the open / close unit has switched from the closed state to the open state In the meantime, at the time of warehousing the captured image at the first imaging time, the imaging is performed by the camera at the first imaging time which is set on the assumption that the food stored in the storage chamber passes the opening. The image is captured by the camera at a second imaging time point which is stored as the image in the storage unit and the food taken out of the storage room passes the opening, and the imaging is performed by the camera, and the second imaging time point Imaging in An image is stored in the storage unit as an exit image, and based on the in-storage image and the exit image, food to be received in the storage room and information on food output from the storage room is acquired based on the in-storage image and the exit image. And an information acquisition unit.
According to the first aspect of the invention, the food information acquisition unit receives the storage image taken at the first imaging time assuming the time when the food stored in the accommodation unit passes through the opening of the accommodation unit, and the delivery from the accommodation room The storage image is stored in the storage unit at the second imaging time point assuming that the food to be stored passes through the opening. Then, the food information acquisition unit further determines the information on the food to be stored in the storage room and the food to be output from the storage room based on the storage time image and the storage time image that are likely to include the image portion of the food. It can be acquired surely.

A second aspect of the invention is based on an illuminance sensor for detecting the illuminance in the storage chamber, and a detected illuminance of the illuminance sensor when it is detected that the open / close unit has switched from the closed state to the open state by the open / close sensor. , Setting a first reference illuminance and a second reference illuminance, setting the first imaging time based on a point in time when the detected illuminance of the illuminance sensor becomes equal to or less than the first reference illuminance, and the detected illuminance of the illuminance sensor And an imaging time point setting unit configured to set the second imaging time point based on a time point at which the first reference illuminance is equal to or less than the second reference illuminance.
According to the second aspect of the invention, the illuminance in the storage chamber changes in accordance with the presence of an object (food, user's hand, etc.) in the storage chamber. Therefore, based on the change in the illuminance in the storage chamber detected by the illuminance sensor, the first imaging time when food stored in the storage chamber passes through the opening, and the food discharged from the storage chamber is opened. It is possible to appropriately set the second imaging time point on the assumption of the time point of passing through the unit.

According to a third aspect of the invention, a point in time when a first predetermined time has elapsed from a point in time when it is detected by the open / close sensor that the open / close unit has switched from the closed state to the open state is set as the first imaging time point. The imaging time point setting unit is configured to set, as the second imaging time point, a time point that is a second predetermined time before a time point when the sensor detects that the open / close unit switches from the open state to the closed state.
According to the third invention, the time from when the user opens the opening / closing part of the containing part until when the food is put in is stored, and the time when the user takes out the food from the containing room until when the opening / closing part is closed It can be expected that it will be a relatively short fixed time in order to maintain the cold storage condition. Therefore, the first imaging time point and the second imaging time point can be set by the simple configuration of setting the first predetermined time and the second predetermined time.

According to a fourth aspect of the invention, the food information acquisition unit is configured to wait until the open / close sensor detects that the open / close unit has switched from the open state to the closed state after the first imaging time point has elapsed. The imaging by the camera is repeatedly performed to store the captured image in the storage unit, and when the open / close sensor detects that the open / close unit has switched from the open state to the closed state, the open / close unit A captured image captured in the second predetermined time before the detection of the switch from the open state to the closed state and stored in the storage unit is stored as the leaving image.
According to the fourth aspect of the invention, the food information acquisition unit captures an image by the camera after the first imaging time point has elapsed and until the open / close sensor detects that the open / close unit has switched from the open state to the closed state. Are repeatedly executed to save the captured image in the storage unit. And thereby, the image pick-up picture in the time of 2nd predetermined time before the time of having detected that the opening-and-closing part changed from the open state to the closed state by open-and-close sensor can be saved as a leaving time image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited by the present embodiment.

[1. Usage of refrigerator]
FIG. 1 shows a usage mode of the refrigerator of the present embodiment. The refrigerator 1 according to the present embodiment is installed in a house H and has a function of communicating with the management server 520 via the gateway 500 and the communication network 510. In addition, the refrigerator 1 has a function of performing communication with a terminal device (a smartphone, a tablet terminal, and the like) 400. The terminal device 400 has a function of communicating with the management server 520 via the gateway 500 and the communication network 510.

  The control unit 100 provided in the refrigerator 1 controls the overall operation of the refrigerator 1 and generates an in-box food list 131 indicating information on the food stored in the storage chamber of the refrigerator 1 to manage the management server 520. Send to The application for management of the refrigerator executed in the terminal device 400 is in the storage by acquiring the data of the storage in the storage list 131 from the management server 520 or by acquiring the data of the storage in the storage list 131 from the refrigerator 1 Information of the food accommodated in the refrigerator 1 is displayed on the display unit based on the food list 131. The user P of the refrigerator 1 can confirm the condition (food name, expiration date, remaining amount, etc.) of the food accommodated in the refrigerator 1 by visually recognizing the display on the terminal device 400.

[2. Configuration of refrigerator]
The configuration of the refrigerator 1 will be described with reference to FIGS. 2 to 5. FIG. 2 is an explanatory view showing the appearance of the refrigerator 1. As shown in FIG. 2, the refrigerator 1 is provided with a main box 2 opened at the front, and in the main box 2, a refrigerator compartment 10 (corresponding to the storage compartment of the present invention), an ice making room 30, an ice making room A switching room 20, a freezing room 40, and a vegetable room 50 which are juxtaposed to 30 and in which the temperature inside the storage can be changed are formed.

  At the front opening of the refrigerator compartment 10, rotary right and left doors 11 and 12 (corresponding to the opening and closing part of the present invention) are provided. Further, drawers 21, 31, 41, 51 for storing food are provided in the switching room 20, the ice making room 30, the freezing room 40, and the vegetable room 50, respectively.

  Next, FIG. 3 is an explanatory view showing the internal configuration of the refrigerator 1 as viewed from the right side, and FIG. 4 is an explanatory view showing the internal configuration of the refrigerator compartment 10 as viewed from the front. is there. With reference to FIG. 3 and FIG. 4, an upper shelf 13, a middle shelf 14, and a lower shelf 15 are disposed in the refrigerator compartment 10 to partition the refrigerator compartment 10 into sub areas. In the refrigerator compartment 10, weight sensors 13a and 13b for detecting the weight of food placed on the upper shelf 13, weight sensors 14a and 14b for detecting weight of food placed on the middle shelf 14, and a lower shelf 15 Weight sensors 15a and 15b are provided to detect the weight of the placed food. Furthermore, in the refrigerator compartment 10, illumination units 16a to 16f for illuminating the inside of the refrigerator compartment 10, illuminance sensors 17a to 17c for detecting the illuminance in the refrigerator compartment 10, and a camera 60 are provided.

  The camera 60 is arranged in the vicinity of the opening 10a in the upper part of the refrigerator compartment 10 so that the opening 10a is included in the imaging range, and images a range from the opening 10a to the vicinity of the front end of each shelf 13, 14, 15. . In addition, a compressor 61, which is an auxiliary device constituting the refrigeration cycle, a cooling fan 62, a cooler 63, and a condenser 64 are provided. Furthermore, on the back surface of the refrigerator compartment 10, a refrigerator compartment duct 70 and cold air discharge ports 71 to 73 for circulating cold air are disposed.

  In the refrigerator compartment 10, an open / close sensor 18b for detecting the opening / closing of the right door 11 and an open / close sensor 18a for detecting the opening / closing of the left door 12 are provided. The open / close sensor 18b outputs a close detection signal when the right door 11 is closed, and outputs an open detection signal when the right door 11 is open. Similarly, the open / close sensor 18a outputs a close detection signal when the left door 12 is closed, and outputs an open detection signal when the left door 12 is open.

  The switching chamber 20 is provided with an open / close sensor 22 for detecting the opening / closing of the opening 20a. The open / close sensor 22 outputs a close detection signal when the drawer 21 is accommodated in the switching chamber 20, and outputs an open detection signal when the drawer 21 is withdrawn from the switching chamber 20.

  Similarly, the ice making chamber 30 is provided with an open / close sensor 32 for detecting the opening / closing of the opening 30 a by the drawer 31. Further, the freezing chamber 40 is provided with an open / close sensor 42 for detecting the opening / closing of the opening 40a by the drawer 41, and the vegetable compartment 50 is provided with the opening / closing sensor 52 for detecting the opening / closing of the opening 50a by the drawer 51. The open / close sensors 32, 42, 52 output the close detection signal and the open detection signal, as in the case of the open / close sensor 22.

  Next, FIG. 5 is a block diagram of the control unit 100. As shown in FIG. Referring to FIG. 5, the control unit 100 is an electronic device constituted by a central processing unit (CPU) 110, an image processing unit 120, a storage unit 130, a wireless communication unit 140, a date and time recognition unit 141, and an interface circuit not shown. It is a circuit unit.

  The CPU 110 executes the control program for controlling the refrigerator 1 stored in the storage unit 130, whereby the passing object detection unit 111, the storage amount recognition unit 112, the storage and storage determination unit 113, the food information acquisition unit 114, and the food information management unit 115, and functions as an imaging time point setting unit 116.

  The control unit 100 includes open / close sensors 18a, 18b, 22, 32, 42, 42, illuminance sensors 17a, 17b, 17c, weight sensors 13a, 13b, 14a, 14b, 15a, 15b, a camera 60, illumination units 16a, 16b, It is connected to 16c, 16d, 16e, 16f and refrigeration cycle accessories 61, 62, 63, 64.

  Detection signals and the like of the open / close sensors 18a, 18b, 22, 32, 42, 42, the illuminance sensors 17a, 17b, 17c, and the weight sensors 13a, 13b, 14a, 14b, 15a, 15b are input to the control unit 100. . In addition, the operation of the illumination units 16a, 16b, 16c, 16d, 16e, 16f, the refrigeration cycle accessories 61, 62, 63, 64 and the like is controlled by the control signal output from the control unit 100.

  Furthermore, the imaging by the camera 60 is controlled via the image processing unit 120 by the control signal output from the control unit 100. The image processing unit 120 converts a video signal output from the camera 60 into a digital gradation signal to generate a captured image, and stores data of the captured image in the image memory 121. The image processing unit 120 performs processing on a captured image stored in the image memory 121 in accordance with a control signal output from the CPU 110.

  The date and time recognition unit 141 counts a clock signal output from a clock circuit (not shown) provided in the control unit 100 to recognize the current date and time (date and time). Information on the current date and time may be acquired from the management server 520 through communication with the management server 520 to recognize the current date and time.

  In the storage unit 130, in addition to the data of the in-warehouse food list 131, the warehousing food information 132 in which the information of the food received in the cold storage 10 is recorded, and the information of the food delivered from the cold storage 10 are recorded Data of leaving food information 133 is stored. Furthermore, the storage unit 130 stores sample images 134 of various foods and data of a shelf life date list 135 indicating general shelf life of various foods.

  The passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 based on the change in the illuminance in the refrigerator compartment 10 detected by the illuminance sensors 17a, 17b and 17c. The storage capacity recognition unit 112 is based on the detection signals of the weight sensors 13a, 13b, 14a, 14b, 15a, 15b for the amount of food stored in the upper shelf 13, middle shelf 14, and lower shelf 15 of the refrigerator compartment 10. And recognize by weight.

  The storage amount recognition unit 112 recognizes the weight of the food stored in the upper shelf 13 based on the detection signals of the weight sensors 13a and 13b, and the weight of the food stored in the middle shelf 14 is the weight sensor 14a, It recognizes based on the detection signal of 14b. Further, the storage amount recognition unit 112 detects the weight of the food stored in the lower shelf 15 based on the detection signal of the weight sensors 15a and 15b.

  The storage / storage determination unit 113 is placed on each shelf 13, 14, 15 recognized by the storage capacity recognition unit 112 when the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10. Based on the change in the amount (weight) of the food, it is determined whether the storage of the food into the cold storage room 10 and the delivery of the food from the cold storage room 10 are performed.

  The food information acquisition unit 114 picks up an object that has passed through the opening 10 a of the refrigerator compartment 10 with the camera 60. Then, when the storage determination unit 113 determines that the storage of the food into the cold storage room 10 or the delivery of the food from the cold storage room is determined, the image part of the food is extracted from the captured image.

  The food information acquisition unit 114 recognizes the characters included in the image portion of the food, or extracts the characteristics of the image portion of the food, thereby to select the food name, content, origin, and taste of the received or delivered food. Acquire information such as deadlines. And the food information acquisition part 114 records the acquired information in the warehousing food information 132 about the food determined to be warehousing (carried food), and the food information 133 for the food determined to be warehousing (displacement food) Record on

  The food information management unit 115 is based on the weight of the food in the cold storage compartment 10 recognized by the storage capacity recognition unit 112, the received food information 132 recorded by the food information acquisition unit 114, the delivered food information 133, etc. The food information recorded in the food list 131 is updated.

  The imaging time point setting unit 116 detects that the right door 11 and the left door 12 are both closed after the right door 11 or the left door 12 is switched from the closed state to the open state by the open / close sensors 18a and 18b. In the meantime, the time point at which imaging is performed by the camera 60 is set. The imaging time point setting unit 116 assumes a first imaging time point assuming that the food stored in the cold storage room 10 passes through the opening 10a and a time when the food discharged from the cold storage room 10 passes through the opening 10a And set a second imaging time point.

[3. Processing by control unit]
Next, with reference to FIGS. 6-14, the process regarding management of the foodstuff accommodated in the refrigerator compartment 10 performed by the control unit 100 is demonstrated.

  FIG. 6 is a flowchart showing an outline of a series of processes executed by the control unit 100. The control unit 100 performs "imaging processing" in step S1, "weight information processing" in step S2, "image analysis processing" in step S3, "weight sensor correction processing" in step S4, and "in-box food management processing" in step S5. ", Respectively. The details of each process will be described below.

[3-1. Imaging process]
The execution procedure of the "imaging process" will be described according to the flowchart shown in FIG. The “imaging process” is executed by the passing object detection unit 111, the food information acquisition unit 114, and the imaging time point setting unit 116. In step S10, the food information acquisition unit 114 recognizes from the detection signals of the open / close sensors 18a and 18b that one or both of the right door 11 and the left door 12 of the cold storage room 10 are switched from the closed state to the open state. From time to time, in step S15, the camera 60 performs imaging at the first imaging time point and the second imaging time point until it is recognized that both the right door 11 and the left door 12 are in the closed state.

  When it is recognized in step S10 that one or both of the right door 11 and the left door 12 of the cold storage 10 have been switched from the closed state to the open state, the food information acquisition unit 114 proceeds to step S10. Steps S11 to S14 are processes by the passing object detection unit 111 and the imaging time point setting unit 116. In the present embodiment, the imaging time point setting unit of the present invention is configured by a configuration in which the passing object detection unit 111 and the imaging time point setting unit 116 are combined.

  In step S11, the passing object detection unit 111 detects the illuminance of each of the upper shelf 13, middle shelf 14 and lower shelf 15 of the refrigerator compartment 10 (illuminance of each space partitioned by the shelves) by the illuminance sensors 17a, 17b and 17c. . Then, the passing object detection unit 111 sets the detected illuminance of the upper shelf 13 as the upper reference illuminance, the illuminance of the middle shelf 14 as the middle reference illuminance, and the illuminance of the lower shelf 15 as the lower reference illuminance. The passing object detection unit 111 stores the data of the above in the storage unit 130.

  In the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 are opened, and an object (a user's hand, food, etc.) The illuminance of each shelf 13, 14 and 15 in the state before passing through. The upper reference illuminance, the middle reference illuminance, and the lower reference illuminance are set to illuminance slightly lower or higher than the detected illuminance of each of the illuminance sensors 17a, 17b, 17c in consideration of the error and fluctuation of the detected illuminance. You may

  Also, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S13 described later and the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S14 may be set to different values. Good. In this case, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S13 correspond to the first reference illuminance of the present invention, and the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance used in step S14. Corresponds to the second reference illuminance of the present invention.

  In the subsequent step S12, the passing object detection unit 111 detects the illuminance of the upper shelf 13, the middle shelf 14 and the lower shelf 15 by the illuminance sensors 17a, 17b and 17c. Then, in the next step S13, the passing object detection unit 111 determines whether there is a shelf whose detected illuminance is lower than the reference illuminance by a predetermined level or more. Specifically, the passing object detection unit 111 determines whether the following conditions (1) to (3) are met.

(1) The detected illuminance of the illuminance sensor 17a has changed from "upper reference illuminance" to "upper reference illuminance-α" or less.
(2) The detected illuminance of the illuminance sensor 17b changes from "medium reference illuminance" to "middle reference illuminance-α" or less.
(3) The detected illuminance of the illuminance sensor 17c has changed from "lower reference illuminance" to "lower reference illuminance-α" or less.

  However, α: determined by experiments, computer simulations, etc., assuming a decrease in the illuminance that occurs when an object (food, user's hand, etc.) enters the upper shelf 13, middle shelf 14, or lower shelf 15. Constant for setting the threshold. Here, α may be a value common to the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, and is set to values individually different from the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance. It is also good.

  The passing object detection unit 111 determines that the object has passed through the opening 10a when at least one of the above (1) to (3) is established. Here, when at least one of the above (1) to (3) is established, the user puts the food in the refrigerator compartment 10 (storage) or takes out the food from the refrigerator compartment 10 (delivery), Alternatively, it is assumed that a hand is placed in the refrigerator compartment 10 in order to change the position of food in the refrigerator compartment 10 or the like. Then, in this case, the passing object detection unit 111 can determine that the object (the user's hand, the food held by the user's hand, etc.) has passed through the opening 10 a of the refrigerator compartment 10.

  Therefore, the imaging time point setting unit 116 sets, as the first imaging time point, a time point when it is determined that the at least one of the above (1) to (3) is established by the passing object detection unit 111 in step S13. The process proceeds to step S20. The first imaging time point is a time point set on the assumption that the food stored in the refrigerator compartment 10 passes through the opening 10 a. It should be noted that even if it is determined in step S13 that the predetermined time has passed from the time when it is determined by the passing object detection unit 111 that at least one of the above (1) to (3) holds, Good.

  On the other hand, when none of the above (1) to (3) is established, the imaging time setting unit 116 proceeds with the process to step S14. Step S14 is processing performed by the passing object detection unit 111 and the imaging time point setting unit 116. The passing object detection unit 111 determines whether or not there is a shelf in which the detected illuminance has changed from “reference illuminance −β” or less to “reference illuminance” or more. Specifically, the passing object detection unit 111 determines whether the following conditions (4) to (6) are met.

(4) The detected illuminance of the illuminance sensor 17a changes from "upper reference illuminance-β" or less to "upper reference illuminance" or more.
(5) The detected illuminance of the illuminance sensor 17b changes from "middle reference illuminance-β" or less to "middle reference illuminance" or more.
(6) The detected illuminance of the illuminance sensor 17c changes from "lower reference illuminance-β" or less to "lower reference illuminance" or more.

  However, β: illuminance generated when an object (food, user's hand, etc.) entering the upper shelf 13, middle shelf 14, or lower shelf 15 passes through the opening 10a and exits the refrigerator compartment 10 A constant for determining the threshold value determined by experiments, computer simulations, etc., assuming an increase in. Note that β may be set commonly for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or is set to values individually different for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance. You may

  When at least one of the above (4) to (6) is established, the passing object detection unit 111 detects an object (food, user's hand, etc.) present in the refrigerator compartment 10 from the refrigerator compartment 10 It is determined that it has exited and passed through the opening 10a. Here, when at least one of the above (4) to (6) is established, the user puts the food into the cold storage room 10 (storage), takes out the food from the cold storage room 10 (delivery), or It is assumed that the position of the food stored in the refrigerator compartment 10 is changed and the hand put in the refrigerator compartment 10 is removed. Then, in this case, the passing object detection unit 111 can determine that the object (the user's hand, the food held by the user's hand, etc.) has passed through the opening 10 a of the refrigerator compartment 10.

  Therefore, the imaging time point setting unit 116 sets, as the second imaging time point, a time point at which it is determined that at least one of the above (4) to (6) is established by the passing object detection unit 111 in step S14. The process proceeds to step S22. The second imaging time point is a time point set on the assumption that the food product delivered from the refrigerator compartment 10 passes through the opening 10a. Note that, in step S14, a time point after a predetermined time from when the passing object detection unit 111 determines that at least one of the above (4) to (6) is satisfied in step S14 is taken as a second imaging time point. It may be set.

  On the other hand, when none of the above (4) to (6) is established, the passing object detection unit 111 proceeds with the process to step S15. In step S15, the passing object detection unit 111 determines whether or not the right door 11 and the left door 12 of the cold storage room 10 are both closed from the detection signals of the open / close sensors 18a and 18b. Then, when the right door 11 and the left door 12 are both closed, the passing object detection unit 111 proceeds to step S15 and ends the “imaging process”. On the other hand, when at least one of the right door 11 and the left door 12 is open, the passing object detection unit 111 proceeds the process to step S12.

  Next, steps S20 to S21 and steps S22 to S23 are processes by the food information acquisition unit 114. In step S <b> 20, the food information acquisition unit 114 images a range including the opening 10 a of the refrigerator compartment 10 with the camera 60 via the image processing unit 120. In the following step S21, the food information acquisition unit 114 stores the data of the image captured in step S20 in the storage unit 130 as data of "image at storage time", and advances the process to step S15. Note that imaging may be performed a plurality of times, and data of a plurality of images may be stored in the storage unit 130 as data of “image at storage”.

  Further, in step S22, the food information acquisition unit 114 images a range including the opening 10a of the refrigerator compartment 10 with the camera 60 via the image processing unit 120. In the following step S23, the food information acquisition unit 114 stores the data of the image captured in step S22 in the storage unit 130 as data of the "delivery image", and the process proceeds to step S15. Note that imaging may be performed a plurality of times, and data of a plurality of images may be stored in the storage unit 130 as data of “image upon leaving”.

  Based on the change in the illuminance of the upper shelf 13, the middle shelf 14 or the lower shelf 15 detected by the illuminance sensors 17 a, 7 b and 17 c by the passing object detection unit 111 by the “imaging process” shown in FIG. An object passing through the ten openings 10a is detected.

  In addition, an “image at storage” which is a captured image in the vicinity of the opening 10 a at the time when the passing object detection unit 111 detects an object that has passed through the opening 10 a and entered any of the shelves 13, 14, 15 Are stored in the storage unit 130. Furthermore, “the image at the time of delivery” which is a captured image in the vicinity of the opening 10a at the time when the passing object detection unit 111 detects an object that has exited any of the shelves 13, 14 or 15 and has passed through the opening 10a. Are stored in the storage unit 130.

  When an object passing through the opening 10a is detected a plurality of times in step 13 before it is detected in step S15 in FIG. 7 that the right door 11 and the left door 12 are closed, steps S20 to S21 are performed. As a result, imaging and storage of a plurality of "image at storage time" are performed. Then, the “image analysis process” is executed for each “image upon storage” of the food of which the storage is determined by “weight information processing” described later.

  Similarly, when an object passing through the opening 10a is detected a plurality of times in step S14 before it is detected in step S15 in FIG. 7 that the right door 11 and the left door 12 are closed, step S22 to step S22 By S23, imaging and storage of a plurality of "outgoing images" are performed. Then, the “image analysis process” is performed on each “food-at-delivery image” of the food of which the delivery is determined by “weight information processing” described later.

[3-2. Weight information processing]
Next, the execution procedure of "weight processing" will be described according to the flowchart shown in FIG. The “weight information processing” is executed by the storage and storage determination unit 113 and the food information acquisition unit 114.

  Steps S30 to S33 and S40 in FIG. In step S30, the storage / receipt determination unit 113 recognizes the weight of each shelf 13, 14, 15 by the storage capacity recognition unit 112, and stores the weight in the storage unit 130. In the subsequent step S31, when an object passing through the opening 10a of the refrigerator compartment 10 is detected by the passing object detection unit 111 in step S13 or step S14 of FIG. 7 (the object has entered or exit the refrigerator compartment 10) At the same time, the storage / withdrawal determination unit 113 advances the process to step S32.

  In step S32, the storage / receipt determination unit 113 causes the storage amount recognition unit 112 to recognize the weight of each shelf 13, 14, 15. In the next step S33, the storage / withdrawal determination unit 113 determines whether there is a shelf in which the weight recognized in step S32 has increased relative to the weight recognized in step S30. Then, when there is a shelf whose weight has increased (in this case, it can be determined that the food has been stored in the refrigerator compartment 10), the storage / storage determination unit 113 proceeds with the process to step S34.

  On the other hand, when there is no shelf whose weight has increased, the storage / storage determination unit 113 proceeds from step 33 to step S40, and there is a shelf in which the weight recognized in step S32 is reduced relative to the weight recognized in step S30. Determine if it is or not. Then, when there is a shelf whose weight has decreased (in this case, it can be determined that the food has been delivered from the refrigerator compartment 10), the storage / storage determination unit 113 proceeds with the process to step S41. In addition, when there is no shelf whose weight has increased (in this case, it can be determined that the storage of the food into the cold storage room 10 and the delivery of the food from the cold storage room 10 have not been performed), Process and end “weight information processing”.

  Steps S34 to S35 and step S41 are processing by the food information acquisition unit 114. In step S34, the food information acquisition unit 114 estimates the storage position of the stored food from the left and right weights of the shelves 13, 14 and 15 recognized by the storage capacity recognition unit 112.

  In the next step S35, the food information acquisition unit 114 records the information on the weight and storage position of the received food in the received food information 132 (see FIG. 5), proceeds to step S36, and performs “weight information processing” Finish. The data of the stored food information 132 is temporarily stored in the storage unit 130 in order to add information to the in-house food list 131, update the information, and the like.

  Further, in step S41, the food information acquisition unit 114 records the information on the weight of the output food in the output food information 133 (see FIG. 5), proceeds the process to step S36, and ends the “weight information processing”. The data of the leaving food information 133 is temporarily stored in the storage unit 130 in order to add or update the food information to the in-chamber food list 131.

[3-3. Image analysis processing]
Next, the execution procedure of the “image analysis process” will be described according to the flowcharts shown in FIG. 9 and FIG. The “image analysis process” is performed by the food information acquisition unit 114. FIG. 9 is a process of acquiring information from the “image at storage time” for the food stored in the cold storage room 10. Moreover, FIG. 10 is a process which acquires information from "the image at the time of leaving" about the foodstuff delivered from the refrigerator compartment 10. As shown in FIG.

  First, with reference to FIG. 9, the “image analysis process” for the “image at time of storage” will be described. The food information acquisition unit 114 causes the image processing unit 120 to perform character extraction processing on the "image at storage time" of the stored food at step S50 in FIG. Here, the stored food is the food which is determined to be stored in the refrigerator compartment 10 by the storage and storage determination unit 113 in step S33 of FIG. 8. In addition, the "image at storage time" of the storage food is the "image at storage time" captured and stored by the food information acquisition unit 114 in steps S20 and S21 of FIG. 7 for the food determined to be storage in step S33 of FIG. ".

  In the subsequent step S51, the food information acquisition unit 114 determines whether the character has been extracted by the image processing unit 120 from the "image at storage time". Then, when the character is extracted, the food information acquisition unit 114 proceeds with the process to step S52, and when the character is not extracted, the food information acquiring unit 114 proceeds with the process to step S60.

  In step S52, the food information acquisition unit 114 analyzes the information of the character extracted by the image processing unit 120. Then, in the subsequent step S53, the food information acquisition unit 114 acquires information of the received food such as the food name, the expiration date, the production area, and the content from the information of the character. In the next step S54, the food information acquisition unit 114 records the acquired information on the stored food in the stored food information 132, and ends the “image analysis process”.

  Further, in step S60, the food information acquisition unit 114 causes the image processing unit 120 to extract the image portion of the received food from the “image at storage time”. In the subsequent step S61, the food information acquisition unit 114 calculates the matching rate between the extracted image portion of the stored food and the sample images 134 (see FIG. 5) of various food items stored in advance in the storage unit 130.

  In the next step S62, the food information acquisition unit 114 estimates and acquires the food name of the sample image 134 with the highest matching rate as the food name of the received food, and advances the process to step S54. By the processing of steps S60 to S62, even if characters can not be extracted from the “image at storage” of the storage food, the food name of the storage food can be estimated and recorded in the storage food information 132.

  Next, with reference to FIG. 10, the “image analysis process” for the “delivery image” will be described. In step S70 of FIG. 10, the food information acquisition unit 114 causes the image processing unit 120 to perform character extraction processing on the “delivery image” of the delivered food. Here, the delivered food is a food which is determined in step S40 of FIG. 8 to have been delivered from the cold storage room 10 by the storage and retrieval unit 113. In addition, the “at-delivery image” of the delivered food refers to the “at-delivery image taken and stored by the food information acquisition unit 114 at steps S22 and S23 of FIG. 7 for the food determined to be delivered at step S40 of FIG. ".

  In the subsequent step S71, the food information acquisition unit 114 determines whether the character has been extracted from the “delivery image” by the image processing unit 120. Then, when the character is extracted, the food information acquisition unit 114 proceeds with the process to step S72. When the character is not extracted, the food information acquiring unit 114 proceeds with the process to step S80.

  In step S72, the food information acquisition unit 114 analyzes the information of the character extracted by the image processing unit 120. Then, in the subsequent step S73, the food information acquisition unit 114 acquires information of the delivered food such as the food name, the expiration date, the production area, and the content from the information on the character. In the next step S74, the food information acquisition unit 114 records the acquired information on the exited food in the exited food information 133, and ends the “image analysis process”.

  In step S80, the food information acquisition unit 114 causes the image processing unit 120 to extract the image portion of the delivered food from the "delivery image". In the subsequent step S81, the food information acquisition unit 114 calculates the matching rate between the extracted image portion of the stored food and the sample images 134 of various food items stored in advance in the storage unit 130.

  In the next step S82, the food information acquisition unit 114 estimates and acquires the food name of the sample image 134 with the highest matching rate as the food name of the delivered food, and the process proceeds to step S74. Even if characters can not be extracted from the “at-delivery image” of the delivered food by the processes at steps S80 to S82, it is possible to estimate the food name of the delivered food and record it in the delivered food information 133.

  The image analysis process of FIG. 9 is performed on the image at the time of storage when the storage in the cold storage room 10 is determined, and the image analysis process of FIG. 10 is a delivery when the storage from the cold storage room 10 is determined. It is performed for the time image. Therefore, when the right door 11 or the left door 12 is detected by the passing object detection unit 111 instead of the food, the image analysis processing is not performed (the captured image is invalidated) because it is not determined to be warehousing or leaving. As a result, it is possible to prevent acquisition of erroneous food information from the image portion of the right door 11 or the left door 12.

[3-4. Weight sensor correction process]
The execution procedure of the "weight sensor calibration process" for correcting the detected weight of the weight sensors 13a, 13b, 14a, 14b, 15a, 15b will be described according to the flowchart shown in FIG. The “weight sensor correction process” is performed by the storage amount recognition unit 112. The “weight sensor correction process” is a process for correcting (calibrating) a detection error component of the weight sensors 13a, 13b, 14a, 14b, 15a, 15b which may occur due to a change with time or the like.

  In step S90 in FIG. 11, when the storage / receipt determination unit 113 determines that the storage of food is in the cold storage (YES in step S33 in FIG. 8), the storage capacity recognition unit 112 proceeds to step S91. . In step S91, the storage capacity recognition unit 112 determines whether or not the information on the content of the stored food is stored in the stored food information 132 (processing of step S54 in FIG. 9). Then, when the information on the content of the received food (the content of the food acquired in step S53 of FIG. 9) is recorded in the received food information 132, the storage capacity recognition unit 112 proceeds the process to step S92.

  In step S 92, the storage capacity recognition unit 112 calculates the ratio between the weight and the internal capacity of the stored food stored in the stored food information 132. Then, the storage amount recognition unit 112 sets the calculated ratio as the correction coefficient of the weight sensor used for detecting the weight of the received food. For example, when the stored food is stored in the upper shelf 13 and the weight is detected by the weight sensors 13a and 13b, the storage amount recognition unit 112 sets the calculated ratio as a correction coefficient of the detected weight of the weight sensors 13a and 13b. Do.

  In the following step S93, the storage amount recognition unit 112 stores the data of the correction coefficient in the storage unit 130, advances the process to step S94, and ends the "weight sensor correction process". Thereafter, the storage amount recognition unit 112 treats a value obtained by multiplying the detection weight of the weight sensors 13a and 13b by the correction coefficient as a detection weight. Similarly, when the stored food is stored in the middle shelf 14, the storage amount recognition unit 112 sets the correction coefficient of the weight sensors 14a and 14b, and when the stored food is stored in the lower shelf 15, the weight is recognized The correction coefficients of the sensors 15a and 15b are set.

  On the other hand, when the information on the content is not recorded in the received food information 132 in step S91, the storage capacity recognition unit 112 advances the process to step S94 and ends the “weight sensor correction process”. In this case, calculation and setting of the correction coefficient are not performed.

[3-5. In-house food control processing]
An execution procedure of the "in-box food management process" for managing the information of the food stored in the cold storage room 10 will be described according to the flowcharts shown in FIG. 12 and FIG. The “in-box food management process” is executed by the food information management unit 115.

  The food information management unit 115 manages information of the food stored in the refrigerator compartment 10 by the in-storage food list 131 (see FIG. 5). In the in-chamber food list 131, as shown in FIG. 14, the number (1, 2, 3,...) Attached to the food stored in the refrigerator compartment 10 and the information of each received food with the number As the food name, weight (first time, current consumption amount), arrival and withdrawal date (first receipt, latest issue, latest receipt) inventory (1: present, 0: no), shelf life, and production area are recorded. . The food information management unit 115 generates and updates the in-chamber food list 131 based on the stored food information 132 and the output food information 133 recorded by the food information acquisition unit 114.

  In step S100 of FIG. 12, the food information management unit 115 determines whether the storage determination of the food to the cold storage room 10 or the delivery of the food from the cold storage room 10 has been determined by the storage / storage determination unit 113, storage or When the delivery is determined, the process proceeds to step S101. In step S101, the food information management unit 115 determines whether or not the storage determination is made, and when the storage determination is made, the process proceeds to step S102. On the other hand, when the storage determination is not made (when the storage determination is made), the food information management unit 115 proceeds the process to step S110.

  In step S102, the food information management unit 115 determines whether or not there is a delivery from the cold storage room 10 within a predetermined time. Then, the process proceeds to step S103 when there is leaving the cold storage room 10 within the predetermined time, and the process proceeds to step S111 when there is no leaving from the cold storage room 10 within the predetermined time. In addition, it can be recognized from the in-warehouse food list 131 that there is a delivery from the refrigerator compartment 10 within a predetermined time. In addition, the predetermined time is set to, for example, about 10 minutes on the assumption that the user returns a part to the cold storage room 10 (reloading) using a part of the food discharged from the cold storage room 10.

  In step S103, the food information management unit 115 determines whether the matching rate between the previously issued food and the currently received food is equal to or greater than a predetermined value. Here, the food information management unit 115 compares the information of the product that has been issued last time recorded in the in-chamber food list 131 with the information of the currently received product and information recorded in the incoming food information 132, From the matching situation of name, expiration date, production area, etc., the matching rate of the food that was issued last time and the food that was received this time is calculated.

  Then, the food information management unit 115 advances the processing to step S104 when the matching rate between the previously delivered food and the currently received food is equal to or more than the predetermined value, and the previously delivered food and the currently received food If the matching rate with is less than the predetermined value, the process proceeds to step S112.

  Step S104 is a process where it is determined that the matching rate between the previously issued food and the currently received food is equal to or higher than a predetermined value, and the previously issued food and the currently received food are identical. It is. In step S104, the food information management unit 115 stores the current weight, the latest warehousing date, and the eating amount (difference between the weight at the first warehousing and the current weight) of the food delivered before the predetermined time in the in-chamber food list 131 It updates based on the information recorded on the information 132, and advances a process to FIG.13 S105.

  Step S112 is processing in the case where it is determined that the matching rate between the delivered food and the currently received food is less than a predetermined value, and that the previously delivered food and the currently received food are different. In step S112, the food information management unit 115 newly registers the stored food on the basis of the information recorded in the stored food information 132 in the in-chamber food list 131, and advances the process to step S105 in FIG.

  Step S111 is processing in the case where there is no delivery within a predetermined time. In step S111, the food information management unit 115 newly registers the stored food on the basis of the information recorded in the stored food information 132 in the in-chamber food list 131, and advances the process to step S105 in FIG.

  Step S110 is processing when it is determined that the food has been discharged from the refrigerator compartment 10. In step S110, the food information management unit 115 updates the current weight of the corresponding food (disbursed food) to 0 in the in-chamber food list 131. Further, the food information management unit 115 updates the eating amount of the corresponding food in the in-chamber food list 131, and updates the latest delivery time based on the information recorded in the delivery food information 133, and step 105 of FIG. Proceed to

  In step S105 of FIG. 13, the food information management unit 115 determines whether there is any food whose weight is currently 0 gram in the process of step S110. Then, the food information management unit 115 advances the process to step S105 when there is a food whose weight is currently 0 gram, and advances the process to step S120 when there is no food whose weight is currently 0 gram.

  In step S106, the food information management unit 115 updates the information of the corresponding food (food determined to have been delivered this time) in the in-chamber food list 131 to 'not in stock' (in stock 0), and the process proceeds to step S107. Proceed with the process. Further, in step S120, the food information management unit 115 updates the information of the corresponding food (food determined to be received this time) in the in-box food list 131 to 'in stock' (in stock 1), and step The process proceeds to step S107.

  In step S107, the food information management unit 115 determines whether or not the information of the expiration date by character recognition is recorded in the incoming food information 132. Then, the food information management unit 115 proceeds the process to step S108 when the information on the shelf life is recorded in the received food information 132, and proceeds to the process to step S121 when the information on the shelf life is not recorded.

  In step S108, the food information management unit 115 registers the expiration date of the food that is currently received in the in-house food list 131 based on the information on the expiration date recorded in the received food information 132, and the process is performed in step S109. Advance. Further, in step S121, the food information management unit 115 refers to the expiration date list 135 stored in the storage unit 130, and searches for a general expiration date of the corresponding food.

  As shown in FIG. 14, the shelf life expiration date list 135 is a list in which the food name is associated with the shelf life of the food, and the food information management unit 115 stores the flavors associated with the food name of the current stored food. Search for deadlines. In the subsequent step S122, the food information management unit 115 registers the expiration date in the data of the corresponding food in the in-chamber food list 131, and advances the process to step S109.

  In step S109, the food information management unit 115 causes the wireless communication unit 140 to transmit the updated data of the stored food list 131 to the management server 520 via the communication network 510, and ends the "stored food management process". Do.

  The management server 520 stores the data of the in-box food list 131 received from the refrigerator 1 in a storage device (not shown), and the eating condition of each product recognized from the in-box food list 131, the remaining number of days until the expiration date, etc. Are sent to the terminal device 400 of the user. Thereby, the user can confirm the condition of the goods accommodated in the refrigerator 1 by the terminal device 400 even when he or she goes out.

[4. Another embodiment of imaging processing]
Next, another embodiment of the “imaging process” will be described according to the flowchart shown in FIG. The flowchart shown in FIG. 15 is executed by the food information acquisition unit 114 and the imaging time point setting unit 116. The imaging time point setting unit 116 sets the time point when the first predetermined time has elapsed since the right door 11 or the left door 12 of the cold storage room 10 was opened as the first imaging time point, and the right door 11 and the left door of the cold room 10 A point in time which is a second predetermined time before the time of closing 12 is set as a second imaging point of time.

  When the food information acquisition unit 114 recognizes that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 has been switched from the closed state to the open state from the detection signals of the open / close sensors 18a and 18b in step S130 Then, in step S137, while it is recognized that both the right door 11 and the left door 12 are switched from the closed state to the open state, the camera 60 performs imaging at the first imaging time and the second imaging time.

  When it is recognized in step S130 that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have been switched from the closed state to the open state, the food information acquisition unit 114 proceeds with the process to step S131. Steps S131 to S132 are processing by the imaging time point setting unit 116.

  In step S131, the imaging time point setting unit 116 starts a timer that uses the first predetermined time as the timed time until time-up. Then, in the subsequent step S132, when the timer times out (when the first imaging time point has come), the process proceeds to step S133.

  Steps S133 to S138 are processing by the food information acquisition unit 114. In step S133, the food information acquisition unit 114 executes imaging with the camera 60 a plurality of times, and in the subsequent step S134, stores data of the imaged image as data of "image at storage" in the storage unit 130.

  The food information acquiring unit 114 continues the loop of steps S135 and S136 until the open / close sensors 18a and 18b detect that the right door 11 and the left door 12 are closed in step S137. Execute the process repeatedly. In step S135, the food information acquisition unit 114 performs imaging with the camera 60 and records the imaging time in the storage unit 130. In step S136, the data of the imaged image is stored in the storage unit 130 as the image candidate data at the time of delivery. save. As a result, data of time-series captured images are sequentially stored in the storage unit 130.

  When the open / close sensors 18a and 18b detect that the right door 11 and the left door 12 are closed in step S137, the food information acquisition unit 114 proceeds to step S138. In step S138, the food information acquisition unit 114 detects that the right door 11 and the left door are both closed in step S137 from among the data at the time of leaving image candidate stored in the storage unit 130. From the time point, the image data captured a second predetermined time ago is stored in the storage unit 130 as a leaving image.

  As another configuration of setting the first imaging time and the second imaging time by the imaging time setting unit 116, the detection condition of the object by the object sensor 80 (see FIGS. 3 to 5) for detecting the object passing through the opening 10a. The first imaging time point and the second imaging time point may be set based on the above. In this case, the imaging time point setting unit 116 sets, for example, the time point when a predetermined time has elapsed from this time point to the first imaging time point based on the time when the object sensor 80 switches from the state where no object is detected to the state detected. Set Further, based on the point in time when the object sensor 80 switches from the state in which the object is detected to the state in which the object is not detected, the imaging time point setting unit 116 then, for example, a time point before a predetermined time Is set to the second imaging time point.

[5. Other embodiments]
In the said embodiment, the passing object detection part 111 detected the object which passes the opening part 10a of the refrigerator compartment 10 by the change of the illumination intensity in the refrigerator compartment 10 detected by illuminance sensor 17a, 17b, 17c. As another configuration of the passing object detection unit 111, as shown in FIG. 3 to FIG. 5, an objective sensor 80 (reflection type or transmission that directly detects an object passing through the opening 10a) includes the opening 10a in the detection range. Type optical sensor, ultrasonic sensor, capacitance sensor, etc.) may be used. In this case, the detection range of the objective sensor 80 may be set on the front side of the imaging range of the camera 60 in the direction from the opening 10 a to the inside of the refrigerator compartment 10. As a result, it is possible to prevent a failure in imaging of the stored food due to a delay time from when an object passing through the opening 10a is detected to when the imaging by the camera 60 is performed.

  Also, when the open / close sensor 18a detects that the left door 12 is opened, and when the open / close sensor 18b detects that the right door 11 is opened, energization of the camera 60 is started, Imaging by 60 may be enabled. According to this configuration, when the left door 12 and the right door are closed, the camera 60 can be deenergized to save power. Further, by starting energization of the camera 60 at the timing when the right door or the left door 12 is opened, it is possible to prevent failure in imaging of the stored food due to the delay in activation of the camera 60.

  Further, the food information acquisition unit 114 switches the left door 12 from the open state to the closed state by the open / close sensor 18a within a predetermined time from when the passing object detection unit 111 detects an object passing through the opening 10a. When it is detected or when the open / close sensor 18b detects that the right door 11 is switched from the open state to the closed state, the image data captured by the camera 60 may be invalidated. According to this configuration, when the left door 12 or the right door 11 is detected by the passing object detection unit 111 and captured by the camera 60 when the left door 12 or the right door 11 is closed, the left door 12 or the right It is possible to prevent extraction of erroneous food information from the image of the door 11.

  In the above embodiment, the storage capacity recognition unit 112 is configured to store the food stored in each shelf 13, 14, 15 of the refrigerator compartment 10 based on the detected weight by the weight sensors 13a, 13b, 14a, 14b, 15a, 15b. Recognized the amount. As another configuration of the storage capacity recognition unit 112, each of the refrigerator compartment 10 based on the detected illuminance by the illuminance sensors 17a, 17b and 17c in the state where the illumination units 16a, 16b, 16c, 16d, 16e and 16f are turned on. The amount of food contained in the shelves 13, 14, 15 may be recognized. Alternatively, the amount of food contained in each shelf 13, 14, 15 of the refrigerator compartment 10 is obtained by extracting the image part of the food from the captured image of the inside of the refrigerator compartment 10 by the camera 60 or a camera separately disposed. It may be recognized.

  In the above embodiment, the image processing unit 120 performs the process on the captured image of the camera 60. However, the CPU 110 may have the function of the image processing unit 120 and the CPU 110 may perform the process on the captured image.

  Although the data of the sample image 134 and the expiration date list 135 are stored in the storage unit 130 in the above embodiment, they are stored in the management server 520 and downloaded from the management server 520 to the control unit 100 of the refrigerator 1 when used. May be

  In addition, although the sample image 134 is stored in the storage unit 130 or the management server 520, an image recognition program in which the features of each food or stored item such as color, shape, and size are learned in advance is installed in the CPU 110 The image recognition program may process the image portion of the incoming food to calculate the matching rate and estimate the same as the food name of the incoming food. As a result, the sample image 134 becomes unnecessary and the storage capacity stored in the storage unit 130 or the management server 520 can be reduced, and the recognition processing time for estimating the food name of the stored food can be processed quickly. The estimation of the food name of the leaving food can be processed in the same manner as described above.

  In the above embodiment, in the cold storage room 10 of the refrigerator 1, the storage / storage determination unit 113 determines the storage and exit of the food, and the food information acquisition unit 114 acquires the information of the storage food and the storage food, and the food information management unit The information of the food stored in the refrigerator compartment 10 was managed by 115. The switching room 20, the freezing room 40, and the vegetable room 50 also have a passing object detection unit that detects an object passing through the openings 20a, 40a, 50a, and a storage capacity recognition unit that recognizes the amount of food stored. Thus, it is possible to determine whether the food is in or out. Moreover, the camera which images the range containing opening part 20a, 40a, 50a is provided, and switching room 20, frozen by acquiring the information of the warehousing food and the leaving food from the captured image (image at the time of warehousing and image at the time of leaving) Information on food stored in the room 40 and the vegetable room 50 can be acquired to manage food.

  As described above, the refrigerator according to the present invention stores goods in the storage room and food stored in the storage room by imaging the food in a storage room by assuming the time of passing through the storage room. Since the information on the food and the food to be delivered from the storage room can be acquired more reliably, the present invention can be applied to an application for managing food stored in a refrigerator.

DESCRIPTION OF SYMBOLS 1 refrigerator 10 cold storage room 10a opening part 11 right door 12 left door 13 upper shelf 13 shelves 13a and 13b weight sensor 14 middle shelf 14a and 14b weight sensor 15 lower shelf 15a and 15b weight sensor 16a, 16b, 16c, 16d, 16r, 16f Lighting unit 17a, 17b, 17c Illuminance sensor 18a, 18b Opening and closing sensor 20 Switching chamber 20a Opening 22 Opening and closing sensor 30 Icemaker 30a Opening 32 Opening and closing sensor 32 Step 32 Opening and closing sensor 40 Step 40 Freezing chamber 40a Opening 42 Opening and closing sensor 50 Vegetables Chamber 50a Opening 52 Opening and closing sensor 60 Camera 61 Compressor 62 Cooling fan 63 Cooler 64 Condenser 100 Control unit 110 CPU
111 passing object detection unit 112 storage capacity recognition unit 113 entry and exit determination unit 114 food information acquisition unit 115 food information management unit 116 imaging time point setting unit 120 image processing unit 121 image memory 130 storage unit 131 in-chamber food list 132 storage food information 133 Outgoing food information 134 sample image 135 expiration date list 140 wireless communication unit 141 date and time recognition unit 400 terminal device 500 gateway 510 communication network 520 management server

Claims (4)

A storage chamber having an opening;
An opening and closing part for opening and closing the opening;
An open / close sensor that detects the open / close state of the open / close unit;
A camera for imaging an imaging range including the opening;
A storage unit,
Between when the open / close sensor detects that the open / close unit has switched from the closed state to the open state, until the open / close sensor detects that the open / close unit switches from the open state to the closed state The image is taken by the camera at a first imaging time which is set on the assumption that food stored in the storage room passes through the opening, and the imaged image at the first imaging time is taken as an image at the time of warehousing A captured image at the second imaging time point is captured by the camera at a second imaging time point, which is stored in a storage unit and is set on the assumption that the food discharged from the storage chamber passes through the opening. Are stored in the storage unit as an exit image, and based on the in-storage image and the exit image, information on food to be received in the storage room and food to be output from the storage room is acquired. Refrigerator, characterized in that it comprises a goods information acquisition unit. An illuminance sensor for detecting the illuminance in the storage chamber;
A first reference illuminance and a second reference illuminance are set based on the detected illuminance of the illuminance sensor at the time when it is detected by the open / close sensor that the open / close unit has switched from the closed state to the open state. The first imaging time is set based on the time when the detected illuminance falls below the first reference illuminance, and after the detected illuminance of the illuminance sensor falls below the first reference illuminance, it becomes equal to or higher than the second reference illuminance The refrigerator according to claim 1, further comprising: an imaging time point setting unit configured to set the second imaging time point based on the time point.   The time at which a first predetermined time has elapsed from when the open / close sensor detects that the open / close unit has switched from the closed state to the open state is set as the first imaging time, and the open / close unit The imaging point setting unit configured to set, as the second imaging point, a point of time which is a second predetermined time before a point of time when it is detected that the open state is switched to the closed state. Description refrigerator. The food information acquisition unit repeatedly performs imaging by the camera after the first imaging time point has elapsed until the open / close sensor detects that the open / close unit has switched from the open state to the closed state. When the open / close sensor detects that the open / close unit has switched from the open state to the closed state, the open / close sensor changes the open / close unit from the open state to the closed state. The captured image stored in the storage unit and captured at a point in time before the second predetermined time before the point in time at which the switching is detected is stored as the image at the time of delivery. refrigerator.

Images