JP2015072109A - Refrigerator - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a refrigerator easy to use which can preserve and maintain a food product without trouble.SOLUTION: A refrigerator easy to use includes: object specifying means 77 to specify an object from a camera for picturing a storage chamber for storing a food product, and a pictured image; a preservation period memory part 79 for memorizing the object specified by the object specifying means; comparison means 80 to compare the image of the object specified by the object specifying means before and after a prescribed time; and compensation calculation means 81 to compensate a preservation period memorized in the preservation period memory part under a prescribed condition. The compensation calculation means is constituted to change and compensate a preservation period memorized in the preservation period memory part, based on a comparison result of the comparison means. As a result of this, when a change of an image of a food product specified by the object specifying means is small, a preservation period is kept as it is, and when a change is big, a preservation period is changed and compensated, determining that the same food product is bought to change to a new product, or a specified food product is consumed.

Description

  The present invention relates to a refrigerator capable of managing the storage period of food stored in a refrigerator room or the like.

  In general, many refrigerators of this type have been proposed. For example, a refrigerator and a refrigeration system described in Patent Document 1 include an operation input unit such as a keyboard, a barcode reader that reads barcode data of an article, or a microphone for performing voice recognition. Either way, the storage period data such as the expiration date of the article is input. The management information held in the refrigerator can be viewed at any time by a terminal device such as a mobile phone through a network.

  The refrigerator and the refrigeration system described in Patent Document 2 use a camera as an observation means for observing the amount of food in and out, the residence period, and the quality state, and characters and / or codes displayed on the food with this camera. From the processing date, the date of purchase, the item, and the expiration date are read, and the storage time limit is managed.

JP 2003-4368 A JP 2003-30421 A

  However, the configuration described in Patent Document 1 is not practical because it requires a troublesome operation for a user to input a storage period using an operation input unit, a barcode reader, or a microphone.

  In addition, in the case of the one described in Patent Document 2, it is necessary to cause the camera to capture characters and / or codes displayed on the food when food is put in the refrigerator. It is not practical.

  Therefore, the present invention has been made in view of the above points, and is advantageous in that an image is captured by a camera or the like, that is, a storage period can be managed using an image recognition technology, which places a burden on the user. It is an easy-to-use refrigerator that can be stored and managed.

  In order to achieve the above-mentioned object, the present invention provides an imaging means for imaging the inside of a storage room, an object specifying means for specifying an object from an image captured by the imaging means, and an object specified by the object specifying means A storage period storage unit that stores a storage period of the object, a comparison unit that compares images of the object specified by the object specifying unit before and after a predetermined time, and a storage period stored in the storage period storage unit. Correction arithmetic means for correcting under a predetermined condition, wherein the correction arithmetic means changes and corrects the storage period stored in the storage period storage unit based on the comparison result of the comparison means.

Thereby, in this refrigerator, the food specified by the object specifying means is compared with the food for a predetermined time, for example, after food or cooking, and the change in the image is small, for example, the amount is small, or Recognize changes such as small size, keep the storage period as it is, and if there is a big change in the image, store the same food as a new replacement or when the specified food is no longer consumed Change and correct the period. Therefore, the user can simply store the food without taking the trouble of imaging characters displayed on the food, and whether the food is continuously used or newly replaced automatically. Since the determination is made, the food storage management can be reliably performed.

  INDUSTRIAL APPLICABILITY The present invention can provide reliable storage management without taking any trouble at the time of food storage and provide an easy-to-use refrigerator.

The perspective view of the refrigerator in embodiment of this invention Schematic sectional view of the refrigerator The perspective view which shows the inside when the storage chamber door of the refrigerator is opened (A) The perspective view which equipped the same refrigerator with the display terminal integrally, (b) The perspective view which equipped the refrigerator with the display terminal detachably An example of the configuration of a display terminal used in the refrigerator is shown. (A) is a cross-sectional view when the display terminal is integrally attached, and (b) is a cross-sectional view when the display terminal is detachably attached. The example of the display visual recognition structure of the display terminal used for the refrigerator is shown, (a) (b) is a schematic sectional view when the cover panel of the door is made of glass, and (c) is configured by the magic mirror of the cover panel of the door. Schematic cross section when Longitudinal sectional view showing an example of camera mounting of the refrigerator An example of camera mounting of the refrigerator is shown, (a) is a cross-sectional view of the refrigerator door portion with the door closed, and (b) is a front view of the inner surface of the door when the door is opened. (A) (b) (c) The longitudinal cross-sectional view which shows the other example of the camera attachment example of the refrigerator Longitudinal sectional view showing an example of camera installation in the case of a single door of the refrigerator An example of camera mounting of the refrigerator is shown, (a) is a cross-sectional view of the refrigerator door portion with the door closed, and (b) is a front view of the door inner surface when the door is opened. Vertical sectional view showing another example of camera mounting in the case of a single door of the refrigerator Explanatory drawing which shows the imaging state in the camera attachment example of FIG. Explanatory drawing which shows an example of the camera attached to the refrigerator System configuration diagram of refrigerator system in the same embodiment Block diagram showing the refrigerator configuration of the refrigerator system Block diagram showing the server configuration of the refrigerator system Flowchart showing imaging and data storage operations when the door of the refrigerator system is closed Flow chart showing imaging and data storage operations during cooling operation of the refrigerator system in the refrigerator system Flowchart showing imaging and data storage operations when the door of the refrigerator system is opened Flowchart showing another example of the imaging operation in FIGS. Flow chart showing the case where the server device performs data storage and composition operation in the refrigerator system Flow chart showing the case where data storage and composition operations are performed on the refrigerator side in the refrigerator system Flow chart showing the case where data storage and composition operations are performed on the display terminal side in the refrigerator system Flow chart showing normal image data transmission / reception in the refrigerator system Flow chart showing transmission / reception of image data when image data is stored in the display terminal in the refrigerator system Flow chart showing the case of switching between individual images and composite images in the refrigerator system Flowchart showing image data transmission / reception upon advance transmission request in the refrigerator system Front view of display terminal for explaining switching of transmission / reception paths in the refrigerator system Flow chart of image display in the refrigerator in the refrigerator system Flow chart for creating image memo in the refrigerator in the refrigerator system Flow chart of continuous reproduction in the refrigerator in the refrigerator system Block diagram of food storage management control in the refrigerator system Flow chart showing food storage management control in the refrigerator system Flow chart showing an example of food storage management control in the refrigerator system Flow chart showing another example of food storage management control in the refrigerator system The figure for demonstrating the image composition process for demonstrating the image composition process in the refrigerator system

  Hereinafter, specific and detailed embodiments of the present invention will be described. First, the main points of the embodiments of the present invention will be described.

  An embodiment of the refrigerator in the present invention is composed of an inner box and an outer box, and a refrigerator main body having an open front surface, a storage room configured by partitioning the refrigerator main body, a heat insulating door that covers the storage room, An imaging unit that is provided in the storage chamber or the heat insulating door and images the inside of the storage chamber, an object specifying unit that specifies an object from an image captured by the imaging unit, and an object specifying unit A storage period storage unit that stores a storage period of the target object, a comparison unit that compares images of the target object specified by the target object specifying unit before and after a predetermined time, and a storage stored in the storage period storage unit Correction calculating means for correcting the period under a predetermined condition, and the correction calculating means changes and corrects the storage period stored in the storage period storage unit based on the comparison result of the comparison means. There as.

  Thereby, in this refrigerator, the food specified by the object specifying means is compared with the food for a predetermined time, for example, after food or cooking, and the change in the image is small, for example, the amount is small, or Recognize changes such as small size, keep the storage period as it is, and if there is a big change in the image, store the same food as a new replacement or when the specified food is no longer consumed Change and correct the period. Therefore, the user can simply store the food without taking the trouble of imaging characters displayed on the food, and whether the food is continuously used or newly replaced automatically. Since the determination is made, the food storage management can be reliably performed.

  Further, when it is determined in the comparison result of the comparison means that the size of the object of the current image is larger than the size of the object of the previous image, the correction calculation means resets the storage period. As a configuration.

  As a result, the fact that foods with characteristics that become smaller for meals become larger means that the food is not a continuous use food but a newly replaced food, so the storage period is reset and reset, Even for the same food, it is possible to distinguish between continuously used foods and newly replaced foods and to manage the storage period corresponding to each.

Furthermore, in the comparison result of the comparison means, when the absence period of the object is longer than a predetermined time, the correction calculation means is configured to reset the storage period.

  As a result, if the absence period of the object is longer than a predetermined time, for example, the time to eat or cook, the food is judged to be newly replaced by purchase, and the storage period is reset and reset. Even for the same food, it is possible to distinguish between continuously used foods and newly replaced foods and to manage the storage period corresponding to each.

  The object specifying unit is configured to specify the object based on a singular point of the object in the image captured by the image capturing unit.

  As a result, even when the food storage position changes each time the food is taken in and out, the object can be accurately specified, and high-precision storage management is possible.

  Further, the singular point is configured to be the hue of the object.

  Thereby, the fine difference of each foodstuff can be discriminate | determined correctly, and also highly accurate preservation | save management is attained.

  Embodiment 1 of the present invention will be described below with reference to the drawings. Note that the present invention will be described by taking as an example a case where it is applied to a refrigerator and its refrigerator system in which the food storage situation in the refrigerator imaged by a small camera or the like can be confirmed by a display terminal such as a smartphone via the Internet line. However, this does not limit the present invention.

[Embodiment 1]
<1. Configuration>
1-1. Configuration of refrigerator 1-1-1. Configuration of French Door Type Refrigerator First, a configuration example in the case of a French door type refrigerator will be described with reference to FIGS.

  1 to 3, the refrigerator body 1 includes a metal (for example, iron plate) outer box 2 that opens forward, an inner box 3 made of hard resin (for example, ABS), the outer box 2 and the inner box 3. It is made of a heat insulating material 4 such as hard foam urethane filled with foam. The refrigerator main body 1 has a plurality of compartments formed therein. This storage room is located from the upper part of the refrigerator body 1 to the refrigerator compartment 5, the switching chamber 6 located below the refrigerator compartment 5, the ice making chamber 7 juxtaposed to the switching chamber 6, the switching compartment 6, and the ice making chamber 7 below. A freezer compartment 8 and a vegetable compartment 9 located below the freezer compartment 8. The front face of the refrigerator compartment 5 is closed freely by a double door 10 and the front parts of the switching compartment 6, ice making compartment 7, freezer compartment 8, and vegetable compartment 9 are drawer type doors 11, 12, 13 , 14 (hereinafter referred to as a pull-out door) can be freely opened and closed.

  There is a cooling chamber 16 on the back of the refrigerator main body 1, a cooler 17 that generates cool air, a blower fan 18 that supplies the cool air to each chamber, and a damper (air volume adjusting means) for adjusting the air volume to the refrigerator compartment. 22 are provided. In addition, a compressor 19 is provided at the back of the main body top of the refrigerator body 1, and a condenser (not shown), a heat radiating pipe 20, a capillary tube 21, and the cooler 17 described above are sequentially provided. A refrigerant is enclosed in a refrigeration cycle connected in a ring shape, and a cooling operation is performed.

Here, each said door 10-14 is filled with hard foaming urethane similarly to the refrigerator main body 1, and has heat insulation, In this refrigerator, the storage room 5 used as a refrigerator compartment among the said doors 10-14. A display terminal 15 as a display means is provided on one front surface of the door 10. The display terminal 15 is formed of a smartphone, tablet PC, or the like that can communicate with the Internet, which will be described later, and integrally includes a display unit such as a liquid crystal panel and a control unit that controls the display unit, as shown in FIG. As shown in FIG. 4 (b), the door 10 can be attached to or detached from the door 10 so that the door 10 can be attached and detached.

  5 (a) and 5 (b) show examples of the configuration of mounting the display terminal 15 on the door 10, FIG. 5 (a) shows a case where the display terminal 15 is integrally attached, and FIG. The case where it is detachably mounted is shown.

  5 (a) and 5 (b), the door 10 mainly includes an outer cover panel 10a facing the front side of the refrigerator, an inner plate 10b facing the refrigerator interior, an outer cover panel 10a and an inner plate. It is comprised with the heat insulating material 4 with which it filled between 10b. The jacket panel 10a is made of a metal member such as a color steel plate or a transparent member such as a glass plate, and the inner plate 10b is a synthetic resin member formed by vacuum forming. A raised wall portion 10b-1 protruding inward in a square shape when viewed from the inner side is provided in a peripheral portion of the inner plate 10b. Further, as already described, for example, hard foamed urethane is used for the heat insulating material 4.

  Further, in the door 10 of FIG. 5A in which the display terminal 15 is integrated, a resin storage frame body 23 is provided on the front side of the door 10 so as to cover the heat insulating material 4 continuously with the jacket panel 10a. (If the jacket panel 10a is a glass plate, it is integrally bonded to the glass plate). The display terminal 15 is fixed to the storage frame body 23 via the terminal holding body 24. A touch panel 25 is provided on the front side of the display terminal 15. A gap 26 is provided between the display terminal 15 and the touch panel 25 so that the touch panel 25 does not malfunction due to noise from the display terminal 15. The display terminal 15 and the touch panel 25 are fixed by attaching a frame body 27 covering the outer periphery of the display terminal 15 and the touch panel 25 to the surface of the door 10. Thus, since the heat insulating material 4 does not touch the display terminal 15 directly by providing the storage frame 23, when the heat insulating material 4 is foamed, the display terminal 15 is exposed to a high temperature or deformed. It is possible to prevent problems such as On the other hand, in the door 10 of FIG. 5B in which the display terminal 15 is detachable, a recess 28 is formed by providing the storage frame 23 on the front surface of the door 10, and the display terminal 15 is attached to and detached from the recess 28. It is comprised so that it may do.

  6A, 6B, and 6C show an example of a display see-through configuration of the door 10 and the display terminal 15, and FIGS. 6A and 6B show that the outer cover panel 10a of the door 10 is a glass plate. In the case of 10a-1, FIG.3 (c) shows the case where the jacket panel 10a of the door 10 is comprised by the magic mirror 10a-2.

  In FIG. 6A, a smoke sheet 29 in which only a portion facing the display terminal 15 is transparent is provided between the display terminal 15 embedded in the door 10 and the glass plate 10a-1, and the display terminal 15 is Even when turned off and looks blackish, the glass sheet 10a-1 other than the display terminal 15 also looks blackish by the smoke sheet 29 and the boundary with the display terminal 15 does not stand out, and the appearance can be maintained well. Further, in FIG. 6B, a transparent liquid crystal display unit is used as the display unit of the display terminal 15 and an entirely black smoke sheet 29a is laid along the back surface of the transparent liquid crystal display unit. Can be made good. Further, in FIG. 6C, the glass plate 10a-1 in FIGS. 6A and 6B is a magic mirror 10a-2, and the smoke sheet 29 is not required. be able to. In any case, an example in which the display terminal 15 is integrally attached to the door 10 is shown. Instead of the smoke sheet 29, the glass plate 10a-1 may be painted so that it looks blackish.

FIGS. 7, 8 (a) and 8 (b) show an example of mounting of a camera for interior imaging and interior lighting, and FIG. 7 is a longitudinal sectional view of the refrigerator with the door closed, FIG. a) is a cross-sectional view showing the inner surface of the door in the refrigerator with the door closed, and FIG. 8B is a front view of the inner surface of the door when the door is opened.

  7, 8 (a), and 8 (b), the storage chamber 5 of the refrigerator is provided with shelf plates 30-1 to 30-4 in a plurality of upper and lower stages. The shelf boards 30-1 to 30-4 are made of a transparent material such as a transparent synthetic resin material or glass except for the surrounding frame.

  Reference numeral 31 denotes a storage container provided so as to be freely drawn out below the lowermost shelf plate 30-4. Although the lowermost shelf plate 30-4 is fixed because it constitutes the upper surface of the storage container 31, the shelf plates 30-1 to 30-3 are respectively protruding side walls provided on both side walls of the storage chamber 5. It is placed on the parts 32-1 to 32-4 and is detachable. In particular, among the shelf plates 30-1 to 30-3 excluding the bottom shelf plate 30-4, the central shelf plate 30-2 is provided with a plurality of corresponding side wall protrusions 32-2 at different heights. The shelf height can be adjusted.

  33-1 to 33-3 are door pockets provided on the inner surface of the door 10 for opening and closing the opening of the storage chamber 5, and are provided in a plurality of stages. The door pockets 33-1 to 33-3 are also formed of a material having permeability.

  34-1 and 34-2 are interior lighting lamps provided on the front opening both side walls and the front opening upper wall surface of the storage chamber 5 in front of the front ends of the shelf plates 30-1 to 30-4. . In addition, the interior lighting lamps 34-1 and 34-2 are provided with a substrate having a plurality of LEDs arranged in the vertical direction, for example, for the purpose of energy saving and covered with a translucent cover.

  Reference numerals 35-1 and 35-2 denote cameras as imaging means provided so as to be located on the front side of the storage chamber 5 in the same manner as the interior lighting lamps 34-1 and 34-2, for example, a CCD camera or a CMOS camera. The type is not particularly limited, but is preferably resistant to dew condensation.

  In this embodiment, there are two cameras and temperature state confirmation means (hereinafter referred to as a thermo camera), one of which is attached to the lower surface of the upper door pocket 33-1 in the front-rear direction substantially in the center. Yes, the other 35-2 is attached near the upper part of the lowermost door pocket 33-3. Here, the front-rear direction of the door pocket 33 is a direction perpendicular to the surface formed by the inner plate 10 b of the door 10, and is the same direction as the front-rear direction of the storage chamber 5 when the door 10 is closed. is there.

  Further, the upper camera 35-1 is provided at a position substantially in front of the uppermost shelf 30-1, and the lower camera 35-2 is provided at a position in front of the third shelf 30-3 in the drawing. In other words, each camera is not a shelf 30-2 placed on a plurality of side wall projections 32-2 provided on the premise of height adjustment, but a shelf that is detachable but is not premised on height adjustment. It is provided at a substantially forward position of the plate 30-1 and the shelf plate 30-3.

As is clear from FIG. 7, both the cameras 35-1 and 35-2 are positioned forward from the front ends of the shelf boards 30-1 to 30-4, and as shown in FIG. 8B. Are provided in the vicinity of the open end side end Y opposite to the shaft support X of the door 10, that is, in the substantially central portion in the left-right direction of the refrigerator 51. An almost entire area can be imaged. In the case of this French door, the cameras 35-1 and 35-2 are provided on the side of the door 10 having a large area, and are set so as to be positioned at a substantially central portion of the refrigerator 51. Further, in this example, the cameras 35-1 and 35-2 are provided on the door 10 side where the center pillar 36 (see FIG. 3) is not provided, and are set so as to be positioned at a substantially central portion of the refrigerator 51. . As is well known, the center pillar 36 is provided on one side surface of the two doors 10 in the case of a French door, and the cool air in the storage chamber 5 is generated from a gap generated between the two doors 10. This is a member provided to prevent leakage.

  For the cameras 35-1, 35-2, 37-1, 37-2, as shown in FIG. 9A, a camera 35-3 dedicated to the inside of the storage container is additionally installed, or FIG. The lower camera 35-2 can be moved up and down as shown in the figure, or the lower camera 35-2 can be turned up and down as shown in FIG. It may be set arbitrarily according to the above.

  Further, as is apparent from FIG. 7, the interior lamps 34-1 and 34-2 are provided at positions on the far side of the storage room from the cameras 35-1 and 35-2, and the interior lamps 34-1 and 34 are provided. -2 light is provided so that it does not directly enter the imaging sections of the cameras 35-1 and 35-2.

  More specifically, as shown to Fig.8 (a), the imaging part of camera 35-1,35-2 is the irradiation amount (maximum irradiation amount) of the irradiation direction of the interior lamps 34-1 and 34-2. ) In the range in which the irradiation amount is 1/2 or more, in this example, the irradiation angle is outside the range of 60 degrees.

  Although not shown, this is provided with a light shielding portion on the camera side of the interior lighting lamps 34-1 and 34-2, or a light source such as an LED constituting the interior lighting lamps 34-1 and 34-2. May be configured such that the light from the interior lighting lamps 34-1 and 34-2 does not directly enter the imaging units of the cameras 35-1 and 35-2. It may be selected appropriately according to the installation conditions of the camera. The same applies to a single door type refrigerator described below.

1-1-2. Configuration of Single Door Type Refrigerator FIGS. 10 to 13 show examples of attachment of a camera and an interior lighting when the door 10 is a single door. In this single-type door refrigerator, the same parts as those of the already described French door-type refrigerator are denoted by the same reference numerals, description thereof is omitted, and only different parts will be described.

  First, an example will be described with reference to FIGS. 10, 11A and 11B.

  10 and 11 (a) and 11 (b), in the case of this single door refrigerator, the upper camera 35-1 has a left and right divided door pocket 33-, which is a typical arrangement example in the single door. 4, 33-5, and 33-6 are provided on the raised wall 10 b-2, and the lower camera 35-2 is provided on the inner surface of the door between the door pockets 33-7 and 33-8. . Each of the cameras 35-1, 35-2, 37-1, and 37-2 is the same as the French door type, but can be attached / detached, but the shelf 30-1 and the shelf 30-3 are not premised on height adjustment. It is provided at a substantially forward position. And as shown in FIG. 10, it is a front position from the front end of each shelf board 30-1-30-4, and the center part vicinity of the door 10 so that it can understand in FIG. 11 (a) (b), ie, It is provided so as to be positioned at a substantially central portion in the left-right direction of the refrigerator 51, and the two cameras 35-1 and 35-2 can capture almost the entire area inside the refrigerator.

  The interior lighting lamps 34-1 and 34-2 are provided in the same manner as the French door type. That is, the interior lamps 34-1 and 34-2 are provided in front of the front ends of the respective shelf plates 30-1 to 30-4, and are provided on the front opening side walls and the front opening upper wall surface of the storage chamber. The position where the light does not directly enter the imaging units of the cameras 35-1 and 35-2, that is, the imaging units of the cameras 35-1 and 35-2 are connected to the interior lamps 34-1 and 34-2. It is provided outside the range where the irradiation amount is 1/2 or more of the irradiation amount in the irradiation direction (maximum irradiation amount), in this example, the irradiation angle is 60 degrees.

  12 (a), 12 (b), and 13 show other examples of the single door type.

  In the case of this example, the cameras 35-1 and 35-2 are provided in the vicinity of the door open end Y on the opposite side of the shaft support X of the door 10 as shown in FIGS. The height positions of the cameras 35-1, 35-2, 37-1, and 37-2 are the same positions as in the case of the single door. When the cameras 35-1 and 35-2 are provided in the vicinity of the door opening end Y, the imaging is performed when the door 10 is opened at a predetermined angle, for example, 30 degrees. As a result, as shown by the broken line, the image can be captured in a wide range up to the vicinity of the opening of the storage chamber 5, and the image is confirmed by the user slightly opening the door from time to time. When viewing the image, it is substantially the same as the viewing state, and when viewing the image on the display terminal 15, the image can be viewed in a natural manner without any sense of incongruity.

1-1-3. Camera Configuration FIG. 14 shows an example of a camera used for imaging in a warehouse and temperature measurement. As described in the above examples, it is most preferable that the cameras are provided at positions that are separated from each other in the vertical direction. However, even if the upper and lower cameras 35-1 and 35-2 are installed close to each other as shown in FIG. As a result of the verification, it was confirmed that the entire area can be imaged.

  Therefore, in such a case, it is conceivable that the cameras 35-1 and 35-2 are integrated to form the camera module 41, and the modularization in this way enables easy installation and cost reduction. The module can also be formed by using a wider-angle lens such as a fisheye lens.

1-2. Next, the whole refrigerator system using the refrigerator as described above will be described with reference to FIGS.

  FIG. 15 shows an overall configuration of a refrigerator system 50 using the above-described refrigerator. As shown in FIG. 15, the refrigerator system 50 is a system for confirming the state of food in the refrigerator 51 using the display terminal 15 or the like.

  The refrigerator system 50 includes a refrigerator 51 and a display terminal 15, and further includes a wireless adapter (communication device) 53, a gateway device (relay device) 54, a router device 55, the Internet 56, and a server device 57.

  The refrigerator 51 is arrange | positioned in a user's residence A, for example, is arrange | positioned in the kitchen room in the residence A.

  The wireless adapter (communication device) 53 is configured to be electrically connected to a control unit (described later) of the refrigerator 51 and communicate with the router device 55 via a gateway device (relay device) 54. The wireless adapter 53 receives the signal transmitted from the router device 55 and outputs the received signal to the control unit of the refrigerator 51. Based on this signal, the refrigerator 51 is configured to perform a corresponding operation.

  The wireless adapter 53 also acquires identification information (for example, a serial number and model number) of the refrigerator 51, image data in a warehouse to be described later, and the like from the storage unit of the refrigerator 51, and a router via the gateway device (relay device) 54. It is configured to transmit to the device 55. The wireless adapter 53 may be provided integrally with the refrigerator 51 or may be configured to be detachable.

The wireless adapter 53 includes a “connect” button 53a. The “connect” button 53a is used to newly connect to the gateway device (relay device) 54. When the “connect” button 53 a is operated by the user, the wireless adapter 53 is configured to acquire identification information (for example, a manufacturing number or a model number) of the refrigerator 51 from the storage unit of the refrigerator 51.

  As described above, the display terminal 15 is a general-purpose portable terminal such as a smartphone or a tablet PC (personal computer). In the present embodiment, the display terminal 15 is detachable from the door 10 of the refrigerator 51 described above. And a function (means) for connecting to the Internet 56 and a function (means) for communicating with a gateway device 54 described later.

  The display terminal 15 (for example, a smartphone) is configured to connect to the Internet 56 via a telephone line network (for example, a 3G line network). In addition or alternatively, for example, the display terminal 15 is configured to connect to the Internet 56 via a public wireless LAN with a Wi-Fi communication function.

  By connecting to the Internet 56, for example, the display terminal 15 obtains image data of food in the refrigerator that is the subject of the present invention and data for energy saving operation (for example, data such as the number of times the door is opened and closed). Can be obtained from the website of the manufacturer of the refrigerator 51 via the Internet 56. By installing the acquired program on the display terminal 15 and starting the installed program, the display terminal 15 can acquire image data of food in the refrigerator. That is, the display terminal 15 can generate and transmit an operation signal for acquiring the image data of the refrigerator 51 by operating the operation screen of the display 15a. Details will be described later.

  The display terminal 15 is also connected to the router device 55 by general-purpose communication not via the Internet 56 such as Wi-Fi communication, Bluetooth (registered trademark) communication, or infrared communication, and the gateway device 54 via the router device 55. Is configured to connect to the communication. A device for that purpose (for example, a Wi-Fi antenna or the like) is incorporated in the display terminal 15.

  The gateway device (relay device) 54 is a device that relays communication between the wireless adapter 53 and the display terminal 15. For example, the gateway device (relay device) 54 and a specific low power wireless special small frequency band (924.0 to 928.0MHz). ) Is used for communication. The communication frequency band between the gateway device 54 and the wireless adapter 53 is preferably a low frequency band that reaches a long distance, and is installed in the residence A of the user where the refrigerator 51 is installed together with the router device 55.

  The gateway device 54 is also configured to be in communication connection with the display terminal 15 via the router device 55. That is, the gateway device 54 is configured so that communication (first communication) can be executed only through the router device 55 without going through the display terminal 15 and the Internet 56.

  The gateway device 54 is further configured to be connectable to the Internet 56 via the router device 55. Accordingly, the gateway device 54 can communicate with the display terminal 15 (second communication) via the Internet 56 and the router device 55 and can also communicate with the server device 57.

Therefore, the gateway device 54 communicates with the display terminal 15 via the Internet 56 (ie, the first communication via the router device 55 only) and via the Internet 56 (ie, via the Internet 56 and the router device 55). Second communication). The reason for this configuration will be described later.

  The server device 57 is prepared by a manufacturer of the refrigerator 51, for example. The server device 57 manages information necessary for access and authentication regarding the display terminal 15, the refrigerator 51, and the wireless adapter 53. For example, the server device 57 determines whether or not the identification information of the display terminal 15 included in the communication request signal matches the identification information of the display terminal 15 stored (registered) in association with the wireless adapter 53. Authenticate (distinguish). And when these correspond, the display terminal 15 to access is authenticated as a regular display terminal, and communication with the refrigerator 51 in the residence A is permitted.

  The server device 57 stores the image data of the food in the refrigerator transmitted from the refrigerator 51, data for energy saving operation, and the like. These data are transmitted to the display terminal 15 based on a data call signal from the display terminal 15. That is, the user can confirm the food image data in the refrigerator by operating the display terminal 15 on the display.

1-3. FIG. 16 shows a system configuration of a refrigerator used in the refrigeration system.

  In FIG. 16, the refrigerator 51 includes an interface (I / F) 58, a control calculation unit 59, a storage unit 60, a door opening / closing detection unit 61, interior lighting lamps 34-1 and 34-2, a camera 35-1, 35-2 and 37-1, 37-2 A fan drive circuit 62, a compressor drive circuit 63, a blower fan 18, and a compressor 19 are provided, and are supplied with power from an AC power source 64 such as a commercial power source in the house A, for example.

  The interface (I / F) 58 exchanges data and the like between the wireless adapter 53 and the control calculation unit 59 of the refrigerator 51.

  The control calculation unit 59 controls the operation of the refrigerator. In addition, the control calculation unit 59 controls the cameras 35-1 and 35-2 and the interior lamps 34-1 and 34-2 to image food in the refrigerator, and the image data is temporarily stored in the storage unit 60. The information is stored and transmitted to the server device 57 of the Internet 56. Further, if there is a request signal for advance transmission of image data or the like from the display terminal 15 via the server device 57, the display terminal 15 is instructed to send it to the display terminal 15 via the server device 57 immediately after imaging food. Details thereof will be described later. Furthermore, the control calculation unit 59 calculates whether there is no change compared with the image stored in the storage unit 60, which is described later, the food image specified by the object specifying unit. Based on the result, the storage period stored in the storage unit 60 is changed and corrected.

  In addition to the identification symbol and the control program, the storage unit 60 rewrites and stores the image data of the food items captured by the cameras 35-1 and 35-2 and the data for energy saving operation for a certain period together with the time information. Further, if there is a food specified by the above-described object specifying means food, the storage period of the food is stored.

  The interior lighting lamps 34-1 and 34-2 are controlled to be turned on / off by the control calculation unit 59 based on a signal from the door opening / closing detection unit 61, and are turned on when the door 10 is opened, and the door 10 is closed. Turns off after a delay of several seconds.

  The cameras 35-1 and 35-2 are controlled by the control calculation unit 59 based on a signal from the door opening / closing detection unit 61, and images food in the refrigerator when the door 10 is closed. Details thereof will be described later.

  The fan drive circuit 62 receives the control signal from the control calculation unit 59 and controls the rotation of the blower fan 18.

  The compressor drive circuit 63 receives a control signal from the control calculation unit 59 and controls driving of the compressor 19 such as the rotational speed.

  The blower fan 18 operates according to the control of the fan drive circuit 62 and generates an air flow for circulating the cool air.

  The compressor 19 compresses a refrigerant (not shown) circulating in the refrigerator 51 according to the control of the compressor drive circuit 63.

1-4. Configuration of Server Device Next, a configuration example of the server device of the refrigerator system in Embodiment 1 will be described using FIG. Here, the configuration of the server device 57 in FIG. 15 is taken as an example.

  As shown in FIG. 17, the server device 57 includes a server interface 65, a server control unit 66, and a data storage unit 67.

  The server interface 65 is a communication unit that exchanges data and the like between the display terminal 15 and the router device 55 and the server device 57 via the Internet 56 under the control of the server control unit 66.

  The server control unit 66 controls the overall operation of the server device 57. The server control unit 66 also combines the image data of the food transmitted from the refrigerator 51 and stores the image data together with the time information in the data storage unit 67. When there is a signal for requesting prior transmission of image data from the display terminal 15, The latest image data of the image data stored in the data storage unit 67 is transmitted to the display terminal 15 each time the image data is transmitted from the refrigerator 51, and rewritten by storing the display terminal 15 together with time information for a certain period. Update. Of course, when a data call signal is transmitted from the display terminal 15 in the absence of a signal for requesting prior transmission of image data, the latest image data stored in the data storage unit 67 is transmitted to the display terminal 15.

  The data storage unit 67 stores necessary data such as a management program and an application program for executing the refrigerator system. Further, the data storage unit 67 rewrites and stores the latest data of the food image data transmitted from the refrigerator 51 by adding time information in accordance with the control of the server control unit 66. The data storage unit 67 is configured by, for example, an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

  The server device 57 can be constructed by either a server composed of a single large computer or a cloud server composed of a large number of computers.

<2. Operation>
Next, operations of the refrigerator and the refrigeration system having the above-described configuration will be described. In the following descriptions, the same operations are described with the same step numbers added.

2-1. Imaging operation of food 2-1-1. Imaging When Door is Closed FIG. 18 is a flowchart showing an operation when imaging food in the storage room when the door 10 of the refrigerator 51 is closed.

  When the door 10 is opened to put in and out the food (Y in Step 1), the interior lamps 34-1 and 34-2 are turned on (Step 2), and when the door 10 is closed (Y in Step 3), The illuminance of the interior lighting lamps 34-1 and 34-2 is reduced (step 4), and after a few seconds, for example, the cameras 35-1 and 35-2 are activated to image food in the interior (step 5). . Thereafter, the captured image data is stored together with time information in the storage unit 60 of the refrigerator 51 (step 6), and the interior lighting lamps 34-1 and 34-2 are turned off (step 7). In addition, the time information which shows an imaging date is acquired via communication from refrigerator itself or a server.

  Here, if the illuminance of the interior lighting lamps 34-1 and 34-2 is reduced during the imaging, food can be clearly captured, which is preferable. That is, when food is imaged by turning on the interior lighting lamps 34-1 and 34-2, the inner surface of the storage room may be white, and the reflected light is applied to the food from the entire peripheral surface except the opening part of the storage room. Reflected strongly (reflects on food with high illuminance without decreasing illuminance), and the food becomes whitish and clear imaging becomes difficult. However, when the illuminance of the interior lighting lamps 34-1 and 34-2 is reduced as in the above-described example, the influence of reflected light from the interior surface of the storage room can be greatly reduced, and a clear image can be captured immediately. It becomes possible.

2-1-2. Image Pickup During Refrigerating Room Cooling Operation FIG. 19 is a flowchart showing an operation when food in the storage room is picked up when the refrigerator is cooled after taking in and out the food in the refrigerator.

  When the door 10 is opened to put food in and out (Y in Step 1), the interior lighting 34-1 and 34-2 are turned on (Step 2), the door 10 is closed (Y in Step 3), and the refrigerator compartment It is determined whether or not it is in the cooling operation state (step 8). If the blower fan 18 is ON and the damper 22 is open and the refrigerator is in the cooling operation, the illuminance of the interior lamps 34-1 and 34-2 is reduced as in the case of closing the door (step 4). ), The cameras 35-1 and 35-2 are activated to image the food in the cabinet (step 5). Thereafter, the captured image data is stored together with time information in the storage unit 60 of the refrigerator 51 (step 6), and the interior lighting lamps 34-1 and 34-2 are turned off (step 7).

  In this way, by taking an image during the cooling room cooling operation, for example, even if condensation occurs on the surface of food newly put in the storage room due to a temperature difference between the storage room and the outside, this condensation is eliminated by the cooling operation. It ’s gone. Therefore, the imaging by the cameras 35-1 and 35-2 is preferably performed so clearly that the characters written on the food packaging container can be read.

  During the above operation, when the door 10 is closed (Y in Step 1), the interior illumination lamps 34-1 and 34-2 are once turned off, and the interior illumination lamps 34-1 and 34 are again turned on before imaging. -2 is turned on to reduce the illuminance thereof, it is possible to suppress wasteful power consumption due to continuous lighting of the interior lighting lamps 34-1 and 34-2.

2-1-3. Imaging When Door is Opened FIG. 20 is a flowchart showing an operation when a single door is used, and food in the storage room is imaged when the door 10 is opened at a predetermined angle.

When the door 10 is opened to put food in and out (Y in Step 1), the interior lamps 34-1 and 34-2 are turned on (Step 2), and the opening angle of the door is a predetermined angle, for example, as shown in FIG. When the opening angle shown is 30 degrees (Y in Step 10), the illuminance of the interior lighting lamps 34-1 and 34-2 is decreased (Step 4), the cameras 35-1 and 35-2 are activated, and the warehouse The food inside is imaged (step 5). Thereafter, the captured image data is stored in the storage unit 60 of the refrigerator 51 together with time information (step 6), and when the door is closed (Y in step 11), the interior lighting lamps 34-1 and 34-2 are turned off ( Step 7).

  If the door 10 is picked up when the door 10 is opened at a predetermined angle as in this example, it is possible to pick up an image of the storage room over a wide range as described above, and the image becomes natural with no sense of incongruity. is there.

  Note that when imaging is performed when the predetermined angle is opened, imaging is performed at least twice during the door opening operation and the door closing operation. At this time, the latest storage room in which food has been taken in and out is stored in the storage unit 60 by storing the last imaged image data (during the door closing operation) in the operation time from one door opening to closing. The state of food can be imaged.

  Needless to say, in step 10 for detecting whether or not the opening angle of the door 10 reaches a predetermined angle, if the door 10 is closed before reaching the predetermined angle, the interior illumination lamp 34-1 is used. , 34-2 is turned off, and it is waited for the door 10 to be opened again.

  In addition to imaging food in the storage room, user information is specified when the door 10 opens / closes or stores food by imaging when the opening angle of the door 10 reaches a predetermined angle (for example, 90 degrees). It becomes possible.

  Moreover, you may mount the manual imaging | photography mode which can image manually by a user when a door opens. Thus, for example, it is possible to easily manage specific food information that the user wants to specially manage, such as the expiration date or the food production area.

  In the above, a representative example of the imaging operation has been described. For example, in addition to the imaging operation, imaging is performed when a predetermined time elapses after a predetermined time or the previous imaging, or when an imaging instruction is given from the display terminal 15 An imaging operation such as, for example, is also conceivable.

2-1-4. Illumination at the time of imaging Food illumination at the time of imaging reduces the illuminance after lighting the interior lighting lamps 34-1 and 34-2, as already described with reference to FIGS. Illuminate and image. Depending on the case, it is also conceivable to illuminate and image by lowering the illuminance as lighting only the interior lighting 34-1 on both sides of the storage room or the interior lighting 34-2 on the upper surface of the storage room. The reason for imaging with the illuminance lowered is as already described.

  As another example of food irradiation at the time of imaging, the following may be considered. That is, FIG. 21 is a flowchart showing the on / off state of the interior lighting lamps 34-1 and 34-2 at the time of imaging by the cameras 35-1 and 35-2. This corresponds to the operation of the broken line portion.

  The imaging in FIGS. 18, 19, and 20 is an example in which the interior illumination lights 34-1 and 34-2 are turned on and the illuminance is lowered, and in this example, the storage room is stored. Turn off one of the interior lamps 34-1 on both sides, in other words, turn on only one of the interior lamps 34-1 on both sides of the storage room to illuminate and image food. is there. In this case, the interior illumination lamp 34-2 on the upper surface of the storage room may be either turned on or off, but any one may be appropriately selected so that the illuminance in the interior is optimal for imaging.

In FIG. 21, before imaging with the cameras 35-1 and 35-2, one of the interior lighting lamps 34-1 on both sides of the storage room that has already been lit, for example, the right interior lighting lamp is turned off. (Step 12), imaging food in a state of being illuminated by the left interior lamp (Step 13)
(Step 14). Thereafter, the right interior lighting is turned on (step 15), and then the left interior lighting is turned off (step 16), and the food being irradiated with the right interior lighting is imaged (step 17). ). The image data is stored (step 18), and the right interior illumination lamp is turned off (step 19). The image data stored by irradiating food from one side in this way is combined into one image and stored together with time information.

  As described above, the interior lighting lamps 34-1 and 34-2 are turned on one by one to image foods and combine them into one as follows. In other words, as described above, the illuminance of the interior lighting lamps 34-1 and 34-2 is reduced during imaging, but if the illuminance reduction is too large, the image becomes dark and lacks in clarity. Therefore, there is a limit to the decrease in illuminance. For this reason, a certain level of illuminance is maintained. However, even when the illuminance is high, there are cases in which the influence of reflected light remains when the volume of the storage room is small. However, as described in this example, when the left and right sides of the interior illumination lamp 34-1 are turned on to image food, the influence of reflected light can be reduced and the interior illumination lamp illumination 34-1 is illuminated. The surface on the side of the image can be imaged with moderate brightness. Therefore, it is preferable to perform a combining process that combines the left and right images of food captured with moderate brightness, so that the whole image becomes clear.

2-2. Next, storage and composition of image data captured by the above-described operations will be described.

2-2-1. Storage and Compositing by Server FIG. 22 shows a case where image data is stored and composited by the server device 57. When the interior is imaged by the operations shown in FIG. 18 to FIG. 53 is connected to the Internet 56 via the gateway device 54 and the router device 55 (step 20), and the individual image data captured and stored by each camera is transmitted to the server device 57 connected to the Internet 56 (step 21). . The image data transmitted via the Internet 56 is subjected to a combining process (step 22) for combining individual image data by the server control unit 66 of the server device 57. The combined image data is stored by adding the current time information acquired from the server to the data storage unit 67 of the server device 57 as the imaging time (step 23).

  If the image is synthesized and stored in the server device 57 as in this example, the load on the control calculation unit 59 and the storage unit 60 on the refrigerator side can be reduced, and an internet-compatible refrigerator is inexpensive. Can be provided.

2-2-2. Storage and Compositing on the Refrigerator Side FIG. 23 shows a case where image data is stored and combined on the refrigerator side. When the interior is imaged by the operations shown in FIGS. The unit 59 takes out individual image data from each camera stored in the storage unit 60, combines them, and combines them (step 24). The synthesized image data is stored (step 25), the refrigerator 51 is connected to the internet 56 from the wireless adapter 53 via the gateway device 54 and the router device 55 (step 20), and the stored image data is connected to the internet 56. The data is transmitted to the server device 57 (step 21). The composite image data transmitted via the Internet 56 is stored in the data storage unit 67 of the server device 57 with the current time information acquired from the server added as the imaging time (step 23).

  In this example, although the load on the control calculation unit 59 and the storage unit 60 on the refrigerator side increases, image data can be transmitted and received with the display terminal 15 without going through the data storage unit 67 of the server device 57, and the system operating cost is reduced. It can be made cheap.

2-2-3. Storage and Compositing on Display Terminal Side FIG. 24 shows a case where image data is stored and combined on the display terminal 15 side. When the interior is imaged by the operations shown in FIG. 18 to FIG. Connects to the Internet 56 from the wireless adapter 53 via the gateway device 54 and the router device 55 (step 20), and transmits image data to the display terminal 15 instead of the server (step 26). The image data transmitted via the Internet 56 is subjected to a combining process (step 27) for combining the image data by the display terminal 15. The synthesized image data is stored in the data storage unit of the display terminal 15 together with the time information of the display terminal itself (step 28).

  Also in this example, although the load on the display terminal 15 increases, the load on the control calculation unit 59 and the storage unit 60 on the refrigerator side can be reduced, and an Internet-compatible refrigerator can be provided at low cost.

  The image data transmission and the composition operation have been described above by exemplifying typical ones. However, the storage and the composition portion are not necessarily performed at the same location, for example, the server device 57. For example, the refrigerator 51 The image data may be combined with each other and the server device 57 may store the image data.

2-3. Normal transmission / reception of image data 2-3-1. Transmission / Reception of Image Data Stored in Server and Refrigerator FIG. 25 is a flowchart showing transmission / reception between server device 57 or refrigerator 51 and display terminal 15, and shows transmission / reception of image data stored in server device 57 or refrigerator 51. is there.

First, the user presses a server connection button (not shown) displayed on the display terminal 15 at a shopping destination or the like, for example, and makes a request to read image data stored in the data storage unit 67 of the server device 57 (see FIG. Step 31). It is detected whether image data has been taken in from the server device 57 (step 32). If no image data is transferred, the display terminal 15 connects to the Internet 56 (step 33). The image data stored in the data storage unit 67 of the server device 57 or the storage unit 60 of the refrigerator is taken out (step 34), and the image data is transferred to the display terminal 15 such as a smartphone that is a data caller via the Internet 56. Is transferred (step 35). It is detected whether image data has been captured from the server device 57 (step 32), and when the image data is captured, the image data is output to the display of the display terminal 15 (step 36).
Thereafter, it is detected whether an enlarge button (not shown) displayed on the display of the display terminal 15 has been pressed (step 37), and if it is detected that the enlarge button has been pressed, the inside of the storage room output on the display is detected. Image data obtained by enlarging image data 1.1 times or more by digital conversion is created. The enlarged and converted image data is output to the display of the display terminal 15 (step 36).

  After that, it is detected whether a local display numeric keypad (not shown) displayed on the display has been pressed (step 38), and when it is detected that the numeric keypad has been pressed, the contents of the storage room output on the display are detected. The enlarged image data is converted into image data at a location designated by the numeric keypad and output to the display (step 36). Further, it is detected whether the numeric keypad is pressed (step 38). Exit if not detected.

  It should be noted that the enlargement button and the numeric keypad for displaying image data at a predetermined location may be of a type in which the surface of the display is slid with a finger. Such a configuration improves usability.

  As described above, according to this refrigerator system, the contents of the refrigerator 51 can be confirmed at a glance at a shopping destination, so that forgetting to buy can be prevented and the waste of purchasing extra ingredients can be eliminated. . In addition, in a family working together, it is possible to check the inside of the storage room 5 of the refrigerator 51 just before returning, and to purchase foods and the like on the way home, thereby reducing the time for the user.

  In addition, by displaying an enlarged display or a predetermined location, it is possible to reliably display foods that are difficult to confirm on the small display terminal 15 by displaying an enlarged image on the display terminal 15 that has been miniaturized like the display terminal 15. Can see.

2-3-2. Transmission / Reception of Image Data Saved on Display Terminal FIG. 26 shows a case where image data is already stored and saved on the display terminal 15. In this case, as described with reference to FIG. 24, the image data is transmitted and stored in the display terminal 15, and when this image data is requested to be read (step 31), the image data is fetched from the storage unit of the display terminal 15. The image data is output to the display of the display terminal 15 (step 36). The subsequent enlargement display operation and local display are as described with reference to FIG.

2-3-3. Image data switching reception The images captured by the cameras 35-1 and 35-2 are the images captured by the respective cameras 35-1 and 35-2 (the upper half or the lower half of the storage room) and the respective captured images. An example is shown in which composite images combined and combined (images combined from the top to the bottom of the storage room into one image) are stored and stored, and these images can be switched and transmitted and displayed on the display terminal 15.

  FIG. 27 is a flowchart showing the operation in such a case.

  First, the user presses an individual image / composite image call button (not shown) displayed on the display of the display terminal 15 (step 41), and calls either an individual image or a composite image (step 42). ). Thereafter, either the individual image data or the composite image is captured by the flow shown in FIG. 25 or 26 and displayed on the display of the display terminal 15.

  When the image displayed on the display terminal 15 can be switched to either an individual image or a composite image as in this example, the state of food in the storage room is usually confirmed at a glance by calling up and displaying the composite image. However, when it is difficult to discriminate food due to the corners of the storage room, etc., it can be surely discriminated by switching to individual images and displayed, improving usability.

2-4. Transmission / Reception at the Time of Prior Transmission Request for Image Data, etc. FIG. 28 shows a flowchart when there is a request for image prior transmission from the display terminal 15.

First, the user operates the display terminal 15 to perform a request operation for a prior request signal (step 51). The server device 57 or the refrigerator 51 receives this request signal (step 52), and reads and transmits image data (individual image data or composite image data, the same applies hereinafter) stored in the data storage unit 67 or the storage unit 60 (step 52). 53). The display terminal 15 detects whether the image data from the server device 57 or the refrigerator 51 has been taken in (step 54). If no image data is transmitted, the display terminal 15 connects to the Internet 56 (step 55). The server devices 57 take out the image data stored in the refrigerator 51 (step 56), and transmit the image data to the display terminal 15 via the Internet 56 (step 57). It is detected whether the image data has been taken in from the server device 57 (step 54). When the image data is detected, the process waits until the image data is called (N in step 58). If there is an image data call (Y in step 58), the image data is output to the display of the display terminal 15 (step 59).

  Thus, when image data is requested on the display terminal 15, even when access to the server device 57 is concentrated and transmission of image data from the server device 57 is delayed, the image data is requested in real time simultaneously with the request for image data. Can be displayed on the display to check the image. Therefore, it is possible to avoid a situation of waiting for image data to be displayed at the shopping destination, and to shorten the shopping time.

  Further, if there is an image pre-transmission request from the display terminal 15, the server device 57 is set to transmit image data transmitted each time the door 10 of the refrigerator 51 is closed to the display terminal 15, and the display terminal If the user having 15 is set to notify when image data is transmitted, the user can know this whenever the door is opened and closed. Therefore, for example, when the number of times of door opening / closing increases and the power consumption is likely to increase, the user can talk to the child to refrain from using the refrigerator 51 by calling his home. Convenience is also improved.

2-5. Switching of transmission / reception path of image data etc. FIG. 29 shows a display on the display terminal 15 when a communication path is selected. The flow of transmission / reception of image data is as described above, but the communication route is changed to speed up the acquisition of image data.

  Specifically, the user can select whether to perform the display terminal 15 directly with the gateway device 54 via the router device 55 or with the server device 57 via the Internet 56.

  29, reference numerals 70 and 71 denote communication path selection buttons displayed on the display of the display terminal 15. Reference numeral 70 denotes a “direct” selection for directly connecting the display terminal 15 to the refrigerator 51 via the router device 55 and the gateway device 54. A button 71 is a “server” selection button for connecting to the server device 57 or the refrigerator 51 via the Internet 56. 72 and 73 are data selection buttons for data to be called, 72 is a button for food image data, and 73 is a button for energy saving related data such as the number of times the door is opened and closed. Reference numeral 74 denotes a transmission button.

  When the “direct” selection button 70 is selected, the display terminal 15 is set to “direct” communication. When the user touches the send button 74 while “direct” communication is set, the image data request signal of the display terminal 15 is directly transmitted from the display terminal 15 to the gateway device (relay device) 54 via the router device 55. Sent. The gateway device 54 transmits the received data request signal to the wireless adapter 53 of the refrigerator 51. When the wireless adapter 53 receives the image data request signal, it transmits this to the control calculation unit 59 of the refrigerator 51. When receiving the image data request signal, the control calculation unit 59 of the refrigerator 51 reads the image data stored in the storage unit 60 and transmits the image data to the display terminal 15 through the wireless adapter 53 and the router device 55.

  Accordingly, when the user is located in the residence A as shown by the broken line in FIG. 15, in a broad sense, when the display terminal 15 is present at a position where it can communicate directly with the router device 55, the display terminal 15 is connected to the router device 55. The image data can be directly acquired from the storage unit 60 of the refrigerator 51 by communicating with the gateway device 54 via the network.

As a result, when a delay occurs in the transmission response of the image data in response to the user's operation to the display terminal 15 due to congestion of the line due to concentration of access to the Internet 56, the user sets the display terminal 15 to “ By switching to “direct” communication, it is possible to acquire image data without being congested with the Internet 56 line. Therefore, the user can check the food status in the storage room 5 by displaying an image on the display of the display terminal 15 in real time without waiting.

  On the other hand, as shown by the solid line in FIG. 15, when the user is located outside the residence A, in a broad sense, when the display terminal 15 exists in a position where the user cannot directly communicate with the router device 55, the user selects the “server” selection button. By touching 71, the display terminal 15 communicates with the server device 57 via the Internet 56 and reads the image data of the refrigerator 51. That is, when the user is outside the residence A and touches the “server” selection button 71 to set “server” communication, the user can obtain the image data of the refrigerator by the display terminal 15 even in a remote place, You can know the storage status of.

  As described above, the refrigerator system in this embodiment allows the user to select either “direct” communication or “server” communication. You can select the communication and view the image in a comfortable state without being affected by the congestion of the Internet line.

2-6. Reproduction of image data on display terminal 2-6-1. Reproduction of normal internal image FIG. 30 shows a flowchart of normal internal image display. The user arbitrarily sets the display period or date / time of the internal image to be displayed from the display terminal 15 and presses the image data selection button 72. For example, a prior transmission request is made from the display terminal 15 and an in-store image is acquired within a specified period. The flow of acquiring image data is as shown in FIG. The internal image is stored in the display terminal 15 (step 60), and the display terminal 15 checks the presence / absence of image data within a specified period from the stored images (step 61), and the image data is time-series. Are output to the display unit of the display terminal 15 (step 62). When the user presses the playback button, the image data read in time series on the display unit of the display terminal 15 is automatically switched (step 64), and is continuously displayed by displaying. As for the timing of image switching, image switching is performed after the display time has elapsed or when an image feed operation is performed by the user (step 63). The image feed operation by the user can be browsed while pressing the feed button or using a seek bar while skipping images by time width. This makes it possible to search for images in the warehouse by specifying the date and time, and to continuously browse images in the time series, greatly reducing the time required for image search, and by using a seek bar, It becomes possible to find the target image effectively.

2-6-2. Creation of in-store image memo FIG. 31 shows a flowchart in the case of creating a memo using an in-store image. The user can input the memo information on the internal image by tracing the internal image output to the display terminal 15 (step 72) (step 73), and can reflect this (step 74). Further, by selecting a tracing range with the finger in the internal image area (step 75), trimming is performed (step 76) and the image is partially transmitted. As a result, the transmission image size can be reduced and the transmission time can be shortened, so that it is easy to use and the creation of memos and the sharing of images in the cabinet can be performed without any trouble.

2-6-3. Storage Change Detection Reproduction FIG. 32 shows a flowchart in the case of detecting the change of the image in the warehouse and continuously reproducing the image in the warehouse. The image data is stored in the display terminal 15 and output to the display section (step 66), and the user traces the displayed image in the cabinet with his / her finger to specify the food material or the area in the cabinet (step 67). By doing this, the specific area between the image displayed last time and the image currently displayed is compared (step 68), and automatic change is stopped when a change is detected. As a result, users can easily know changes in the specific area in the warehouse or the presence or absence of food by performing automatic playback without checking image data one by one, making inventory management easy. Can do. Furthermore, by designating a specific area, the amount of image change detection processing can be reduced, and a constant processing speed can be maintained. Note that it is preferable to compare the specific area between the previous display image and the current display image and detect the change by the control calculation unit 59 of the refrigerator or the server control unit 66 of the server device 57.

2-6-4. Reproduction of postscript information In addition to in-chamber image information, additional information such as user information on door opening / closing or food storage, or specific food information manually imaged by the user, as described in section 2-1-3, is displayed simultaneously. Also good.

  Note that only a part of the user information or specific food information is displayed on the in-store image, and all information can be displayed by performing a tap operation or the like on the terminal.

  The user information to be displayed includes image information, text information (name, etc.) as a result of performing face recognition from face information registered in advance.

<3. Management>
3-1. Food Storage Management Next, food storage management, which is the gist of the present invention, will be described.

  This is performed using the above-described concept of in-chamber change detection and regeneration. FIG. 33 shows a food storage management control unit composed of the refrigerator control calculation unit 59 and the storage unit 60 shown in FIG. FIG. 34 is a block diagram showing the food storage management control.

  In FIG. 33, reference numeral 77 denotes an object specifying means for specifying a food to be stored. In this embodiment, the object food is displayed by touching or selecting an area on the display screen of the display terminal 15 so that it can be specified. It is. The object specifying means 77 extracts the singular points of the object specified by the image operation and specifies the object. Reference numeral 78 denotes an image storage unit configured in the storage unit 60 of the refrigerator, which stores an image of the target food specified by the target object specifying unit 77 among images captured by the camera. 79 is a storage period storage unit similarly configured in the storage unit 60 of the refrigerator, and the elapsed time from the time when the target food identified by the object identification means 77 is first imaged, that is, put into the storage room, That is, the storage period is stored. Reference numeral 80 denotes a comparison unit that compares the target food image in the image captured by the camera with the target food image captured last time and stored in the image storage unit. Reference numeral 81 denotes correction calculation means for changing the retention period stored in the retention period storage unit 79 if there is a change in the target food image as a result of comparison by the comparison means 80.

Next, the food storage management operation configured as described above will be described with reference to the flowchart of FIG.
After the storage room is imaged and stored by the operation shown in FIGS. 18 to 21, when the target food is specified by the object specifying means 77 (step 81), the image storage unit 78 stores the target food image. At the same time, the storage period storage unit 79 stores the elapsed time from the input time (step 82).

  Thereafter, the door is opened and closed (step 83), that is, when food is taken in and out, the storage room is imaged by the operation shown in FIGS. 18 to 21 (step 84), and the comparison means 80 is newly imaged. The new image of the target food is compared with the old image of the target food stored when the door was opened / closed last time (step 85).

  If there is a large change in the image as a result of the comparison between the old and new images, the correction calculation means 81 determines that the target food is the same food and has been replaced and newly replaced or no longer consumed. The retention period stored in the storage unit 79 is reset and changed (step 86), and when it is newly replaced, the elapsed time from that point is newly stored. Then, for example, it transmits to the display terminal 15 via the internet, and alert | reports that the preservation | save period changed (step 87). If there is no change in the image, the object image is retained and the storage period is continued.

  FIG. 35 is a flowchart showing an example of detection of a new and old image change of a target food. When a change is detected in a new and old image (step 85), it is confirmed whether the target object exists (step 88). It is determined that the object is no longer consumed, cancels the object image and the storage period (step 91), and transmits to the display terminal 15 that the object is no longer present (step 87). If the object is present, the old and new image changes to confirm whether the size of the food is decreasing, for example, if the food is decreasing (step 88). The object image is retained while the object is being used for a specific period of time, and the storage period is continued.If the object is not decreased, it is determined that the target food is the same food and that it has been replaced with a new one, and the storage period is reset. Change (step 86) and newly store the elapsed time from that point.

  FIG. 36 is a flowchart showing still another example of detection of new and old image changes of the target food. When there is no change in the new and old image comparison (step 85), the target image is held and the storage period is continued. When it is detected that there is a change in the old and new images, it is further checked whether or not the food absence time until the new and old image comparison of the target food is longer than a predetermined time (step 92). Even if they are the same food, it is determined that they have been replaced, and the retention period is reset and changed (step 86), and when it is newly replaced, the elapsed time from that point is newly stored. This is because the target food is absent, that is, if it is not stored in the storage room for a long time, it is unlikely that it will be taken out by cooking or eating, and it is considered that the user went shopping and newly replaced the same food, In such a case, the retention period is reset.

  Here, since the object specifying unit 77 is operated by the user and specifies the object, the object specific means 77 continues to extract and specify the singular point peculiar to the object from the image captured by the camera. Even if the storage position of the container changes every time it is put in or taken out, it is possible to accurately specify the object, and it is possible to perform highly accurate storage management.

3-2. Extraction of Singularity Next, extraction of singularity of the target food will be described.

  As described in the previous section, the object specifying unit 77 extracts the hue component data from the image data near the center of the food specified by the user's image operation, and stores this as singular point data of the food. Next, the object specifying means 77 acquires change area data representing the position and size of the change area of the old and new images from the old and new image change detection. The target object specifying unit 77 limits the target food to be specified based on the change area data, and extracts the hue component data near the center of the change area data. This hue component data is compared with the singular point data of each food stored in advance, and those that match more than a certain level are recognized as specific foods.

  It is desirable to extract the hue component data at the same position as the singular point data stored in advance as much as possible in the extraction region of the hue component data of the target food to be recognized. For this reason, the size of the area for extracting the hue component data is not constant, and the size of the change area depends on the position of the food by changing the size of the hue extraction area in proportion to the size of the change area data. Even if the brightness changes, it is possible to extract the hue component data at substantially the same position.

  In addition, the singular point data of each food does not need to be one. For example, a plurality of singular point data may be stored for foods having different hues depending on directions and angles.

  Also, in the vicinity of the interior lighting lamps 34-1 and 34-2, the color component due to the lighting characteristics may affect the hue component data of the food, and may be different from the original hue component data of the food. Accordingly, there is a possibility that the food component cannot be specified because it does not match the hue component data of the stored singular point. In order to avoid this, the hue component data of the lighting is stored in advance, and if the change area indicated by the food change area data is close to the lighting, the hue component of the singular point that stored the lighting hue component In addition, erroneous detection can be avoided by correcting the hue component data and determining matching.

  Further, the shelf boards 30-1 to 30-4 are configured such that the transparency is 70% or less at a position of at least 80% or more of the shelf board. Thereby, when the object specifying means 77 acquires change area data from the old and new image change detection, it is possible to suppress erroneous detection of foodstuffs on the back side of each of the shelf boards 30-1 to 30-4. The detection accuracy of the change area data can be improved.

  In this embodiment, the singular point of the target food is extracted based on the hue component data. However, the present invention is not limited to this, and a means for determining the target food can be used by extracting the outline of the food. It is.

  In addition, since the refrigerator and refrigerator system which were illustrated in embodiment of this invention have the following advantages, it describes.

  That is, in the refrigerator of this embodiment, the cameras 35-1 and 35-2 image food in the storage room 5 from the front side of the storage room 5, but at this time, the interior lighting lamps 13-0 and 34-2 Is also provided on the front side of the storage room 5, and the food in the storage room 5 is irradiated from the same front side as the imaging direction by the cameras 35-1 and 35-2. Therefore, the captured image is not as dark or difficult to see as when the interior lighting lamps 34-1 and 34-2 are provided on the back wall of the storage chamber 5, and becomes a clear image. Further, it is not necessary to request the server device 57 to perform image processing that brightens and sharpens the captured image.

  Moreover, since the said cameras 35-1, 35-2 and the interior lighting lamps 34-1, 34-2 are provided so that it may be located ahead from the front end of the said shelf boards 30-1 to 30-4, Images in the storage room 5 captured by the cameras 35-1 and 35-2 are the same as when the user actually opens the door 10 of the refrigerator 51 and sees it. Therefore, when the user displays the image data on the display terminal 15, the food condition in the storage room 5 can be confirmed without any sense of incongruity. Moreover, since the interior lighting lamps 34-1 and 34-2 illuminate the food on the shelf board from the front front of the shelf boards 30-1 to 30-4, they are mounted on the front edge parts of the shelf boards 30-1 to 30-4. The placed food can also be brightly and clearly imaged.

  The cameras 35-1 and 35-2 are provided on the inner surface of the door 10 of the refrigerator 51, and the interior lamps 34-1 and 34-2 are provided on both side walls of the opening edge of the storage chamber 5. First, the cameras 35-1 and 35-2 are positioned at the left and right central portions of the storage chamber 5. Therefore, the image captured by the cameras 35-1 and 35-2 can be made closer to a state where the user actually opens the door 10 of the refrigerator 51 and sees it. In addition, the distance between the cameras 35-1, 35-2 and the food on the shelf can be increased to image a wide range in the storage room 5, and a wider range of food storage status is displayed. This can be confirmed on the terminal 15.

  Further, since the cameras 35-1 and 35-2 are provided so as to be positioned at a substantially central portion in the front-rear direction of the door pockets 33-1 and 33-2 on the inner surface of the door, not only the food in the storage chamber 5 but also the door. A part of the food stored in the pockets 33-1 and 33-2, such as a milk pack, can also be imaged. Since the user has stored it by himself, the user can recognize what it is even if only a part of it is shown. Therefore, it is possible to image almost all food in the storage room.

  Further, as shown in FIG. 9A, if the camera is provided with a dedicated camera 14-3, the food in the storage container 31 can be clearly imaged. Further, as shown in FIG. 9 (b), the camera 35-2 can be moved up and down, and the food in the storage container 31 is also made up of two cameras by making the camera 35-2 rotatable up and down as shown in FIG. 9 (c). Images can be taken efficiently with a small number of cameras.

  On the other hand, the interior lighting lamps 34-1 and 34-2 are provided at positions farther from the cameras 35-1 and 35-2, as is apparent from FIGS. Since the light of 34-2 is provided so as not to directly enter the imaging units of the cameras 35-1 and 35-2, the lights of the interior lighting lamps 34-1 and 34-2 are emitted from the cameras 35-1 and 35-2. The camera 35-1 and 35-2 can be prevented from being deteriorated, and a good image can always be obtained. This is because a light shielding part is provided on the camera side of the interior lighting lamps 35-1 and 35-2, or light sources such as LEDs constituting the interior lighting lamps 34-1 and 34-2 are inclined obliquely rearwardly. The same effect can be obtained by adopting a configuration such as.

  In addition, the interior lighting lamps 34-1 and 34-2 have an illuminance in the storage room of 150 to 200 lux, and when the image is taken with the illuminance as it is, the food is whitish as described in the section of the imaging operation in FIG. It is difficult to see and distinguish. Therefore, in this embodiment, it is set as the structure which reduces illuminance and images. This enables clear imaging even with a CCD camera or a CMOS camera. The illuminances of the interior lighting lamps 34-1 and 34-2 are controlled by the control calculation unit 59 so that the illuminance is lower than normal only when the door 10 is closed or the door 10 is opened for imaging. By performing such control, it is possible to perform clear imaging while maintaining the same illuminance and making it easy to see the interior during normal times when imaging is not performed. The decrease in illuminance may be caused by, for example, turning off a part of one or both of the interior lamps 34-1 and 34-2, or lighting both interior lamps 34-1 and 34-2. Various considerations such as lowering the illuminance of the lamps 34-1 and 34-2 are conceivable and may be appropriately selected.

  The images in the storage room 5 captured by the cameras 35-1 and 35-2 are transmitted to the server device 57 from the respective cameras 35-1 and 35-2, and the server device 57 or the refrigerator 51. Alternatively, the combination processing is performed in the display terminal 15, and in this embodiment, the image combining processing is facilitated.

  That is, the cameras 35-1 and 35-2 are door inner surfaces corresponding to the front ends of the shelf boards 30-1 to 30-4, in this example, the upper shelf board 30-1 and the third shelf board 30-. 3 is provided at the inner surface of the door corresponding to the front end of the shelf 3, and the images of the cameras 35-1 and 35-2 are the shelf plates 30-1 and 30-3 so that the line Z in FIG. The image is taken around the center.

Therefore, the image data in the storage chamber 5 captured by the two cameras 35-1 and 35-2 can be combined at the second shelf 30-2 portion that becomes the boundary line Z. That is, as shown in the “captured raw image” in FIG. 37, the shelf plate 30-2 is shown in common in each of the upper and lower images. If the common shelf board 30-2 moved to the upper and lower images is matched after the “image”, the captured image of the entire storage room is obtained. As a result, it is not necessary to perform extremely difficult correction processing to align and combine each food placed on the shelf board at the time of image combining processing, and image combining processing is easy. Become. Therefore, the processing program for image combining can be simplified, and the capacity of the control unit of the server device 57, the refrigerator 51, or the display terminal 15 for image combining processing is also required to be large. This can contribute to speeding up of processing speed.

  Further, since the two cameras 35-1 and 35-2 are provided substantially in front of the shelf plate 30-1 and the shelf plate 30-3 that cannot be adjusted in height, the shelf plate 30-that can be adjusted in height. Even when the height of 2 is changed, the image combining process can be easily performed by adjusting the range to be combined according to the height of the shelf 30-2 during the image combining process.

  As mentioned above, although the specific structure was illustrated and the refrigerator and refrigerator system of this invention were demonstrated, this invention is not limited to the said structure. In other words, the embodiment disclosed this time should be considered as illustrative in all points and not restrictive. That is, the scope of the present invention is shown not by the above description but by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

  For example, in the present embodiment, as a food storage management, it is applied to a refrigerator and a refrigeration system in which the storage state can be confirmed using the Internet, and when the storage period of the target food exceeds a predetermined time, a portable display terminal 15 may be notified to the notification means for displaying and notifying the operating state provided in the refrigerator itself, or applied to a single refrigerator that does not use the Internet or the like. It ’s good. If the portable display terminal is notified to 15, there is an advantage that the notification can be surely known regardless of the location of the user.

  Moreover, although the case where the image storage unit 78, the storage period storage unit 79, the comparison unit 80, the correction calculation unit 81, and the like are configured by the control calculation unit 59 and the storage unit 60 on the refrigerator side is illustrated, this is the server device 57 side. It may be configured, and may be shared by the refrigerator and the server device 57.

  Further, the object specifying means 77 is exemplified by the object food that can be specified by touching the target food displayed on the display screen of the display terminal 15, but this is a method in which, for example, a food name is directly input by voice or the like. In addition to the portable display terminal, the notification means provided in the refrigerator itself may be configured. In the case of a refrigerator alone, the notification means may be configured. Is.

  In the above embodiment, the gateway device 54 is exemplified in the communication path between the display terminal 15, the wireless adapter 53 of the refrigerator 51, and the Internet 56. However, the gateway device 54 is not necessarily required. In addition, it may be configured to communicate directly with the display terminal 15, the wireless adapter 53 of the refrigerator 51, and the Internet 56 via the router device 55, thereby simplifying the system configuration.

  In the above embodiment, the image data is acquired from the server device 57 when the user is outside the residence A. However, the Internet 56, the router device 55, and the gateway device are connected from the display terminal 15 without using the server device 57. The image data stored in the storage unit 60 of the refrigerator 51 may be directly captured from the refrigerator 51. In this case and when “direct” communication is performed, it is preferable to perform image combining processing in the storage unit 60 of the refrigerator 51 and transmit the image data after the combining processing.

Further, in this embodiment, a “direct” selection button 70 and a “server” selection button 71 are displayed on the display of the display terminal 15 so that the user can perform “direct” communication and “server”.
Although the case of selecting communication is illustrated, this is because the display terminal 15 has a function of automatically switching to “direct” communication when the radio wave of the router device 55 is caught, and “direct” communication is automatically selected. This may be done and the usability can be improved.

  INDUSTRIAL APPLICABILITY Since the present invention can provide an easy-to-use refrigerator that can be reliably stored and managed without taking any trouble at the time of food storage, it can be applied to various refrigerators including home refrigerators.

DESCRIPTION OF SYMBOLS 1 Refrigerator main body 2 Outer box 3 Inner box 4 Heat insulating material 5 Refrigeration room (storage room)
6 Switching room 7 Ice making room 8 Freezing room 9 Vegetable room 10 Door 11-11 Pull-out door (door)
15 Display terminal (image display means)
DESCRIPTION OF SYMBOLS 15a Display 16 Cooling chamber 17 Cooler 18 Blower fan 19 Compressor 20 Radiation pipe 21 Capillary tube 22 Damper 23 Storage frame body 24 Terminal holding body 25 Touch panel 26 Gap 27 Frame body 28 Recess 29, 29a Smoke sheet 30-1-30-4 Shelf plate 31 Storage container 32-1 to 32-4 Side wall convex part 33-1 to 33-8 Door pocket 34-1 and 34-2 Interior lighting lamp 35-1, 35-2, 35-3 Camera (imaging means) )
36 Center pillar 41 Camera module 50 Refrigerator system 51 Refrigerator 53 Wireless adapter (communication device)
53a "Connect" button 54 Gateway device (relay device)
55 Router device 56 Internet 57 Server device 58 Interface (I / F)
59 Control Calculation Unit 60 Storage Unit 61 Door Open / Close Detection Unit 62 Fan Drive Circuit 63 Compressor Drive Circuit 64 AC Power Supply 65 Server Interface 66 Server Control Unit 67 Data Storage Unit 70 “Direct” Selection Button 71 “Server” Selection Button 72 Image Data Selection button 73 Energy saving related data selection button 74 Send button 77 Object identification means 78 Image storage section 79 Storage period storage section 80 Comparison means 81 Correction calculation means A Residential

Claims (5)

A refrigerator body which is composed of an inner box and an outer box and has an open front,
A storage room configured by partitioning the refrigerator body;
A heat insulating door covering the storage room;
Imaging means provided in the storage chamber or the heat insulating door for imaging the inside of the storage chamber;
Object identifying means for identifying an object from an image captured by the imaging means;
A storage period storage unit for storing a storage period of the object specified by the object specifying means;
Comparison means for comparing images of the object specified by the object specifying means before and after a predetermined time;
Correction calculation means for correcting the storage period stored in the storage period storage unit under a predetermined condition,
The said correction calculating means changes and correct | amends the preservation | save period memorize | stored in the said preservation | save period memory | storage part based on the comparison result of the said comparison means, The refrigerator characterized by the above-mentioned. The correction calculation means resets the storage period when it is determined in the comparison result of the comparison means that the size of the current object is larger than the size of the object. Item 10. The refrigerator according to Item 1. 3. The refrigerator according to claim 1, wherein, in the comparison result of the comparison unit, when the absence period of the target object is longer than a predetermined time, the correction calculation unit resets the storage period. . The refrigerator according to any one of claims 1 to 3, wherein the object specifying unit specifies the object based on a singular point of the object in the image captured by the imaging unit. The refrigerator according to claim 4, wherein the singular point is a hue of an object.

Images