JP2018004247A - Refrigerator - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a refrigerator facilitating recipe search only by voice.SOLUTION: A refrigerator includes: voice recognition means for recognizing voice from the outside; communication means for transmitting voice information by the voice recognition means to an external device; display means for displaying information received from the external device; search means for searching for a recipe related to food materials displayed on the display means; door opening/closing detecting means for detecting opening/closing of a door; and a human sensor sensing whether a person is present in front of a storage room or not. The voice recognition means is actuated when it is detected that the door is opened, or it is detected that a person is present with the human sensor, and the name of a food item can be input by voice and it is possible to easily search for a recipe only by voice. Furthermore, owing to that it is not necessary to always actuate the voice recognition means by interlocking with door opening/closing and the human sensor, it is possible to prevent erroneous recognition and achieve energy saving.SELECTED DRAWING: Figure 29

Description

  The present invention relates to a refrigerator capable of searching for recipes related to food in a storage only by voice.

  In general, many refrigerators and refrigeration systems of this type have been proposed. For example, the refrigerator and the refrigeration system described in Patent Document 1 are provided with a small camera in the storage, and the food storage status in the storage captured by the small camera is displayed on a display terminal such as a mobile phone via the Internet line. It is configured so that it can be confirmed.

  Patent Document 2 also discloses a refrigerator and a refrigeration system similar to Patent Document 1, and the one described in Patent Document 1 captures the state of food in a storage with a small camera when the refrigerator door is closed. It is comprised so that it may do.

  Furthermore, the refrigerator and the refrigeration system described in Patent Document 3 are provided with a display terminal on the surface of the refrigerator in addition to the structure described in Patent Document 1, and the display terminal includes a fixed terminal and a refrigerator incorporated on the refrigerator surface. It is composed of a display terminal such as a mobile phone that can be taken away from the surface, and both terminals can display the food storage status in the storage taken with a small camera via the Internet line on the display part, and when they are out Of course, it is configured so that the food storage status in the storage can be known without opening the refrigerator door even at home.

JP 2002-236798 A JP 2007-46833 A JP 2002-81818 A

  As described above, many refrigerators and refrigerator systems of this type have been proposed, that is, refrigerators and refrigerator systems in which food in a storage can be photographed and the storage status can be checked with a display terminal or the like. Nothing has been put to practical use yet.

  Therefore, the inventors have made studies for practical application of the refrigerator and the refrigerator system as described above, and have verified the practical application. As a result, the present inventors have found that this type of refrigerator and its refrigerator system have a number of problems that need to be solved before practical use.

  In order to search for recipes while checking the food in the storage, one of the operations is to store the food to be searched once and enter the name of the food in a restricted place where the display terminal can be used. It is necessary to search for recipes, for example, if your hands are dirty while preparing a table, you must wash your hands to operate the screen, and it is difficult to operate the screen if your hands are full However, there is a problem that recipe search cannot be performed.

  The present invention has been made in view of the above points, and it is possible to perform a recipe search without memorizing the food to be searched without operating the screen by hand, and only by voice while checking the food. It is an object of the present invention to provide a refrigerator and a refrigeration system that can easily perform a recipe search only by voice without opening a storage box when it is not necessary to perform a recipe search or confirm food.

  In order to achieve the above object, the present invention achieves the above-described object, a storage room that is partitioned by a heat insulating wall and a heat insulating door and stores stored items, a heat insulating door that covers the storage room, a voice recognition unit that recognizes sound from the outside, and a voice recognition unit The communication means for transmitting the voice information to the external device, the display means for displaying the information received from the external device, and the search via the communication means based on the content displayed on the display means or the voice information by the voice recognition means A search means to perform, a door opening / closing detection means for detecting opening / closing of the door, and a human sensor for detecting the presence / absence of a person in front of the storage room, and when the opening / closing detection means detects the opening of the heat insulating door, or When the presence of a person is detected by the human sensor, the voice recognition means is operated, and the name of the food can be input by voice, so that recipe search and the like can be easily performed only by voice. In addition, since it is not necessary to always operate the voice recognition means by interlocking with the door opening / closing or the human sensor, it is possible to prevent erroneous recognition and to save energy.

  ADVANTAGE OF THE INVENTION According to this invention, the refrigerator which can perform a recipe search etc. easily only with an audio | voice without using a hand can be provided.

The perspective view of the refrigerator in embodiment of this invention Schematic sectional view of the refrigerator The perspective view which shows the inside when the storage door of the refrigerator is opened (A) The perspective view which equipped the same refrigerator with the display terminal integrally, (b) The perspective view which equipped the display terminal with the display terminal so that attachment or detachment was possible. The example of the mounting configuration of the display terminal used for the refrigerator is shown, (a) is a cross-sectional view when the display terminal is attached integrally, (b) is a cross-sectional view when the display terminal is detachably attached. The display visual recognition structural example of the display terminal used for the refrigerator is shown, (a) (b) is a schematic sectional view when the cover panel of the door is made of glass, (c) is a cover panel of the door configured with a magic mirror Schematic cross section when Longitudinal sectional view showing an example of camera mounting of the refrigerator The camera attachment example of the refrigerator is shown, (a) is a cross-sectional view of the refrigerator door portion with the door closed, and (b) is a front view of the door inner surface when the door is opened. (A) (b) (c) The longitudinal cross-sectional view which shows the other example of the camera attachment example of the refrigerator Explanatory drawing which shows an example of the camera attached to the refrigerator System configuration diagram of refrigerator system in the same embodiment Block diagram showing the refrigerator configuration of the refrigerator system Block diagram showing the server configuration of the refrigerator system Flowchart showing photographing to data storage operation when the door of the refrigerator system is closed Flow chart showing photographing to data storage operation during cooling room cooling operation of the refrigerator system Flowchart showing photographing to data storage operation when the door of the refrigerator system is opened FIG. 14 to FIG. 16 are flowcharts showing other operation examples of the photographing operation. The flowchart which shows the case where data storage and composition operation in the refrigerator system are performed by the server The flowchart which shows the case where the data preservation and composition operation in the refrigerator system are done on the refrigerator side The flowchart which shows the case where the data storage and composition operation in the refrigerator system are performed on the display terminal Flow chart showing normal image data transmission / reception in the refrigerator system The flowchart which shows transmission / reception of the image data in case the image data is preserve | saved at the display terminal in the refrigerator system The flowchart which shows the case where a separate image and a synthesized image are switched and transmitted / received in the refrigerator system Flowchart showing image data transmission / reception upon advance transmission request in the refrigerator system Front view of a display terminal for explaining switching of transmission / reception paths in the refrigerator system The figure for demonstrating the image composition process in the refrigerator system The perspective view of the refrigerator in embodiment of this invention The perspective view which shows the inside when the storage door of the refrigerator is opened The block diagram which shows the refrigerator structure regarding the voice recognition of the refrigerator system Flow chart showing voice recognition to data storage operation when the door is opened for voice recognition of the refrigerator system Flow chart showing voice recognition to data storage operation in presence of human sensor relating to voice recognition of the refrigerator system The flowchart which shows the voice recognition stop operation at the time of the door closing concerning voice recognition of the refrigerator system Flow chart showing voice recognition stop operation when human sensor relating to voice recognition of the refrigerator system is not available The flowchart which shows the case where the recipe search process in the voice recognition of the refrigerator system is performed by the server device The flowchart which shows the case where the recipe search process in the voice recognition of the refrigerator system is performed on the display terminal side The flowchart which shows the case where the recipe search result in the voice recognition of the refrigerator system is displayed on the display terminal

  Hereinafter, specific and detailed embodiments of the present invention will be described. First, the main points of the embodiments of the present invention will be described.

  An embodiment of a refrigerator according to the present invention includes a storage room partitioned by a heat insulating wall and a heat insulating door to store storage items, a heat insulating door that covers the storage room, a voice recognition unit that recognizes sound from the outside, and a voice recognition unit The communication means for transmitting the voice information to the external device, the display means for displaying the information received from the external device, and the search via the communication means based on the content displayed on the display means or the voice information by the voice recognition means A search means to perform, a door opening / closing detection means for detecting opening / closing of the door, and a human sensor for detecting the presence / absence of a person in front of the storage room, and when the opening / closing detection means detects the opening of the heat insulating door, or The voice recognition means is operated when the presence of a person is detected by a human sensor.

  As a result, when the opening of the door is detected or when the presence sensor detects the presence of a person, it is possible to operate the voice recognition means and input the name of the food by voice. Recipe search etc. can be easily done from the above database. In addition, since it is not necessary to always operate the voice recognition means by interlocking with the door opening / closing or the human sensor, it is possible to prevent erroneous recognition and to save energy.

  Embodiment 1 of the present invention will be described below with reference to the drawings. The present invention is not limited to the following embodiment.

[Embodiment 1]
<1. Configuration>
1-1. Configuration of refrigerator 1-1-1. Configuration of French Door Type Refrigerator First, a configuration example in the case of a French door type refrigerator will be described with reference to FIGS.

  1 to 3, the refrigerator body 1 includes a metal (for example, iron plate) outer box 2 that opens forward, an inner box 3 made of hard resin (for example, ABS), the outer box 2 and the inner box 3. It is made of a heat insulating material 4 such as hard foam urethane filled with foam. The refrigerator main body 1 has a plurality of storage compartments formed therein. This storage is located from the upper part of the refrigerator body 1 to the refrigerator compartment 5, the switching chamber 6 located below the refrigerator compartment 5, the ice making chamber 7 juxtaposed to the switching chamber 6, the switching chamber 6 and the ice making chamber 7 below. There are a freezer compartment 8 and a vegetable compartment 9 located below the freezer compartment 8. Further, the front surface of the refrigerator compartment 5 is closed freely by the double doors 10-1 and 10-2, and the front portions of the switching chamber 6, the ice making chamber 7, the freezer compartment 8, and the vegetable compartment 9 are of a drawer type. The doors 12a, 12b, 13, 14 (hereinafter referred to as pull-out doors) are opened and closed freely.

  There is a cooling chamber 16 on the back of the refrigerator main body 1, a cooler 17 that generates cool air, a blower fan 18 that supplies the cool air to each chamber, and a damper (air volume adjusting means) for adjusting the air volume to the refrigerator compartment. 22 are provided. In addition, a compressor 19 is provided at the back of the main body top of the refrigerator body 1, and a condenser (not shown), a heat radiating pipe 20, a capillary tube 21, and the cooler 17 described above are sequentially provided. A refrigerant is enclosed in a refrigeration cycle connected in a ring shape, and a cooling operation is performed.

  Here, the doors 10-1, 10-2, 12a, 12b, 13, 14 are filled with hard foamed urethane like the refrigerator body 1 to provide heat insulation, and the door 10 of the storage 5 serving as a refrigerator compartment. Among the doors 10-1 and 10-2, one door 10-1 has a narrower width than the other door 10-2, and a center pillar 11 for preventing cold air leakage is attached to the open end. And in this refrigerator, the display terminal 15 is equipped in the front surface of the other wide door 10-2. The display terminal 15 includes a display unit such as a liquid crystal panel and a control unit that controls the display unit 15 as shown in FIG. 4A. The display terminal 15 is attached to the door 10 integrally or as shown in FIG. 4B. It is formed by a smartphone or a tablet PC, and is detachable from the door 10 and may be equipped by any method.

  5 (a) and 5 (b) show examples of the configuration of mounting the display terminal 15 on the door 10, FIG. 5 (a) shows a case where the display terminal 15 is integrally attached, and FIG. The case where it is detachably mounted is shown.

  5 (a) and 5 (b), the door 10 mainly includes an outer cover panel 10a facing the front side of the refrigerator, an inner plate 10b facing the refrigerator interior, an outer cover panel 10a and an inner plate. It is comprised with the heat insulating material 4 with which it filled between 10b. The jacket panel 10a is made of a metal member such as a color steel plate or a transparent member such as a glass plate, and the inner plate 10b is a synthetic resin member formed by vacuum forming. A raised wall portion 10b-1 protruding inward in a square shape when viewed from the inner side is provided in a peripheral portion of the inner plate 10b. Further, as already described, for example, hard foamed urethane is used for the heat insulating material 4.

  Further, in the door 10 of FIG. 5A in which the display terminal 15 is integrated, a resin storage frame body 23 is provided on the front side of the door 10 so as to cover the heat insulating material 4 continuously with the jacket panel 10a. (If the jacket panel 10a is a glass plate, it is integrally bonded to the glass plate). The display terminal 15 is fixed to the storage frame body 23 via the terminal holding body 24. A touch panel 25 is provided on the front side of the display terminal 15. A gap 26 is provided between the display terminal 15 and the touch panel 25 so that the touch panel 25 does not malfunction due to noise from the display terminal 15. The display terminal 15 and the touch panel 25 are fixed by attaching a frame body 27 covering the outer periphery of the display terminal 15 and the touch panel 25 to the surface of the door 10. Thus, since the heat insulating material 4 does not touch the display terminal 15 directly by providing the storage frame 23, when the heat insulating material 4 is foamed, the display terminal 15 is exposed to a high temperature or deformed. It is possible to prevent problems such as On the other hand, in the door 10 of FIG. 5B in which the display terminal 15 is detachable, a recess 28 is formed by providing the storage frame 23 on the front surface of the door 10, and the display terminal 15 is attached to and detached from the recess 28. It is comprised so that it may do.

  6A, 6B, and 6C show an example of a display see-through configuration of the door 10 and the display terminal 15, and FIGS. 6A and 6B show that the outer cover panel 10a of the door 10 is a glass plate. In the case of 10a-1, FIG.3 (c) shows the case where the jacket panel 10a of the door 10 is comprised by the magic mirror 10a-2.

  In FIG. 6A, a smoke sheet 29 in which only a portion facing the display terminal 15 is transparent is provided between the display terminal 15 embedded in the door 10 and the glass plate 10a-1, and the display terminal 15 is Even when turned off and looks black, the glass sheet 10a-1 other than the display terminal 15 also looks black due to the smoke sheet 29 and the boundary with the display terminal 15 does not stand out, and the appearance can be maintained well. Further, in FIG. 6B, a transparent liquid crystal display unit is used as the display unit of the display terminal 15 and an entirely black smoke sheet 29a is laid along the back surface of the transparent liquid crystal display unit. Can be made good. Further, in FIG. 6C, the glass plate 10a-1 in FIGS. 6A and 6B is a magic mirror 10a-2, and the smoke sheet 29 is not required. be able to. In any case, an example in which the display terminal 15 is integrally attached to the door 10 is shown. Instead of the smoke sheet 29, the glass plate 10a-1 may be painted so that it looks blackish.

  7, 8 (a) and 8 (b) show an example of attachment of the camera and the interior lighting lamp as the main part of the present invention, and FIG. 4 is a longitudinal sectional view of the refrigerator with the door closed, FIG. 8 (a) is a cross-sectional view showing the inner surface of the door in the refrigerator with the door closed, and FIG. 8 (b) is a front view of the inner surface of the door when the door is opened.

  In FIG. 7, FIG. 8 (a), (b), the storage board 5 of the refrigerator is provided with shelf plates 30-1 to 30-4 in a plurality of upper and lower stages. The shelf boards 30-1 to 30-4 are made of a transparent material such as a transparent synthetic resin material or glass except for the surrounding frame.

  Reference numeral 31 denotes a storage container provided so as to be freely drawn out below the lowermost shelf plate 30-4. Although the lowermost shelf plate 30-4 is fixed because it constitutes the upper surface of the storage container 31, the shelf plates 30-1 to 30-3 are respectively protruding side walls provided on both side walls of the storage 5. It is mounted on 32-1 to 32-4 and is detachable. In particular, among the shelf plates 30-1 to 30-3 excluding the bottom shelf plate 30-4, the central shelf plate 30-2 is provided with a plurality of corresponding side wall protrusions 32-2 at different heights. The shelf height can be adjusted.

  33-1 to 33-3 are door pockets provided on the inner surface of the door 10 that opens and closes the opening of the storage 5 and is provided in a plurality of stages. The door pockets 33-1 to 33-3 are also formed of a material having permeability.

  Reference numerals 34-1 and 34-2 denote illumination devices (hereinafter referred to as interior illumination lamps) serving as interior illumination lamps, which are front openings of the storage 5 in front of the front ends of the shelf plates 30-1 to 30-4. It is provided on both side walls and the front opening upper wall surface. In addition, the interior lighting lamps 34-1 and 34-2 are provided with a substrate having a plurality of LEDs arranged in the vertical direction, for example, for the purpose of energy saving and covered with a translucent cover.

  35-1 and 35-2 are imaging devices (hereinafter referred to as cameras) such as cameras provided so as to be located on the front side of the storage 5 like the interior lamps 34-1 and 34-2. It is composed of a CCD camera, a CMOS camera, and the like, and the type is not particularly limited, but is preferably resistant to dew condensation.

  The cameras 35-1 and 35-2 are provided on the inner surface of the wide door 10-2 that does not have the center pillar 11 among the doors 10-1 and 10-2 for the storage 5. It is provided in the vicinity of the open end of the door 10-2 and is set so as to be located in the approximate center of the refrigerator 51.

  In this embodiment, one camera 35-1 is attached to the lower surface of the upper middle door pocket 33-1 in the front-rear direction and the other camera 35-2 is attached near the upper portion of the lowermost door pocket 33-3. is there. Here, the front-rear direction of the door pocket 33 is a direction perpendicular to the surface formed by the inner plate 10b of the door 10, and is the same direction as the front-rear direction of the storage 5 when the door 10 is closed. .

  Further, the upper camera 35-1 is provided substantially in front of the uppermost shelf 30-1, and the lower camera 35-2 is provided in front of the third shelf 30-3 in the drawing. . In other words, each camera is not a shelf 30-2 placed on a plurality of side wall projections 32-2 provided on the premise of height adjustment, but a shelf that is detachable but is not premised on height adjustment. It is provided so as to be positioned substantially in front of the plate 30-1 and the shelf plate 30-3.

  As described above, any of the above cameras 35-1 and 35-2 is provided on the door 10-2 side where the center pillar 11 (see FIG. 3) is not provided, and is positioned at a substantially central portion of the refrigerator 51. 7 and, as is clear from FIG. 7, is located in front of the front ends of the shelf boards 30-1 to 30-4, and as shown in FIG. Is provided so as to be located in the vicinity of the opposite open end side end Y, that is, substantially in the center in the left-right direction of the refrigerator 51 so that the two cameras 35-1 and 35-2 can shoot almost the entire area inside the cabinet. It is.

  The cameras 35-1 and 35-2 may be additionally provided with a dedicated camera 35-3 as shown in FIG. 9A, or the lower camera 35 as shown in FIG. -2 can be moved up and down, or the lower camera 35-2 can be turned up and down as shown in FIG. Good.

  Further, as is apparent from FIG. 7, the interior lamps 34-1 and 34-2 are provided at positions on the far side of the storage from the cameras 35-1 and 35-2, and the interior lamps 34-1 and 34-2 are provided. The second light is provided so as not to directly enter the photographing units of the cameras 35-1 and 35-2.

  More specifically, as shown to Fig.8 (a), the imaging | photography part of the cameras 35-1 and 35-2 is the irradiation amount (maximum irradiation amount) of the irradiation direction of the interior lamps 34-1 and 34-2. ) In the range in which the irradiation amount is 1/2 or more, in this example, the irradiation angle is outside the range of 60 degrees.

  Although not shown, this is provided with a light shielding portion on the camera side of the interior lighting lamps 34-1 and 34-2, or a light source such as an LED constituting the interior lighting lamps 34-1 and 34-2. May be configured such that the light from the interior lighting lamps 34-1 and 34-2 does not directly enter the photographing units of the cameras 35-1 and 35-2. It may be selected appropriately according to the installation conditions of the camera.

1-1-3. Camera Configuration FIG. 10 shows an example of a camera used for internal shooting. As described in the above examples, it is most preferable that the camera is provided at a position that is vertically separated as described in the above examples. However, even if the upper and lower cameras 35-1 and 35-2 are installed close to each other as shown in FIG. As a result of the verification, it was confirmed that the entire area can be photographed.

  Therefore, in such a case, it is conceivable that the cameras 35-1 and 35-2 are integrated to form the camera module 41, and the modularization in this way enables easy installation and cost reduction. The module can also be formed by using a wider-angle lens such as a fisheye lens.

1-1-4. Microphone Configuration FIGS. 27 and 28 show examples of microphones used for speech recognition. Since there is a case where voice recognition is performed without opening the refrigerator door and a case where voice recognition is performed without opening the refrigerator door, the microphone is configured to be attached to the front surface 100-1 of the refrigerator and the side surface 100-2 inside the refrigerator. This makes it possible to increase the voice input level during voice recognition.

1-1-5. Human Sensor Configuration FIG. 27 shows an example of a human sensor used for voice recognition. Since voice recognition may be performed without opening the refrigerator door, the human sensor 101 is attached to the front surface of the refrigerator. This eliminates the need for starting / stopping the voice recognition device.

1-2. Next, the whole refrigerator system using the refrigerator as described above will be described with reference to FIGS.

  FIG. 11 shows an overall configuration of a refrigerator system 50 using the above-described refrigerator. As shown in FIG. 11, the refrigerator system 50 is a system for confirming the state of food in the refrigerator 51 using the display terminal 15 or the like.

  The refrigerator system 50 includes a refrigerator 51 and a display terminal 15, and further includes a wireless adapter (communication device) 53, a gateway device (relay device) 54, a router device 55, the Internet 56, and a server device 57.

  The refrigerator 51 is arrange | positioned in a user's residence A, for example, is arrange | positioned in the kitchen room in the residence A.

  The wireless adapter (communication device) 53 is configured to be electrically connected to a control unit (described later) of the refrigerator 51 and communicate with the router device 55 via a gateway device (relay device) 54. The wireless adapter 53 receives the signal transmitted from the router device 55 and outputs the received signal to the control unit of the refrigerator 51. Based on this signal, the refrigerator 51 is configured to perform a corresponding operation.

  The wireless adapter 53 also acquires identification information (for example, a serial number and model number) of the refrigerator 51, image data in a warehouse to be described later, and the like from the storage unit of the refrigerator 51, and a router via the gateway device (relay device) 54. It is configured to transmit to the device 55. The wireless adapter 53 may be provided integrally with the refrigerator 51 or may be configured to be detachable.

  The wireless adapter 53 includes a “connect” button 53a. The “connect” button 53a is used to newly connect to the gateway device (relay device) 54. When the “connect” button 53 a is operated by the user, the wireless adapter 53 is configured to acquire identification information (for example, a manufacturing number or a model number) of the refrigerator 51 from the storage unit of the refrigerator 51.

  As described above, the display terminal 15 is a general-purpose portable terminal such as a smartphone or a tablet PC (personal computer). In the present embodiment, the display terminal 15 is detachable from the door 10 of the refrigerator 51 described above. And a function (means) for connecting to the Internet 56 and a function (means) for communicating with a gateway device 54 described later.

  The display terminal 15 (for example, a smartphone) is configured to connect to the Internet 56 via a telephone line network (for example, a 3G line network). In addition or alternatively, for example, the display terminal 15 is configured to connect to the Internet 56 via a public wireless LAN with a Wi-Fi communication function.

  By connecting to the Internet 56, for example, the display terminal 15 obtains image data of food in the refrigerator that is the subject of the present invention and data for energy saving operation (for example, data such as the number of times the door is opened and closed). Can be obtained from the website of the manufacturer of the refrigerator 51 via the Internet 56. By installing the acquired program on the display terminal 15 and starting the installed program, the display terminal 15 can acquire image data of food in the refrigerator. That is, the display terminal 15 can generate and transmit an operation signal for acquiring the image data of the refrigerator 51 by operating the operation screen of the display 15a. Details will be described later.

  The display terminal 15 is also connected to the router device 55 by general-purpose communication not via the Internet 56 such as Wi-Fi communication, Bluetooth (registered trademark) communication, or infrared communication, and the gateway device 54 via the router device 55. Is configured to connect to the communication. A device for that purpose (for example, a Wi-Fi antenna or the like) is incorporated in the display terminal 15.

  The gateway device (relay device) 54 is a device that relays communication between the wireless adapter 53 and the display terminal 15. For example, the gateway device (relay device) 54 and a specific low power wireless special small frequency band (924.0 to 928.0MHz). ) Is used for communication. The communication frequency band between the gateway device 54 and the wireless adapter 53 is preferably a low frequency band that reaches a long distance, and is installed in the residence A of the user where the refrigerator 51 is installed together with the router device 55.

  The gateway device 54 is also configured to be in communication connection with the display terminal 15 via the router device 55. That is, the gateway device 54 is configured so that communication (first communication) can be executed only through the router device 55 without going through the display terminal 15 and the Internet 56.

  The gateway device 54 is further configured to be connectable to the Internet 56 via the router device 55. Accordingly, the gateway device 54 can communicate with the display terminal 15 (second communication) via the Internet 56 and the router device 55 and can also communicate with the server device 57.

  Therefore, the gateway device 54 communicates with the display terminal 15 via the Internet 56 (ie, the first communication via the router device 55 only) and via the Internet 56 (ie, via the Internet 56 and the router device 55). Second communication). The reason for this configuration will be described later.

  The server device 57 is prepared by a manufacturer of the refrigerator 51, for example. The server device 57 manages information necessary for access and authentication regarding the display terminal 15, the refrigerator 51, and the wireless adapter 53. For example, the server device 57 determines whether or not the identification information of the display terminal 15 included in the communication request signal matches the identification information of the display terminal 15 stored (registered) in association with the wireless adapter 53. Authenticate (distinguish). And when these correspond, the display terminal 15 to access is authenticated as a regular display terminal, and communication with the refrigerator 51 in the residence A is permitted.

  The server device 57 stores the image data of the food in the refrigerator transmitted from the refrigerator 51, data for energy saving operation, and the like. These data are transmitted to the display terminal 15 based on a data call signal from the display terminal 15. That is, the user can confirm the food image data in the refrigerator by operating the display terminal 15 on the display.

1-3. System part configuration of refrigerator FIG. 12 shows a system part configuration of a refrigerator used in the refrigeration system.

  In FIG. 12, this refrigerator 51 includes an interface (I / F) 58, a control unit 59, a storage unit 60, a door opening / closing detection unit 61, interior lighting lamps 34-1 and 34-2, and cameras 35-1 and 35. -2, a fan drive circuit 62, a compressor drive circuit 63, a blower fan 18, and a compressor 19, for example, are supplied with power from an AC power source 64 such as a commercial power source in the house A.

  The interface (I / F) 58 exchanges data and the like between the wireless adapter 53 and the control unit 59 of the refrigerator 51.

  The control unit 59 controls the operation of the refrigerator. The control unit 59 controls the cameras 35-1 and 35-2 and the interior lighting lamps 34-1 and 34-2 to photograph food in the refrigerator, and temporarily stores the image data in the storage unit 60. It is stored and transmitted to the server device 57 on the Internet 56. Further, if there is a request signal for image data pre-transmission from the display terminal 15 via the server device 57, the display terminal 15 is instructed to execute transmission immediately via the server device 57 after photographing food. Details thereof will be described later.

  The storage unit 60 rewrites and stores the image data of the food in the cabinet and the data for energy saving operation taken by the cameras 35-1 and 35-2, in addition to the identification symbol and the control program.

  The interior lighting lamps 34-1 and 34-2 are controlled to be turned on / off by the control unit 59 based on a signal from the door opening / closing detection unit 61, and are turned on when the door 10 is opened, and are several seconds when the door 10 is closed. Turns off after a delay time of.

  The cameras 35-1 and 35-2 are controlled by the control unit 59 based on a signal from the door opening / closing detection unit 61, and shoot food in the refrigerator when the door 10 is closed. Details thereof will be described later.

  The fan drive circuit 62 receives a control signal from the control unit 59 and controls the rotation of the blower fan 18.

  The compressor driving circuit 63 receives a control signal from the control unit 59 and controls driving of the compressor 19 such as the rotational speed.

  The blower fan 18 operates according to the control of the fan drive circuit 62 and generates an air flow for circulating the cool air.

  The compressor 19 compresses a refrigerant (not shown) circulating in the refrigerator 51 according to the control of the compressor drive circuit 63.

  FIG. 29 shows the system configuration of the refrigerator related to speech recognition used in the refrigeration system.

  29, the refrigerator 51 includes an interface (I / F) 58, a control unit 59, a storage unit 60, a door opening / closing detection unit 61, interior lighting lamps 34-1 and 34-2, and microphones 100-1 and 100. -2, a fan drive circuit 62, a compressor drive circuit 63, a blower fan 18, and a compressor 19, for example, are supplied with power from an AC power source 64 such as a commercial power source in the house A.

  The interface (I / F) 58 exchanges data and the like between the wireless adapter 53 and the control unit 59 of the refrigerator 51.

  The control unit 59 controls the operation of the refrigerator. In addition, the control unit 59 controls the microphones 100-1 and 100-2 to recognize a voice from the outside, temporarily stores the voice recognition data in the storage unit 60, and the server device 57 of the Internet 56. Send to. Further, if there is a request signal for image data pre-transmission from the display terminal 15 via the server device 57, the display terminal 15 is instructed to execute transmission immediately via the server device 57 after photographing food. Details thereof will be described later.

  The storage unit 60 rewrites and stores data recognized by the microphones 100-1 and 100-2 and data for energy saving operation, in addition to the identification symbol and the control program.

  The interior lighting lamps 34-1 and 34-2 are controlled to be turned on / off by the control unit 59 based on a signal from the door opening / closing detection unit 61, and are turned on when the door 10 is opened, and are several seconds when the door 10 is closed. Turns off after a delay time of.

  The microphones 100-1 and 100-2 are controlled by the control unit 59 based on a signal from the door opening / closing detection unit 61. When the door 10 is opened, the voice recognition device is activated, and when the door 10 is closed, a certain time elapses. Later, the voice recognition device is stopped. Details thereof will be described later.

  The fan drive circuit 62 receives a control signal from the control unit 59 and controls the rotation of the blower fan 18.

  The compressor driving circuit 63 receives a control signal from the control unit 59 and controls driving of the compressor 19 such as the rotational speed.

  The blower fan 18 operates according to the control of the fan drive circuit 62 and generates an air flow for circulating the cool air.

  The compressor 19 compresses a refrigerant (not shown) circulating in the refrigerator 51 according to the control of the compressor drive circuit 63.

1-4. Configuration of Server Device Next, a configuration example of the server device of the refrigerator system in Embodiment 1 will be described using FIG. Here, the configuration of the server device 57 in FIG. 11 is taken as an example.

  As shown in FIG. 13, the server device 57 includes a server interface 65, a server control unit 66, and a data storage unit 67.

  The server interface 65 is a communication unit that exchanges data and the like between the display terminal 15 and the router device 55 and the server device 57 via the Internet 56 under the control of the server control unit 66.

  The server control unit 66 controls the overall operation of the server device 57. The server control unit 66 also combines the image data of the food transmitted from the refrigerator 51 and stores it in the data storage unit 67. When there is a signal for requesting prior transmission of image data from the display terminal 15, the data storage unit The latest image data of the image data stored in 67 is transmitted to the display terminal 15 each time the image data is transmitted from the refrigerator 51, and the display terminal 15 is rewritten and updated. Of course, when a data call signal is transmitted from the display terminal 15 in the absence of a signal for requesting prior transmission of image data, the latest image data stored in the data storage unit 67 is transmitted to the display terminal 15.

  The data storage unit 67 stores necessary data such as a management program and an application program for executing the refrigerator system. Further, the data storage unit 67 rewrites and stores the latest data of the food image data transmitted from the refrigerator 51 under the control of the server control unit 66. The data storage unit 67 is configured by, for example, an HDD (Hard Disc Drive), an SSD (Solid State Drive), or the like.

  The server device 57 can be constructed by either a server composed of a single large computer or a cloud server composed of a large number of computers.

<2. Operation>
Next, the operation of the refrigerator and the refrigeration system configured as described above will be described. In the following descriptions, the same operations are described with the same step numbers added.

2-1. Food photographing operation 2-1-1. Photographing when the door is closed FIG. 14 is a flowchart showing an operation when photographing food in the storage when the door 10 of the refrigerator 51 is closed.

  When the door 10 is opened to put in and out the food (Y in Step 1), the interior lamps 34-1 and 34-2 are turned on (Step 2), and when the door 10 is closed (Y in Step 3), The illuminance of the interior lamps 34-1 and 34-2 is decreased (step 4), and after a few seconds, for example, a few seconds have elapsed, the cameras 35-1 and 35-2 are activated to photograph the food in the interior (step 5). . Thereafter, the captured image data is stored in the storage unit 60 of the refrigerator 51 (step 6), and the interior lighting lamps 34-1 and 34-2 are turned off (step 7).

  Here, if the illuminance of the interior lamps 34-1 and 34-2 is reduced during the above photographing, the food can be clearly photographed, which is preferable. That is, when food is photographed by turning on the interior lighting lamps 34-1 and 34-2, the inner surface of the storage is white, and the reflected light is strongly reflected from the entire peripheral surface except the opening of the storage. However, the food is whitish with high illuminance without lowering the illuminance, and the food becomes whitish and clear photographing becomes difficult. However, if the illuminance of the interior lamps 34-1 and 34-2 is reduced as in the above example, the influence of the reflected light from the inner surface of the storage can be greatly reduced, and a sharp image can be taken. It becomes.

2-1-2. Photographing at the time of cooling room cooling operation FIG. 15 is a flowchart showing an operation when photographing food in the storage room at the time of cooling the refrigerator after the food in and out of the refrigerator is taken in and out.
When the door 10 is opened to put food in and out (Y in Step 1), the interior lighting 34-1 and 34-2 are turned on (Step 2), the door 10 is closed (Y in Step 3), and the refrigerator compartment It is determined whether or not it is in the cooling operation state (step 8). If the blower fan 18 is ON and the damper 22 is open and the refrigerator is in the cooling operation, the illuminance of the interior lamps 34-1 and 34-2 is reduced as in the case of closing the door (step 4). ), The cameras 35-1 and 35-2 are activated, and the food in the cabinet is photographed (step 5). Thereafter, the captured image data is stored in the storage unit 60 of the refrigerator 51 (step 6), and the interior lighting lamps 34-1 and 34-2 are turned off (step 7).

  In this way, by taking a picture after the cooling room cooling operation, even if dew condensation occurs due to a temperature difference between the inside and outside of the storage, for example, on the surface of the food newly put in the storage, the condensation is eliminated by the cooling operation and disappears. . Therefore, photographing with the cameras 35-1 and 35-2 is preferably performed so clearly that the characters written on the food packaging container can be read.

  During the above operation, when the door 10 is closed (Y in Step 1), the interior lighting lamps 34-1 and 34-2 are once turned off, and the interior lighting lamps 34-1 and 34 are again turned on before photographing. -2 is turned on to reduce the illuminance thereof, it is possible to suppress wasteful power consumption due to continuous lighting of the interior lighting lamps 34-1 and 34-2.

2-1-3. Photographing when door is opened FIG. 16 is a flowchart showing an operation when photographing a food in a storage when the door 10 is opened at a predetermined angle in the case of a single door.

  When the door 10 is opened to put food in and out (Y in Step 1), the interior lamps 34-1 and 34-2 are turned on (Step 2), and the opening angle of the door is a predetermined angle, for example, as shown in FIG. When the opening angle shown is 30 degrees (Y in Step 10), the illuminance of the interior lighting lamps 34-1 and 34-2 is decreased (Step 4), the cameras 35-1 and 35-2 are activated, and the warehouse The food inside is photographed (step 5). Thereafter, the photographed image data is stored in the storage unit 60 of the refrigerator 51 (step 6). When the door is closed (Y in step 11), the interior lighting lamps 34-1 and 34-2 are turned off (step 7). .

  If the door 10 is photographed when the door 10 is opened at a predetermined angle as in this example, the interior of the storage can be photographed over a wide area as described above, and the image becomes natural and uncomfortable. It is.

  Note that when shooting is performed when the predetermined angle is opened, shooting is performed at least twice during the door opening operation and the door closing operation. At this time, in the operation time from one door opening to closing, the last taken image data (at the time of the door closing operation) is stored in the storage unit 60, so that the latest storage that has finished taking in and out the food is stored. The state of food can be photographed.

  Needless to say, in step 10 for detecting whether or not the opening angle of the door 10 reaches a predetermined angle, if the door 10 is closed before reaching the predetermined angle, the interior illumination lamp 34-1 is used. , 34-2 is turned off, and it is waited for the door 10 to be opened again.

  As described above, a typical example of the shooting operation has been described. However, in addition to the shooting operation, for example, shooting is performed when a predetermined time elapses after a predetermined time or the previous shooting, or when a shooting instruction is given from the display terminal 15 A photographing operation such as, for example, is also conceivable.

2-1-4. Illumination at the time of photographing Food illumination at the time of photographing reduces the illuminance after lighting the interior lighting lamps 34-1 and 34-2, as already described in FIG. 14, FIG. 15, and FIG. To shoot and shoot. Depending on the case, it is also conceivable to illuminate and shoot with reduced illumination as lighting of only the interior lighting 34-1 on both sides of the storage or the interior lighting 34-2 on the upper surface of the storage. The reason for shooting with the illuminance lowered will be described later.

  As another example of food irradiation at the time of photographing, the following may be considered. That is, FIG. 17 is a flowchart showing the state of lighting and extinguishing of the interior lighting lamps 34-1 and 34-2 at the time of photographing by the cameras 35-1 and 35-2, which are shown in FIGS. This corresponds to the operation of the broken line portion.

  The imaging in FIGS. 14, 15, and 16 is an example in which the illumination lamps 34-1 and 34-2 are turned on and the illuminance is lowered, and in this example, both sides of the storage are taken. Each one of the interior lighting lamps 34-1 on the surface is turned off, in other words, only one of the interior lighting lamps 34-1 on both sides of the storage is turned on, and the food is irradiated and photographed. In this case, the interior illumination lamp 34-2 on the upper surface of the storage may be either turned on or off, but any one may be appropriately selected so that the illuminance in the interior is optimal for photographing.

  In FIG. 17, before shooting with the cameras 35-1 and 35-2, one of the interior lighting lamps 3-1 on both sides of the storage that has already been lit, for example, the right interior lighting lamp is turned off ( Step 12), taking a picture of the food that has been irradiated by the left interior lamp (step 13) and storing it (step 14). Thereafter, the right interior lamp is turned on (step 15), and then the left interior lamp is turned off (step 16), and the food being irradiated with the right interior lamp is photographed (step 17). ). The image data is stored (step 18), and the right interior illumination lamp is turned off (step 19). Thus, the image data stored by irradiating food from one side is synthesized and stored as one image.

  As described above, the interior lighting lamps 34-1 and 34-2 are turned on one by one to photograph foods and combine them into one. In other words, as described above, the illuminance of the interior lighting lamps 34-1 and 34-2 is reduced during photographing, and if the illuminance reduction is too large, the image becomes dark and lacks in clarity. Therefore, there is a limit to the decrease in illuminance. For this reason, a certain level of illuminance is maintained. However, even if the illuminance is high, the effect of reflected light may remain when the internal volume of the storage is small. However, as described in this example, when food is photographed by lighting one of the left and right sides of the interior illumination lamp 34-1, the influence of the reflected light can be reduced, and the interior illumination lamp illumination lamp 34-1 is illuminated. The surface on the side you are shooting can be taken with moderate brightness. Therefore, it is preferable to perform a combining process that combines the left and right images of food taken with moderate brightness, so that the whole image becomes clear.

2-2. Image Storage and Composition Next, storage and composition of image data taken by the above-described operations will be described.

2-2-1. Storage and Compositing by Server FIG. 18 shows a case where image data is stored and combined by the server device 57. When the interior is photographed by the operations shown in FIGS. 14 to 16 including FIG. The refrigerator 51 is connected to the Internet 56 from the wireless adapter 53 via the gateway device 54 and the router device 55 (step 20), and individual image data photographed and stored by each camera is transferred to the server device 57 connected to the Internet 56. Transmit (step 21). The image data transmitted via the Internet 56 is subjected to a combining process (step 22) for combining individual image data by the server control unit 66 of the server device 57. The combined image data is stored in the data storage unit 67 of the server device 57 (step 23).

  If the server device 57 synthesizes the image and stores it as in this example, the load on the control unit 59 and the storage unit 60 on the refrigerator side can be reduced, and an Internet-compatible refrigerator can be made inexpensively. Can be provided.

2-2-2- Storage and Compositing on Refrigerator Side FIG. 19 shows a case where image data is stored and combined on the refrigerator side. When the interior is photographed by the operations shown in FIGS. 14 to 16 including FIG. The control unit 59 of the refrigerator 51 takes out individual image data from each camera stored in the storage unit 60, combines them, and combines them (step 24). The synthesized image data is stored (step 25), the refrigerator 51 is connected to the internet 56 from the wireless adapter 53 via the gateway device 54 and the router device 55 (step 20), and the stored image data is connected to the internet 56. The data is transmitted to the server device 57 (step 21). The composite image data transmitted via the Internet 56 is stored in the data storage unit 67 of the server device 57 (step 23).

  In this example, although the load on the control unit 59 and the storage unit 60 on the refrigerator side increases, image data can be transmitted and received with the display terminal 15 without going through the data storage unit 67 of the server device 57, and the system operation cost is low. Can be.

2-2-3. Storage and Compositing on Display Terminal Side FIG. 20 shows a case where image data is stored and combined on the display terminal 15 side. When the interior is photographed by the operation shown in FIGS. 14 to 16 including FIG. Connects to the Internet 56 from the wireless adapter 53 via the gateway device 54 and the router device 55 (step 20), and transmits image data to the display terminal 15 instead of the server (step 26). The image data transmitted via the Internet 56 is subjected to a combining process (step 27) for combining the image data by the display terminal 15. The combined image data is stored in the data storage unit of the display terminal 15 (step 28).

  Even in this example, although the load on the display terminal 15 increases, the load on the control unit 59 and the storage unit 60 on the refrigerator side can be reduced, and an Internet-compatible refrigerator can be provided at low cost.

  The image data transmission and the composition operation have been described above by exemplifying typical ones. However, the storage and the composition portion are not necessarily performed at the same location, for example, the server device 57. For example, the refrigerator 51 The image data may be combined with each other and the server device 57 may store the image data.

2-3. Normal transmission / reception of image data 2-3-1. Transmission / Reception of Image Data Stored in Server and Refrigerator FIG. 21 is a flowchart showing transmission / reception between the server device 57 or the refrigerator 51 and the display terminal 15, and shows transmission / reception of image data stored in the server device 57 or the refrigerator 51. is there.

First, the user presses a server connection button (not shown) displayed on the display terminal 15 at a shopping destination or the like, for example, and makes a request to read image data stored in the data storage unit 67 of the server device 57 (see FIG. Step 31). It is detected whether image data has been taken in from the server device 57 (step 32). If no image data is transferred, the display terminal 15 connects to the Internet 56 (step 33). The image data stored in the data storage unit 67 of the server device 57 or the storage unit 60 of the refrigerator is taken out (step 34), and the image data is transferred to the display terminal 15 such as a smartphone that is a data caller via the Internet 56. Is transferred (step 35). It is detected whether image data has been captured from the server device 57 (step 32), and when the image data is captured, the image data is output to the display of the display terminal 15 (step 36).
Thereafter, it is detected whether or not an enlarge button (not shown) displayed on the display of the display terminal 15 has been pressed (step 37). Image data obtained by enlarging image data 1.1 times or more by digital conversion is created. The enlarged and converted image data is output to the display of the display terminal 15 (step 36).

  After that, it is detected whether a numeric key button (not shown) displayed on the display is pressed (step 38). When it is detected that the numeric key button is pressed, the enlarged image data in the storage output on the display is displayed. The image data at the location designated by the numeric keypad is output to the display (step 36). Further, it is detected whether the numeric keypad is pressed (step 38). Exit if not detected.

  It should be noted that the enlargement button and the numeric keypad for displaying image data at a predetermined location may be of a type in which the surface of the display is slid with a finger. Such a configuration improves usability.

  As described above, according to this refrigerator system, the contents of the refrigerator 51 can be confirmed at a glance at a shopping destination, so that forgetting to buy can be prevented and the waste of purchasing extra ingredients can be eliminated. . In addition, in a family working together, it is possible to check the inside of the storage 5 of the refrigerator 51 just before returning, and to purchase foods and the like on the way home, thereby reducing time for the user.

  In addition, by displaying an enlarged display or a predetermined location, it is possible to reliably display foods that are difficult to confirm on the small display terminal 15 by displaying an enlarged image on the display terminal 15 that has been miniaturized like the display terminal 15. Can see.

2-3-2. Transmission / Reception of Image Data Saved on Display Terminal FIG. 22 shows a case where image data is already stored and saved on the display terminal 15. In this case, as described with reference to FIG. 24, the image data is transmitted and stored in the display terminal 15, and when this image data is requested to be read (step 31), the image data is fetched from the storage unit of the display terminal 15. The image data is output to the display of the display terminal 15 (step 36). The subsequent enlargement display operation and local display are as described with reference to FIG.

2-3-3. Image data switching reception Images taken by the cameras 35-1 and 35-2 are obtained by taking the respective photographed images of the cameras 35-1 and 35-2 (the upper half or the lower half in the storage) and the respective photographed images. An example is shown in which composite images combined and combined (images combined in the storage from the top to the bottom) are stored and saved, and these images can be switched and transmitted and displayed on the display terminal 15.

  FIG. 23 is a flowchart showing the operation in such a case.

  First, the user presses an individual image / composite image call button (not shown) displayed on the display of the display terminal 15 (step 41), and calls either an individual image or a composite image (step 42). ). Thereafter, either the individual image data or the composite image is captured by the flow shown in FIG. 25 or 26 and displayed on the display of the display terminal 15.

  When the image displayed on the display terminal 15 can be switched to either an individual image or a composite image as in this example, the state of food in the storage can be confirmed at a glance by generally calling and displaying the composite image. However, when it is difficult to discriminate the food due to the corner of the storage, etc., it can be surely discriminated by switching to an individual image and displayed, which improves usability.

2-4. Transmission / Reception at the Time of Pre-transmission Request for Image Data etc. FIG. 24 shows a flowchart in the case where an image pre-transmission request is received from the display terminal 15.

  First, the user operates the display terminal 15 to perform a request operation for a prior request signal (step 51). The server device 57 or the refrigerator 51 receives this request signal (step 52), and reads and transmits image data (individual image data or composite image data, the same applies hereinafter) stored in the data storage unit 67 or the storage unit 60 (step 52). 53). The display terminal 15 detects whether the image data from the server device 57 or the refrigerator 51 has been taken in (step 54). If no image data is transmitted, the display terminal 15 connects to the Internet 56 (step 55). The server devices 57 take out the image data stored in the refrigerator 51 (step 56), and transmit the image data to the display terminal 15 via the Internet 56 (step 57). It is detected whether the image data has been taken in from the server device 57 (step 54). When the image data is detected, the process waits until the image data is called (N in step 58). If there is an image data call (Y in step 58), the image data is output to the display of the display terminal 15 (step 59).

  Thus, when image data is requested on the display terminal 15, even when access to the server device 57 is concentrated and transmission of image data from the server device 57 is delayed, the image data is requested in real time simultaneously with the request for image data. Can be displayed on the display to check the image. Therefore, it is possible to avoid a situation of waiting for image data to be displayed at the shopping destination, and to shorten the shopping time.

  In addition, if there is an image pre-transmission request from the display terminal 15, the server device 57 is set to transmit image data transmitted each time the door 10 of the refrigerator 51 is closed to the display terminal 15, and the display terminal 15 If the image data is transmitted, the user who has a can know this every time the door is opened or closed. Therefore, for example, when the number of times of door opening / closing increases and the power consumption is likely to increase, the user can talk to the child to refrain from using the refrigerator 51 by calling his home. Convenience is also improved.

2-5. Switching of transmission / reception path of image data etc. FIG. 25 shows a display on the display terminal 15 when a communication path is selected. The flow of transmission / reception of image data is as described above, but the communication route is changed to speed up the acquisition of image data.

  Specifically, the user can select whether to perform the display terminal 15 directly with the gateway device 54 via the router device 55 or with the server device 57 via the Internet 56.

  In FIG. 25, reference numerals 70 and 71 are communication path selection buttons displayed on the display of the display terminal 15, and 70 is a “direct” selection for directly connecting the display terminal 15 to the refrigerator 51 via the router device 55 and the gateway device 54. A button 71 is a “server” selection button for connecting to the server device 57 or the refrigerator 51 via the Internet 56. 72 and 73 are data selection buttons for data to be called, 72 is a button for food image data, and 73 is a button for energy saving related data such as the number of times the door is opened and closed. Reference numeral 74 denotes a transmission button.

  When the “direct” selection button 70 is selected, the display terminal 15 is set to “direct” communication. When the user touches the send button 74 while “direct” communication is set, the image data request signal of the display terminal 15 is directly transmitted from the display terminal 15 to the gateway device (relay device) 54 via the router device 55. Sent. The gateway device 54 transmits the received data request signal to the wireless adapter 53 of the refrigerator 51. When the wireless adapter 53 receives the image data request signal, it transmits it to the control unit 59 of the refrigerator 51. When receiving the image data request signal, the control unit 59 of the refrigerator 51 reads the image data stored in the storage unit 60 and transmits the image data to the display terminal 15 via the wireless adapter 53 and the router device 55.

  Therefore, as shown by the broken line in FIG. 11, when the user is located in the residence A, in a broad sense, when the display terminal 15 exists in a position where the user can directly communicate with the router device 55, the display terminal 15 is connected to the router device 55. The image data can be directly acquired from the storage unit 60 of the refrigerator 51 by communicating with the gateway device 54 via the network.

  As a result, when a delay occurs in the transmission response of the image data in response to the user's operation to the display terminal 15 due to congestion of the line due to concentration of access to the Internet 56, the user sets the display terminal 15 to “ By switching to “direct” communication, it is possible to acquire image data without being congested with the Internet 56 line. Therefore, the user can check the food state in the storage 5 by displaying an image on the display of the display terminal 15 in real time without waiting.

  On the other hand, as shown by the solid line in FIG. 11, when the user is located outside the residence A, in a broad sense, when the display terminal 15 is present at a position where it cannot communicate directly with the router device 55, the user selects the “server” selection button. By touching 71, the display terminal 15 communicates with the server device 57 via the Internet 56 and reads the image data of the refrigerator 51. That is, when the user is outside the residence A and touches the “server” selection button 71 to set “server” communication, the user can obtain the image data of the refrigerator by the display terminal 15 even in a remote place, You can know the storage status of.

  As described above, the refrigerator system in this embodiment allows the user to select either “direct” communication or “server” communication. You can select the communication and view the image in a comfortable state without being affected by the congestion of the Internet line.

  As mentioned above, although the specific structure was illustrated and the refrigerator and refrigerator system of this invention were demonstrated, this invention is not limited to the said structure.

  For example, in the above embodiment, the gateway device 54 is interposed in the communication path between the display terminal 15, the wireless adapter 53 of the refrigerator 51, and the Internet 56. However, the gateway device 54 is not necessarily required, and the router device 55. It may be configured to communicate directly with the display terminal 15, the wireless adapter 53 of the refrigerator 51, and the Internet 56, and in this way, the system configuration can be simplified.

  In the above embodiment, the image data is acquired from the server device 57 when the user is outside the residence A. However, the Internet 56, the router device 55, and the gateway device are connected from the display terminal 15 without using the server device 57. The image data stored in the storage unit 60 of the refrigerator 51 may be directly captured from the refrigerator 51. In this case and when “direct” communication is performed, it is preferable to perform image combining processing in the storage unit 60 of the refrigerator 51 and transmit the image data after the combining processing.

  Furthermore, in this embodiment, a case where the “direct” selection button 70 and the “server” selection button 71 are displayed on the display of the display terminal 15 and the user selects “direct” communication or “server” communication is illustrated. However, this may be configured so that the display terminal 15 has a function of automatically switching to “direct” communication when the radio wave of the router device 55 is caught, so that “direct” communication is automatically selected. Can be improved.

2-6. Voice recognition operation for food name 2-6-1. Voice Recognition Operation when Door is Opened FIG. 30 is a flowchart showing the operation when the voice recognition device is activated when the door 10 of the refrigerator 51 is opened.

  When the door 10 is opened to confirm the food (Y in Step 101), the voice recognition devices 100-1 and 100-2 are activated (Step 102), and the interior lamps 34-1 and 34-2 are turned on. (Step 103). When the voice is recognized by the voice recognition device (Y in Step 104), the voice input level of the voice recognition devices 100-1 and 100-2 is confirmed by the microphone discrimination process to determine which voice recognition data is to be used. (Step 105), the voice data adopted by the voice analysis process is analyzed to determine the name of the input food (Step 106), and the determined voice data is stored in the storage unit 60 of the refrigerator 51 (Step 107). . If no voice is recognized by the voice recognition device (N in Step 104), the state of the door 10 is confirmed. If the door 10 is not closed, the voice of the voice recognition device is checked again (N in Step 108). When the door 10 is closed (Y in Step 108), the interior lighting lamps 34-1 and 34-2 are turned off (Step 109), and after a few seconds, for example, the voice recognition devices 100-1 and 100-2 are passed. Is stopped (step 110).

  Here, it is preferable that the voice recognition device is interlocked with the door 10 so that the voice recognition device can be activated when the door is opened and the voice recognition device can be stopped when the door is closed. That is, since it is unnecessary to always operate the voice recognition device, it is possible to prevent erroneous voice recognition and to save energy.

2-6-2. Voice recognition operation by presence of human sensor FIG. 31 is a flowchart showing an operation when the voice recognition device is activated when it is detected that a person is present in front of the refrigerator 51.

  When the presence sensor 101 detects that a person is present in front of the refrigerator (Y in step 111), the voice recognition devices 100-1 and 100-2 are activated (step 102). When the voice is recognized by the voice recognition device (Y in Step 104), the voice input level of the voice recognition devices 100-1 and 100-2 is confirmed by the microphone discrimination process to determine which voice recognition data is to be used. (Step 105), the voice data adopted by the voice analysis process is analyzed to determine the name of the input food (Step 106), and the determined voice data is stored in the storage unit 60 of the refrigerator 51 (Step 107). . When the voice is not recognized by the voice recognition device (N in Step 104), if the person is detected by checking the state of the human sensor 101, the voice of the voice recognition device is checked again (N in Step 112). . If no person is detected (Y in step 112), the speech recognition apparatuses 100-1 and 100-2 are stopped for a while, for example, after a few seconds have elapsed (step 110).

  Here, it is preferable that the voice recognition apparatus is interlocked with the human sensor 101 so that the voice recognition apparatus can be activated when a person is detected, and the voice recognition apparatus can be stopped when no person is detected. That is, since it is unnecessary to always operate the voice recognition device, it is possible to prevent erroneous voice recognition and to save energy.

  As described above, the typical voice recognition operation has been described. However, in addition to the voice recognition operation, for example, a voice recognition operation such as activation when a voice recognition activation instruction is issued from the display terminal 15 is also conceivable.

2-6-3. Voice Recognition Operation when Door is Closed FIG. 32 is a flowchart showing the operation when the voice recognition device is stopped when the door 10 of the refrigerator 51 is closed.
When the door 10 is closed (Y in Step 108), the interior lighting lamps 34-1 and 34-2 are turned off (Step 109), and after a few seconds, for example, the voice recognition devices 100-1 and 100-2 are passed. Is stopped (step 110). If the door 10 is not closed, the state of the door is confirmed again (N in step 108).

  Here, it is preferable that the voice recognition device can be stopped when the door is closed by interlocking the voice recognition device with the door 10. That is, there is no need to manually stop the voice recognition device and there is no need to forget to stop the voice recognition device, so that erroneous voice recognition can be prevented and energy saving can be realized.

2-6-4. FIG. 33 is a flowchart showing an operation when the voice recognition device is stopped when it is detected that no person is present in front of the refrigerator 51.
When the state of the human sensor 101 is confirmed and no person is detected (Y in step 112), the voice recognition devices 100-1 and 100-2 are stopped for a while, for example, after several seconds have passed (step 110). If a person is detected, the state of the human sensor is confirmed again (N in step 112).

  Here, it is preferable to link the voice recognition device to the human sensor 101 so that the voice recognition device can be stopped when no person is detected. That is, there is no need to manually stop the voice recognition device and there is no need to forget to stop the voice recognition device, so that erroneous voice recognition can be prevented and energy saving can be realized.

  As described above, the typical voice recognition operation has been described. However, in addition to the voice recognition operation, for example, a voice recognition operation such as stopping when a voice recognition stop instruction is issued from the display terminal 15 is also conceivable.

2-7. Recipe Search for Food Recognized by Voice Next, recipe search using data voice-recognized by the above-described operations will be described.

2-7-1. 34. Recipe Search by Server FIG. 34 shows a case where a recipe search is performed by the server device 57 according to the voice recognition data. When the voice is recognized by the operation shown in FIGS. 30 and 31, the refrigerator 51 is switched from the wireless adapter 53 to the gateway device 54, the router. It connects to the Internet 56 via the device 55 (Step 113), and transmits the voice-recognized data to the server device 57 connected to the Internet 56 (Step 114). The speech recognition data transmitted via the Internet 56 is subjected to a recipe search process (step 115) using the speech recognition data by the server control unit 66 of the server device 57. The recipe search result is stored in the data storage section 67 of the server device 57 (step 116).

  If the server device 57 performs recipe search and stores and saves as in this example, the load on the control unit 59 and the storage unit 60 on the refrigerator side can be reduced, and an Internet-compatible refrigerator can be made inexpensively. Can be provided.

2-7-2. Recipe search on the display terminal side FIG. 35 shows a case where the recipe search is performed on the display terminal 15 side according to the voice recognition data. When the voice is recognized by the operation shown in FIG. 30 and FIG. 54, the router 56 is connected to the Internet 56 (step 113), and the voice-recognized data is transmitted not to the server but to the display terminal 15 (step 117). The voice recognition data transmitted via the Internet 56 is subjected to recipe search processing (step 118) using the voice recognition data by the display terminal 15. The result of the recipe search is stored in the data storage unit of the display terminal 15 (step 119).

  Even in this example, although the load on the display terminal 15 increases, the load on the control unit 59 and the storage unit 60 on the refrigerator side can be reduced, and an Internet-compatible refrigerator can be provided at low cost.

  As described above, the transmission of the voice-recognized data and the recipe search operation have been exemplified and described. However, the storage and the recipe search location are not necessarily the same location, for example, the server device 57. For example, the recipe search processing is performed on the display terminal 15 side, and the recipe search result is stored in the server device 57. The setting may be arbitrarily set according to the system.

2-8. Recipe display on display terminal 2-8-1. Display of Recipe Search Result on Display Terminal FIG. 36 is a flowchart showing transmission / reception between the server device 57 or the refrigerator 51 and the display terminal 15, and shows transmission / reception of data stored in the server device 57 or the refrigerator 51.

  If there is voice recognition, the display terminal 15 receives the voice recognition data identified by the voice analysis processing of the refrigerator 51 and displays it on the display terminal 15 (step 120). After the voice recognition data is displayed, a recipe search is performed on the server device 57 or the display terminal 15 by pressing a search start button on the display terminal 15 or by saying a specific keyword, and the recipe search status is displayed until the recipe search is completed. Confirmation is continued (N in Step 121). When the recipe search is completed (Y in step 121), the recipe search result is displayed on the display terminal 15 (step 122). Note that the recipe search result may be automatically adjusted so that the optimum display position is obtained according to the height of the user.

  As described above, according to this refrigerator system, the result of the recipe search can be displayed on the display terminal 15, and by removing the display terminal 15, the place where the display terminal is viewed is not limited and cooking is performed. You can carry it to the immediate vicinity and check it, which will improve the cooking efficiency.

<3. Effect>
As described above, the refrigerator and the refrigerator system in the embodiment of the present invention have various functions and effects as described above, but the refrigerator that is the gist of the present invention, that is, the front surface of the refrigerator door, By providing the voice recognition device on the side surface inside the refrigerator, it is possible to recognize the voice regardless of whether the refrigerator door is open or closed, and to provide a refrigerator capable of retrieving recipes only by voice. it can. In addition, since the voice recognition means does not always have to operate by interlocking with the door opening / closing or the human sensor, it is possible to provide a refrigerator that can prevent erroneous recognition and contribute to energy saving.

  In the refrigerator of this embodiment, the cameras 35-1 and 35-2 photograph foods in the storage 5 from the front side of the storage 5. At this time, the interior lighting lamps 34-1 and 34-2 are also stored in the storage. 5, the food in the storage 5 is irradiated from the same front side as the photographing direction by the cameras 35-1 and 35-2. Therefore, the captured image is not as dark or difficult to see as when the interior lighting lamps 34-1 and 34-2 are provided on the back wall of the storage 5, and becomes a clear image. Further, it is not necessary to request the server device 57 to perform image processing that brightens and sharpens the captured image.

  Moreover, since the said cameras 35-1, 35-2 and the interior lighting lamps 34-1, 34-2 are provided so that it may be located ahead from the front end of the said shelf boards 30-1 to 30-4, Images in the storage 5 taken by the cameras 35-1 and 35-2 are the same as when the user actually opens the door 10 of the refrigerator 51 and sees it. Therefore, when the user causes the display terminal 15 to display the image data, the food state in the storage 5 can be confirmed without any discomfort. Moreover, since the interior lighting lamps 34-1 and 34-2 illuminate the food on the shelf board from the front front of the shelf boards 30-1 to 30-4, they are mounted on the front edge parts of the shelf boards 30-1 to 30-4. The food that is placed can be photographed brightly and clearly.

  The cameras 35-1 and 35-2 are provided on the inner surface of the door 10 of the refrigerator 51, and the interior lamps 34-1 and 34-2 are provided on both side walls of the opening edge of the storage 5. First, 35-1 and 35-2 are positioned at the substantially right and left central portions of the storage 5. Accordingly, the images taken by the cameras 35-1 and 35-2 can be made closer to a state in which the user opens the door 10 of the refrigerator 51 and actually sees it. In addition, the distance between the cameras 35-1 and 35-2 and the food on the shelf can be increased so that a wide range in the storage 5 can be photographed. 15 can be confirmed.

  In addition, since the cameras 35-1 and 35-2 are provided so as to be positioned at a substantially central portion in the front-rear direction of the door pockets 33-1 and 33-2 on the inner surface of the door, not only the food in the storage 5 but also the door pockets. A part of the food stored in 33-1 and 33-2, for example, milk cartons can be photographed. Since the user has stored it by himself, the user can recognize what it is even if only a part of it is shown. Therefore, it is possible to photograph almost all food items in the storage.

  Further, as shown in FIG. 9A, if the camera is provided with a camera 35-3 dedicated to the storage container, the food in the storage container 31 can be clearly photographed. Further, as shown in FIG. 9 (b), the camera 35-2 can be moved up and down, and the food in the storage container 31 is also made up of two cameras by making the camera 35-2 rotatable up and down as shown in FIG. 9 (c). You can shoot efficiently with a small number of cameras.

  On the other hand, the interior lamps 34-1 and 34-2 are provided at positions on the back side of the cameras 35-1 and 35-2 as is apparent from FIG. 7, and the interior lamps 34-1 and 34-2 are provided. The light from the interior lighting lamps 34-1 and 34-2 enters the cameras 35-1 and 35-2. It is possible to prevent the images of the cameras 35-1 and 35-2 from being deteriorated, and a good image can always be obtained. This can be achieved by providing a light-shielding portion on the camera side of the interior lighting lamps 35-1 and 35-2, or tilting the light source such as the LED constituting the interior lighting lamps 34-1 and 34-2 obliquely backward. The same effect can be obtained even if a configuration such as arrangement is adopted.

  The interior lighting lamps 34-1 and 34-2 have an illuminance in the storage of 150 lux to 200 lux. When photographing with the illuminance as it is, the food is whitish as described in the section of the photographing operation of FIG. It is difficult to see and distinguish. For this reason, in this embodiment, the illuminance is lowered to take a picture. This enables clear photographing even with a CCD camera or CMOS camera. The server controller 66 controls the illuminance of the interior lighting lamps 34-1 and 34-2 so that the illuminance is lower than normal only when the door 10 is closed or the door 10 is opened for photographing. By performing such control, it is possible to make clear photographing while keeping the illuminance as usual and making it easy to see the inside in the normal time when photographing is not performed. The decrease in illuminance may be caused by, for example, turning off a part of one or both of the interior lamps 34-1 and 34-2, or lighting both interior lamps 34-1 and 34-2. Various considerations such as lowering the illuminance of the lamps 34-1 and 34-2 are conceivable and may be appropriately selected.

  Moreover, the image in the storage 5 image | photographed with the said cameras 35-1 and 35-2 transmits the image of each camera 35-1 and 35-2 to the server apparatus 57, the server apparatus 57, the refrigerator 51, or The combination processing is performed by combining within the display terminal 15, but in this embodiment, the image combining processing is facilitated.

  That is, the cameras 35-1 and 35-2 are door inner surfaces corresponding to the front ends of the shelf boards 30-1 to 30-4, in this example, the upper shelf board 30-1 and the third shelf board 30-. 3 is provided on the inner surface of the door corresponding to the front end of the camera 3, so that the cameras 35-1 and 35-2 are photographed around the shelf boards 30-1 and 30-3 as indicated by the line Z in FIG. I will shoot.

  Accordingly, the image data in the storage 5 photographed by the two cameras 35-1 and 35-2 can be combined at the second shelf 30-2. That is, as shown in the “photographed raw image” in FIG. 27, the shelf 30-2 is commonly shown in the upper and lower images. After “image”, if the common shelf 30-2 moved to the upper and lower images is matched, a captured image of the entire storage is obtained. As a result, it is not necessary to perform extremely difficult correction processing to align and combine each food placed on the shelf board at the time of image combining processing, and image combining processing is easy. Become. Therefore, the processing program for image combining can be simplified, and the capacity of the control unit of the server device 57, the refrigerator 51, or the display terminal 15 for image combining processing is also required to be large. This can contribute to speeding up of processing speed.

  Further, since the two cameras 35-1 and 35-2 are provided substantially in front of the shelf plate 30-1 and the shelf plate 30-3 that cannot be adjusted in height, the shelf plate 30-that can be adjusted in height. Even when the height of 2 is changed, the image combining process can be easily performed by adjusting the range to be combined according to the height of the shelf 30-2 during the image combining process.

  INDUSTRIAL APPLICATION This invention can provide the refrigerator which can search the recipe relevant to the foodstuff in a storage only with an audio | voice, and its refrigerator system, and can be applied with a household refrigerator.

DESCRIPTION OF SYMBOLS 1 Refrigerator main body 2 Outer box 3 Inner box 4 Heat insulation material 5 Storage (refrigeration room)
6 switching room (storage)
7 Ice making room (storage)
8 Freezer room (storage)
9 Vegetable room (storage)
DESCRIPTION OF SYMBOLS 10 Door 10a Outer panel 10a-1 Glass plate 10a-2 Magic mirror 10b Inner plate 10b-1, 10b-2 Raised wall part 11 Center pillar 12a, 12b, 13, 14 Pull-out door 15 Display terminal 15a Display 16 Cooling room DESCRIPTION OF SYMBOLS 17 Cooler 18 Blower fan 19 Compressor 20 Radiation pipe 21 Capillary tube 22 Damper 23 Storage frame body 24 Terminal holding body 25 Touch panel 26 Gap 27 Frame body 28 Recess 29, 29a Smoke sheet 30-1 to 30-4 Shelf plate 31 Storage container 32-1 to 32-4 Side wall convex part 33-1 to 33-8 Door pocket 34-1 and 34-2 Interior lamp (illumination device)
35-1, 35-2, 35-3 Camera (photographing device)
41 Camera module 50 Refrigerator system 51 Refrigerator 53 Wireless adapter (communication device)
53a "Connect" button 54 Gateway device (relay device)
55 Router device 56 Internet 57 Server device 58 Interface (I / F)
59 Controller (control unit)
60 Storage Unit 61 Door Open / Close Detection Unit 62 Fan Drive Circuit 63 Compressor Drive Circuit 64 AC Power Supply 65 Server Interface 66 Server Control Unit 67 Data Storage Unit 70 “Direct” Selection Button 71 “Server” Selection Button 72 Image Data Selection Button 73 Energy Saving Related data selection button 74 Send button 100-1, 100-2 Microphone (voice recognition device)
101 Human sensor A Housing

In order to achieve the above object, the present invention achieves the above-described object, a storage room that is partitioned by a heat insulating wall and a heat insulating door and stores stored items, a heat insulating door that covers the storage room, a voice recognition unit that recognizes sound from the outside, and a voice recognition unit The communication means for transmitting the voice information to the external device, the display means for displaying the information received from the external device, and the search via the communication means based on the content displayed on the display means or the voice information by the voice recognition means a search unit for performing, with a motion sensor for sensing the presence or absence of a person in the yield Osameshitsu forward to operate the speech recognition means when detecting a within the human by the human sensor, to voice input the name of the food This makes it possible to easily retrieve recipes and the like by voice alone. By interlocking the or motion sensor, always because it is not necessary to operate the speech recognition means, also energy saving it is possible to prevent erroneous recognition.

An embodiment of a refrigerator according to the present invention includes a storage room partitioned by a heat insulating wall and a heat insulating door to store storage items, a heat insulating door that covers the storage room, a voice recognition unit that recognizes sound from the outside, and a voice recognition unit The communication means for transmitting the voice information to the external device, the display means for displaying the information received from the external device, and the search via the communication means based on the content displayed on the display means or the voice information by the voice recognition means a search unit for performing, with a motion sensor for sensing the presence or absence of a person in the yield Osameshitsu front, the display means has a function for operating the speech recognition means by issuing the voice recognition start instruction, the human sensor Thus, the voice recognition means is operated when the presence of a person is detected.

Thus, by operating the voice recognition unit when it is detected that there are people in the motion sensor, the name of the food becomes possible to voice input, the recipe search like only audio input from a database of the cloud is Easy to do. By interlocking the or motion sensor, always because it is not necessary to operate the speech recognition means, also energy saving it is possible to prevent erroneous recognition.

Claims (4)

A storage room that is partitioned by a heat insulating wall and a heat insulating door to store storage items, a heat insulating door that covers the storage room, a voice recognition unit that recognizes sound from the outside, and voice information by the voice recognition unit is transmitted to an external device. Communication means, display means for displaying information received from an external device, search means for performing a search via the communication means based on content displayed on the display means or voice information by the voice recognition means, A door opening / closing detection means for detecting the opening / closing of the door, and a human sensor for detecting the presence / absence of a person in front of the storage room, and detecting the opening of the heat insulating door by the door opening / closing detection means, or A refrigerator characterized in that the voice recognition means is operated when the presence of a person is detected by a human sensor. 2. The control unit according to claim 1, wherein the control unit terminates the operation of the voice recognition unit when the door opening / closing detection unit detects the closing of the heat insulating door and after a predetermined time elapses. Refrigerator. The refrigerator according to claim 1, wherein the voice recognition unit is arranged on the heat insulating door and the heat insulating wall and switches voice recognition data to be employed according to a voice input level of each voice recognition unit. . The refrigerator according to claim 1 or 2, wherein the display device changes a display position according to a height of a user based on a detection result of the human sensor.

Images