JP7386407B2 - system, refrigerator - Google Patents

Description

TECHNICAL FIELD The present invention relates to a system and a refrigerator that can search for recipes related to foods in storage using only voice.

In general, many refrigerators and refrigeration systems of this type have been proposed. For example, in the refrigerator and refrigeration system described in Patent Document 1, a small camera is installed in the storage, and the food storage situation in the storage, which is photographed by the small camera, is displayed on a display terminal such as a mobile phone via an Internet line. It is configured so that you can check it.

Further, Patent Document 2 also discloses a refrigerator and refrigeration system similar to Patent Document 1, and the one described in Patent Document 1 uses a small camera to photograph the state of food inside the storage when the refrigerator door is closed. It is configured to do so.

Furthermore, in addition to the configuration described in Patent Document 1, the refrigerator and refrigeration system described in Patent Document 3 further includes a display terminal on the surface of the refrigerator, and this display terminal is combined with a fixed terminal built into the surface of the refrigerator. It consists of a display terminal such as a mobile phone that can be removed from the surface and carried around, and both terminals are capable of displaying the food storage situation in the storage room as photographed with a small camera via an Internet connection, so that the display can display the food storage status in the storage room when the user is out and about. Of course, the structure is such that even when you are at home, you can check the food storage status in the refrigerator without opening the refrigerator door.

Japanese Patent Application Publication No. 2002-236798 Japanese Patent Application Publication No. 2007-46833 Japanese Patent Application Publication No. 2002-81818

An object of the present invention is to provide a system and a refrigerator that can search for recipes .

The system of the present invention is a system including a refrigerator and a display terminal, in which the refrigerator includes a refrigerator compartment, a door provided in the refrigerator compartment, and a first a second voice recognition means for recognizing the voice, which is provided on the side surface of the inside of the refrigerator compartment, a door opening/closing detection means for detecting the opening/closing state of the door, and a control means. When the open state of the door is detected by the door opening/closing detection means, the control means activates the first voice recognition means and the second voice recognition means, and activates the first voice recognition means and the second voice recognition means. When the voice is recognized by the voice recognition means, the control means selects one of the first voice data recognized by the first voice recognition means and the second voice data recognized by the second voice recognition means. After determining the audio data to be adopted based on the input level and determining the audio data by the control means, the audio data is displayed on the display terminal, and a confirmation is made to determine whether or not to execute a recipe search using the audio data. The present invention is characterized in that a means is provided, and when it is determined by the confirmation means to execute a recipe search using voice data, the result of the recipe search is displayed on the display terminal.
Further, the refrigerator of the present invention includes a refrigerating compartment, a door provided in the refrigerating compartment, a first voice recognition means for recognizing voice provided in the front of the door, and a side surface inside the refrigerating compartment. a second voice recognition means for recognizing voice, a door opening/closing detection means for detecting the opening/closing state of the door, a control means, and a display means, which are provided in the door opening/closing detection means. When the open state of the door is detected, the control means activates the first voice recognition means and the second voice recognition means, and when the voice is recognized by the first voice recognition means and the second voice recognition means. , the control means selects the voice data to be adopted based on the voice input level from among the first voice data recognized by the first voice recognition means and the second voice data recognized by the second voice recognition means. After the voice data is determined by the control means, the voice data is displayed on the display means, and a confirmation means is provided for determining whether or not to execute a recipe search using the voice data. If it is determined to execute a recipe search using , the recipe search results are displayed on the display means.

According to the present invention, it is possible to provide a system and a refrigerator that can search for recipes .

A perspective view of a refrigerator according to an embodiment of the present invention Schematic cross-sectional view of the refrigerator A perspective view showing the inside of the refrigerator when the storage door is opened. (a) A perspective view in which the refrigerator is integrally equipped with a display terminal, (b) A perspective view in which the refrigerator is detachably equipped with a display terminal. An example of the installation configuration of the display terminal used in the refrigerator is shown. (a) is a sectional view when the display terminal is integrally attached, and (b) is a sectional view when the display terminal is detachably attached. An example of the display visibility configuration of the display terminal used in the refrigerator is shown. (a) and (b) are schematic cross-sectional views when the outer cover panel of the door is made of glass, and (c) is a schematic cross-sectional view when the outer cover panel of the door is made of a one-way mirror. Schematic cross-sectional view when Vertical cross-sectional view showing an example of camera installation on the same refrigerator An example of camera installation on the same refrigerator is shown. (a) is a cross-sectional view of the refrigerator door with the door closed, and (b) is a front view of the inside of the door when the door is open. (a) (b) (c) Vertical cross-sectional views showing other examples of camera installation on the same refrigerator An explanatory diagram showing an example of a camera attached to the refrigerator System configuration diagram of the refrigerator system in the same embodiment Block diagram showing the refrigerator configuration of the refrigerator system Block diagram showing the server configuration of the refrigerator system A flowchart showing the photographing and data storage operations of the refrigerator system when the door is closed. A flowchart showing the photographing and data storage operations during the refrigerator cooling operation of the refrigerator system. A flowchart showing the operation from photographing to data storage when the door of the refrigerator system is opened. Flowchart showing other operation examples of the photographing operation in FIGS. 14 to 16 Flowchart showing the case where data storage and synthesis operations in the refrigerator system are performed by a server device Flowchart showing the case where data storage and synthesis operations in the refrigerator system are performed on the refrigerator side Flowchart showing the case where data storage and composition operations in the refrigerator system are performed on the display terminal side Flowchart showing normal image data transmission and reception in the refrigerator system A flowchart showing the transmission and reception of image data when image data is stored on the display terminal in the same refrigerator system. A flowchart showing the case of switching and transmitting and receiving individual images and composite images in the same refrigerator system. Flowchart showing image data transmission and reception at the time of advance transmission request in the refrigerator system Front view of a display terminal for explaining switching of transmission and reception paths in the refrigerator system Diagram for explaining image composition processing in the refrigerator system A perspective view of a refrigerator according to an embodiment of the present invention A perspective view showing the inside of the refrigerator when the storage door is opened. A block diagram showing the refrigerator configuration related to voice recognition of the refrigerator system. A flowchart showing voice recognition when the door is opened to data storage operation regarding the voice recognition of the refrigerator system. Flowchart showing voice recognition to data storage operation when a human sensor is present regarding voice recognition of the refrigerator system A flowchart showing the voice recognition stop operation when the door is closed regarding the voice recognition of the refrigerator system. A flowchart showing the operation of stopping voice recognition when the human sensor does not work regarding the voice recognition of the refrigerator system. A flowchart showing a case where the server device performs recipe search processing in voice recognition of the refrigerator system. Flowchart showing the case where the recipe search process in voice recognition of the refrigerator system is performed on the display terminal side A flowchart showing the case where recipe search results are displayed on a display terminal using voice recognition of the refrigerator system.

Hereinafter, specific and detailed embodiments of the present invention will be described, but first, the main points of the embodiments of the present invention will be described.

An embodiment of the refrigerator according to the present invention includes a storage chamber partitioned by a heat insulating wall and a heat insulating door to store stored items, a heat insulating door covering the storage room, a voice recognition means for recognizing voice from the outside, and a voice recognition means. a communication means for transmitting voice information to an external device; a display means for displaying information received from the external device; and a search means for performing a search via the communication means based on the content displayed on the display means or the voice information from the voice recognition means. The display means has a function of operating the voice recognition means by issuing a voice recognition activation instruction, and the display means has a function of operating the voice recognition means by issuing a voice recognition activation instruction. A voice recognition means is operated when the presence of a person is detected, and the voice recognition means are arranged on the front of the heat insulating door and the side of the storage chamber, and are adopted according to the voice input level of each voice recognition means. The configuration is such that the voice recognition data can be switched.

As a result, when the presence of a person is detected by the human sensor, the voice recognition means is activated and the name of the food can be input by voice, and recipes can be searched from a database on the cloud just by inputting the voice. It's easy to do. Furthermore, by linking with the human sensor, it is not necessary to constantly operate the voice recognition means, which prevents erroneous recognition and also saves energy.

Embodiment 1 of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to the following embodiments.

[Embodiment 1]
<1. Configuration>
1-1. Configuration of refrigerator 1-1-1. Configuration of French Door Type Refrigerator An example of the configuration of a French door type refrigerator will first be described using FIGS. 1 to 9.

In FIGS. 1 to 3, a refrigerator main body 1 includes an outer box 2 made of metal (for example, iron plate) that opens at the front, an inner box 3 made of hard resin (for example, ABS), and these outer box 2 and inner box 3. It consists of a heat insulating material 4 such as hard foamed urethane filled with foam between the holes. The refrigerator main body 1 has a plurality of partitioned storage compartments formed therein. This storage is located from the top of the refrigerator main body 1 to a refrigerating compartment 5, a switching compartment 6 located below the refrigerating compartment 5, an ice making compartment 7 arranged in parallel to the switching compartment 6, and a lower part of the switching compartment 6 and the ice making compartment 7. There is a freezer compartment 8 and a vegetable compartment 9 located at the bottom of the freezer compartment 8. The front of the refrigerator compartment 5 can be freely opened and closed by double doors 10-1 and 10-2, and the front of the switching compartment 6, ice-making compartment 7, freezer compartment 8, and vegetable compartment 9 can be closed with pull-out doors. It is closed by doors 12a, 12b, 13, and 14 (hereinafter referred to as pull-out doors) that can be freely opened and closed.

There is a cooling compartment 16 on the back of the refrigerator body 1, which includes a cooler 17 that generates cold air, a blower fan 18 that supplies cold air to each compartment, and a damper (air volume adjustment means) that adjusts the air volume to the refrigerator compartment. 22 is provided. Further, a compressor 19 is provided at the back of the top surface of the refrigerator main body 1, and a compressor 19 is installed in order to connect a condenser (not shown), a heat radiation pipe 20 for heat radiation, a capillary tube 21, and the above-mentioned cooler 17. A refrigerant is sealed in a refrigeration cycle connected in an annular manner to perform a cooling operation.

Here, the doors 10-1, 10-2, 12a, 12b, 13, and 14 are filled with hard urethane foam to provide insulation, similar to the refrigerator main body 1, and the door 10 of the storage 5 serving as the refrigerating room -1 and 10-2, one door 10-1 is narrower in width than the other door 10-2, and a center pillar 11 for preventing cold air leakage is attached to its open end. In this refrigerator, a display terminal 15 is installed on the front side of the other wide door 10-2. This display terminal 15 is composed of a display section such as a liquid crystal panel and a control section for controlling it, as shown in FIG. 4(a), and can be integrally attached to the door 10, or as shown in FIG. It is formed using a smartphone or a tablet PC, and is removably attached to the door 10, and can be installed in any of the ways.

5(a) and 5(b) show examples of mounting configurations of the display terminal 15 on the door 10. FIG. 5(a) shows the case where the display terminal 15 is attached integrally, and FIG. The case is shown when it is detachably attached.

In FIGS. 5A and 5B, the door 10 mainly includes an outer cover panel 10a facing the front side of the refrigerator, an inner plate 10b facing the inside of the refrigerator, an outer cover panel 10a and an inner plate. 10b and a heat insulating material 4 filled between the two. The outer cover panel 10a is made of a metal member such as a colored steel plate or a transparent member such as a glass plate, and the inner plate 10b is a synthetic resin member formed by vacuum forming. A raised wall portion 10b-1 protruding toward the inside of the refrigerator is provided in a peripheral portion of the inner side plate 10b in a square shape when viewed from the inside of the refrigerator. Further, as described above, the heat insulating material 4 is made of, for example, hard urethane foam.

Furthermore, in the door 10 of FIG. 5(a) in which the display terminal 15 is integrated, a resin storage frame 23 is provided on the front side of the door 10 and continues with the outer covering panel 10a and covers the heat insulating material 4. (If the outer cover panel 10a is a glass plate, it is provided by being integrally adhered to the glass plate). The display terminal 15 is fixed to the storage frame 23 via a terminal holder 24. Further, a touch panel 25 is provided on the front side of the display terminal 15. A gap 26 is provided between the display terminal 15 and the touch panel 25 to prevent the touch panel 25 from malfunctioning due to noise from the display terminal 15. The display terminal 15 and the touch panel 25 are fixed by attaching a frame 27 covering their outer peripheries to the surface of the door 10. In this way, by providing the storage frame 23, the heat insulating material 4 does not come into direct contact with the display terminal 15, so when foaming the heat insulating material 4, the display terminal 15 is not exposed to high temperature or deformed. It is possible to prevent problems such as On the other hand, in the door 10 shown in FIG. 5B in which the display terminal 15 is detachable, a storage frame 23 is provided on the front surface of the door 10 to form a recess 28, and the display terminal 15 can be attached and detached in the recess 28. It is configured to do so.

6(a), (b), and (c) show an example of a transparent display configuration of the door 10 and the display terminal 15, and in FIGS. 6(a) and (b), the outer cover panel 10a of the door 10 is made of glass plate. 10a-1, FIG. 3(c) shows a case where the outer cover panel 10a of the door 10 is configured with a one-way mirror 10a-2.

In FIG. 6(a), a smoked sheet 29 is provided between the display terminal 15 embedded in the door 10 and the glass plate 10a-1, with only the portion facing the display terminal 15 being transparent. Even when the glass plate 10a-1 other than the display terminal 15 looks black due to the smoke sheet 29, the boundary with the display terminal 15 is not noticeable, and a good appearance can be maintained. In addition, in FIG. 6(b), a transparent liquid crystal display section is used as the display section of the display terminal 15, and an entirely black smoked sheet 29a is laid along the back surface of the transparent liquid crystal display section. can be made good. Furthermore, in FIG. 6(c), the glass plate 10a-1 in FIGS. 6(a) and 6(b) is replaced with a magic mirror 10a-2, eliminating the need for the smoked sheet 29, which also maintains a good appearance. be able to. In each example, the display terminal 15 is integrally attached to the door 10. Note that instead of the smoke sheet 29, the back side of the glass plate 10a-1 may be painted so as to appear black.

Figures 7, 8(a) and 8(b) show an example of how the camera and interior illumination light, which are the main parts of the present invention, are installed, and Figure 4 is a vertical cross-sectional view of the refrigerator with the door closed. 8(a) is a cross-sectional view showing the inner surface of the door of the refrigerator when the door is closed, and FIG. 8(b) is a front view of the inner surface of the door when the door is opened.

In FIGS. 7, 8(a) and 8(b), the refrigerator storage 5 is provided with shelf boards 30-1 to 30-4 in multiple upper and lower stages. The shelf boards 30-1 to 30-4, except for the surrounding frames, are made of a transparent material such as transparent synthetic resin or glass.

Reference numeral 31 denotes a storage container provided below the lowest shelf board 30-4 so as to be freely drawn out. Although the lowermost shelf board 30-4 constitutes the upper surface of the storage container 31 and is therefore fixed, the shelf boards 30-1 to 30-3 are connected to side wall protrusions provided on both side walls of the storage container 5, respectively. It is mounted on 32-1 to 32-4 and is removable. In particular, a plurality of corresponding side wall convex portions 32-2 are provided at different heights on the central shelf board 30-2 among the shelf boards 30-1 to 30-3 excluding the lowest shelf board 30-4. The height of the shelf can be adjusted.

Reference numerals 33-1 to 33-3 are door pockets provided on the inner surface of the door 10 for opening and closing the opening of the storage 5, and are provided in a plurality of upper and lower stages. These door pockets 33-1 to 33-3 are also made of a transparent material.

Reference numerals 34-1 and 34-2 are lighting devices (hereinafter referred to as interior lighting lights) serving as interior lighting lights, which are located at the front opening of the storage 5 in front of the front ends of the shelf boards 30-1 to 30-4. They are provided on both side walls and the upper wall of the front opening. In order to save energy, the interior lighting lights 34-1 and 34-2 are provided with a board on which a plurality of LEDs are arranged, for example, in the vertical direction and covered with a light-transmitting cover.

Numerals 35-1 and 35-2 are photographing devices (hereinafter referred to as cameras) such as cameras installed at the front side of the storage 5, similar to the interior illumination lights 34-1 and 34-2. It is composed of a CCD camera, a CMOS camera, etc., and the type thereof is not particularly limited, but it is preferable that it is resistant to dew condensation and the like.

The cameras 35-1 and 35-2 are provided on the inner surface of the wider door 10-2 that does not have the center pillar 11 among the doors 10-1 and 10-2 for the storage 5, It is provided near the open end of the door 10-2 so as to be located approximately in the center of the refrigerator 51.

In this embodiment, one camera 35-1 is attached to the lower surface of the upper door pocket 33-1 at the approximate center in the front-rear direction, and the other camera 35-2 is attached near the top of the lowermost door pocket 33-3. be. Here, the front-rear direction of the door pocket 33 is a direction perpendicular to the surface formed by the inner plate 10b of the door 10, and is the same direction as the front-rear direction of the storage 5 when the door 10 is closed. .

Further, the upper camera 35-1 is located approximately in front of the top shelf board 30-1, and the lower camera 35-2 is located in front of the third shelf board 30-3 in the figure. . In other words, each camera is not a shelf board 30-2 that is placed on a plurality of side wall protrusions 32-2 that is provided with the premise of height adjustment, but a shelf that is removable but not premised on height adjustment. It is provided so as to be located approximately in front of the board 30-1 and the shelf board 30-3.

As described above, both of the cameras 35-1 and 35-2 are installed on the door 10-2 side where the center pillar 11 (see FIG. 3) is not provided, and are positioned approximately at the center of the refrigerator 51. and, as is clear from FIG. 7, at a position forward from the front end of each shelf board 30-1 to 30-4, and as shown in FIG. is located near the opposite open end side end Y, that is, approximately at the center in the left-right direction of the refrigerator 51, so that the two cameras 35-1 and 35-2 can photograph almost the entire area inside the refrigerator. It is set as.

Note that the cameras 35-1 and 35-2 may be replaced by an additional camera 35-3 dedicated to the interior of the storage container as shown in FIG. -2 so that it can move up and down, or the lower camera 35-2 can be set up so that it can rotate up and down, as shown in Figure (c), depending on the type of refrigerator to be installed. good.

Further, as is clear from FIG. 7, the interior illumination lights 34-1, 34-2 are provided at positions on the back side of the storage warehouse from the cameras 35-1, 35-2, and The light from the camera 35-1 and the camera 35-2 are provided so that the light does not directly enter the photographing sections of the cameras 35-1 and 35-2.

More specifically, as shown in FIG. 8(a), the imaging units of the cameras 35-1 and 35-2 detect the amount of irradiation (the maximum amount of irradiation) in the irradiation direction of the interior illumination lights 34-1 and 34-2. ), which is outside the range where the irradiation angle is 60 degrees in this example.

Although not shown, this may be done by providing a light shielding part on the camera side of the interior lighting lights 34-1, 34-2, or by providing a light source such as an LED that constitutes the interior lighting lights 34-1, 34-2. The light from the interior illumination lights 34-1 and 34-2 may be prevented from directly entering the photographing section of the cameras 35-1 and 35-2 by arranging them diagonally backward. This can be selected appropriately depending on the camera installation conditions, etc.

1-1-3. Camera Configuration FIG. 10 shows an example of a camera used for photographing inside the refrigerator. When photographing inside the refrigerator, it is most preferable to install the cameras in vertically separated positions as explained in each example above, but even if the upper and lower cameras 35-1 and 35-2 are installed close together as shown in FIG. The verification results confirmed that it was possible to photograph the entire area within the area.

Therefore, in such a case, it is conceivable to integrate the cameras 35-1 and 35-2 into the camera module 41. By modularizing in this way, installation can be facilitated and costs can be reduced. It is also possible to create a module by using a wider-angle lens such as a fisheye lens.

1-1-4. Microphone Configuration FIGS. 27 and 28 show an example of a microphone used for voice recognition. Since voice recognition may be performed by opening the refrigerator door or voice recognition may be performed without opening the refrigerator door, the microphones are configured to be attached to the front surface 100-1 of the refrigerator and the side surface 100-2 inside the refrigerator. This makes it possible to increase the voice input level during voice recognition.

1-1-5. Human Sensor Configuration FIG. 27 shows an example of a human sensor used for voice recognition. Since voice recognition may be performed without opening the refrigerator door, a human sensor 101 is attached to the front of the refrigerator. This eliminates the need for starting/stopping the voice recognition device.

1-2. Overall Configuration of Refrigerator System Next, the entire refrigerator system using the refrigerators as described above will be explained using FIGS. 11 to 26.

FIG. 11 shows the overall configuration of a refrigerator system 50 using the above-described refrigerator. As shown in FIG. 11, this refrigerator system 50 is a system for checking the food status in the refrigerator 51 using the display terminal 15 or the like.

This refrigerator system 50 includes a refrigerator 51 and a display terminal 15, and further includes a wireless adapter (communication device) 53, a gateway device (relay device) 54, a router device 55, the Internet 56, and a server device 57.

The refrigerator 51 is placed in the user's residence A, for example, in the kitchen room of the residence A.

The wireless adapter (communication device) 53 is electrically connected to a control section of the refrigerator 51, which will be described later, and is configured to communicate with a router device 55 via a gateway device (relay device) 54. Wireless adapter 53 receives a signal transmitted from router device 55 and outputs the received signal to the control unit of refrigerator 51. Based on this signal, the refrigerator 51 is configured to perform a corresponding action.

The wireless adapter 53 also acquires identification information (for example, serial number, model number, etc.) of the refrigerator 51 and internal image data, which will be described later, from the storage section of the refrigerator 51, and transmits the information to the router via the gateway device (relay device) 54. and configured to transmit to device 55. Note that the wireless adapter 53 may be provided integrally with the refrigerator 51 or may be configured to be detachable.

The wireless adapter 53 includes a "connect" button 53a. The "Connect" button 53a is for newly connecting to the gateway device (relay device) 54. When the "connect" button 53a is operated by the user, the wireless adapter 53 is configured to acquire identification information (eg, serial number and model number) of the refrigerator 51 from the storage section of the refrigerator 51.

As described above, the display terminal 15 is, for example, a general-purpose portable terminal such as a smartphone or a tablet PC (personal computer), and in the case of this embodiment, it is detachable from the door 10 of the refrigerator 51 mentioned above. It is equipped with a function (means) for connecting to the Internet 56 and a function (means) for communicating with a gateway device 54, which will be described later.

The display terminal 15 (for example, a smartphone) is configured to connect to the Internet 56 via a telephone network (for example, a 3G network). Additionally or alternatively, for example, the display terminal 15 is configured to connect to the Internet 56 via a public wireless LAN with a Wi-Fi communication function.

By connecting to the Internet 56, for example, the display terminal 15 can run a program that acquires image data of food in the refrigerator and data for energy-saving operation (for example, data such as the number of times the door is opened and closed), which is the object of the present invention. can be obtained from the homepage of the manufacturer of the refrigerator 51 via the Internet 56. By installing this acquired program on the display terminal 15 and activating the installed program, the display terminal 15 can acquire image data of food in the refrigerator, etc. That is, the display terminal 15 can generate and transmit an operation signal for acquiring image data of the refrigerator 51 by operating the operation screen of the display 15a. Details will be described later.

The display terminal 15 also connects to a router device 55 via general-purpose communication that does not go through the Internet 56, such as Wi-Fi communication, Bluetooth (registered trademark) communication, or infrared communication, and connects to the gateway device 54 via the router device 55. is configured to have a communication connection to. A device for this purpose (for example, a Wi-Fi antenna, etc.) is built into the display terminal 15.

The gateway device (relay device) 54 is a device that relays communication between the wireless adapter 53 and the display terminal 15. For example, the gateway device (relay device) 54 is a device that relays communication between the wireless adapter 53 and the display terminal 15. ) is configured to communicate using signals. The frequency band for communication between the gateway device 54 and the wireless adapter 53 is preferably a low frequency band that can reach a long distance, and is installed in the user's residence A where the router device 55 and the refrigerator 51 are installed.

The gateway device 54 is also configured to communicate with the display terminal 15 via the router device 55. That is, the gateway device 54 is configured so that communication (first communication) can be performed only through the router device 55 without going through the display terminal 15 and the Internet 56.

The gateway device 54 is further configured to be connectable to the Internet 56 via a router device 55. Thereby, the gateway device 54 can communicate with the display terminal 15 (second communication) via the Internet 56 and the router device 55, and can also communicate with the server device 57.

Therefore, the gateway device 54 communicates with the display terminal 15 through communication not via the Internet 56 (i.e., first communication via only the router device 55) and communication via the Internet 56 (i.e., via the Internet 56 and the router device 55). (second communication). The reason for this configuration will be described later.

The server device 57 is prepared, for example, by the manufacturer of the refrigerator 51. The server device 57 manages information necessary for access and authentication regarding the display terminal 15, refrigerator 51, and wireless adapter 53. For example, the server device 57 determines whether the identification information of the display terminal 15 included in the communication request signal matches the identification information of the display terminal 15 stored (registered) in association with the wireless adapter 53. Authenticate (identify). If these match, the server device 57 authenticates the accessing display terminal 15 as an authorized display terminal, and permits communication with the refrigerator 51 in residence A.

Further, the server device 57 stores the image data of the food in the refrigerator, the data for energy-saving operation, etc. transmitted from the refrigerator 51, as described above. These data are transmitted to the display terminal 15 based on a data call signal from the display terminal 15. That is, by operating the display terminal 15, the user can display and check the food image data in the refrigerator on the display.

1-3. System Section Configuration of Refrigerator FIG. 12 shows the system section structure of the refrigerator used in the above-mentioned refrigeration system.

In FIG. 12, this refrigerator 51 includes an interface (I/F) 58, a control section 59, a storage section 60, a door opening/closing detection section 61, interior lighting lights 34-1 and 34-2, and cameras 35-1 and 35. -2, a fan drive circuit 62, a compressor drive circuit 63, a blower fan 18, and a compressor 19, and is supplied with power from an AC power source 64 such as a commercial power source in residence A, for example.

An interface (I/F) 58 exchanges data and the like between the wireless adapter 53 and the control unit 59 of the refrigerator 51.

The control unit 59 controls the operation of the refrigerator. The control unit 59 also controls the cameras 35-1, 35-2 and the interior lighting lights 34-1, 34-2 to photograph the food inside the refrigerator, and also stores this image data in the storage unit 60 once. It is stored and transmitted to a server device 57 on the Internet 56. Furthermore, if there is a request signal for pre-transmission of image data from the display terminal 15 via the server device 57, an instruction is given to the display terminal 15 to execute the transmission via the server device 57 immediately after photographing the food. The details will be described later.

The storage unit 60 rewrites and stores, in addition to identification symbols and control programs, image data of food in the refrigerator photographed by the cameras 35-1 and 35-2, and data for energy-saving operation.

The interior lighting lights 34-1 and 34-2 are controlled to turn on and off by the control unit 59 based on the signal from the door opening/closing detection unit 61, and are turned on when the door 10 is opened and for several seconds when the door 10 is closed. The light goes out after a delay time of .

The cameras 35-1 and 35-2 are controlled by the control section 59 based on the signal from the door opening/closing detection section 61, and photograph the food inside the refrigerator when the door 10 is closed. The details will be described later.

The fan drive circuit 62 receives a control signal from the control unit 59 and controls the rotation of the blower fan 18 .

The compressor drive circuit 63 receives a control signal from the control unit 59 and controls the drive of the compressor 19, such as the rotation speed.

The blower fan 18 operates under the control of the fan drive circuit 62 to generate an airflow for circulating cool air.

Compressor 19 compresses refrigerant (not shown) circulating within refrigerator 51 under the control of compressor drive circuit 63 .

Further, FIG. 29 shows a system configuration of a refrigerator related to voice recognition used in the above-mentioned refrigeration system.

In FIG. 29, this refrigerator 51 includes an interface (I/F) 58, a control section 59, a storage section 60, a door opening/closing detection section 61, interior lighting lights 34-1 and 34-2, and microphones 100-1 and 100. -2, a fan drive circuit 62, a compressor drive circuit 63, a blower fan 18, and a compressor 19, and is supplied with power from an AC power source 64 such as a commercial power source in residence A, for example.

An interface (I/F) 58 exchanges data and the like between the wireless adapter 53 and the control unit 59 of the refrigerator 51.

The control unit 59 controls the operation of the refrigerator. The control unit 59 also controls the microphones 100-1 and 100-2 to recognize external voices, temporarily stores this voice recognition data in the storage unit 60, and also controls the microphones 100-1 and 100-2 to Send to. Furthermore, if there is a request signal for pre-transmission of image data from the display terminal 15 via the server device 57, an instruction is given to the display terminal 15 to execute the transmission via the server device 57 immediately after photographing the food. The details will be described later.

The storage unit 60 rewrites and stores data recognized by voice using the microphones 100-1 and 100-2 and data for energy-saving operation, in addition to identification symbols and control programs.

The interior lighting lights 34-1 and 34-2 are controlled to turn on and off by the control unit 59 based on the signal from the door opening/closing detection unit 61, and are turned on when the door 10 is opened and for several seconds when the door 10 is closed. The light goes out after a delay time of .

The microphones 100-1 and 100-2 are controlled by the control unit 59 based on the signal from the door opening/closing detection unit 61, and when the door 10 is opened, the voice recognition device is activated, and when the door 10 is closed, a certain period of time has elapsed. Then stop the speech recognition device. The details will be described later.

The fan drive circuit 62 receives a control signal from the control unit 59 and controls the rotation of the blower fan 18 .

The compressor drive circuit 63 receives a control signal from the control unit 59 and controls the drive of the compressor 19, such as the rotation speed.

The blower fan 18 operates under the control of the fan drive circuit 62 to generate an airflow for circulating cool air.

Compressor 19 compresses refrigerant (not shown) circulating within refrigerator 51 under the control of compressor drive circuit 63 .

1-4. Configuration of Server Apparatus Next, a configuration example of the server apparatus of the refrigerator system in Embodiment 1 will be described using FIG. 13. Here, the configuration of the server device 57 in FIG. 11 will be taken as an example.

As shown in FIG. 13, this server device 57 includes a server interface 65, a server control section 66, and a data storage section 67.

The server interface 65 is a communication unit that exchanges data between the display terminal 15 and the router device 55 and the server device 57 via the Internet 56 under the control of the server control unit 66 .

The server control unit 66 controls the overall operation of this server device 57. The server control unit 66 also combines and processes the food image data transmitted from the refrigerator 51 and stores it in the data storage unit 67, and when there is a signal requesting advance transmission of image data from the display terminal 15, the data storage unit The latest image data of the image data stored in 67 is transmitted to the display terminal 15 each time the image data is transmitted from the refrigerator 51, and the display terminal 15 is rewritten and updated. Of course, if a data call signal is transmitted from the display terminal 15 without a signal requesting advance transmission of image data, the latest image data stored in the data storage section 67 is transmitted to the display terminal 15.

The data storage unit 67 stores necessary data such as management programs and application programs for running the refrigerator system. Further, the data storage section 67 rewrites and stores the latest food image data transmitted from the refrigerator 51 under the control of the server control section 66. The data storage unit 67 is configured by, for example, an HDD (Hard Disc Drive) or an SSD (Solid State Drive).

Note that the server device 57 can be constructed as either a server configured with a single large computer or a cloud type server configured with a group of many computers.

<2. Operation>
Next, the operation of the refrigerator and refrigeration system having the above configuration will be explained. Note that in each of the following descriptions, the same operations will be described with the same step numbers added.

2-1. Food photography operations 2-1-1. Photographing when the door is closed FIG. 14 is a flowchart showing the operation when photographing the food inside the storage when the door 10 of the refrigerator 51 is closed.

When the door 10 is opened to take food in and out (Y in step 1), the interior lighting lights 34-1 and 34-2 are turned on (step 2), and when the door 10 is closed (Y in step 3), The illuminance of the interior lighting lights 34-1 and 34-2 is reduced (step 4), and after a while, for example, several seconds, the cameras 35-1 and 35-2 are activated to photograph the food inside the refrigerator (step 5). . Thereafter, the photographed image data is stored in the storage unit 60 of the refrigerator 51 (step 6), and the interior lighting lights 34-1 and 34-2 are turned off (step 7).

Here, it is preferable to lower the illuminance of the interior illumination lights 34-1 and 34-2 during the above-mentioned photographing, since the food can be photographed clearly. In other words, when the interior lighting lights 34-1 and 34-2 are turned on and the food is photographed, the reflected light is strongly reflected on the food from the entire circumference except for the opening of the storage, partly because the inside of the storage is white. (the illuminance remains high and bounces off the food without reducing the illuminance), making the food whitish and making it difficult to take clear photographs. However, by reducing the illumination of the interior lighting lights 34-1 and 34-2 as in the example above, the influence of reflected light from the inside of the storage chamber can be significantly reduced, making it possible to take clear images. It becomes.

2-1-2. Photographing during Refrigerator Cooling Operation FIG. 15 is a flowchart showing the operation when the food in the storage compartment is photographed when the refrigerator compartment is being cooled after the food in the refrigerator is taken out.
When the door 10 is opened to take food in and out (Y in step 1), the interior lights 34-1 and 34-2 are turned on (step 2), the door 10 is closed (Y in step 3), and the refrigerator compartment is opened. It is determined whether or not the cooling operation is in progress (step 8). If the blower fan 18 is ON and the damper 22 is open, and the refrigerator compartment is being cooled, the illuminance of the interior lights 34-1 and 34-2 is reduced (step 4), as in the case when the door is closed. ), the cameras 35-1 and 35-2 are activated to photograph the food inside the refrigerator (step 5). Thereafter, the photographed image data is stored in the storage unit 60 of the refrigerator 51 (step 6), and the interior lighting lights 34-1 and 34-2 are turned off (step 7).

In this way, by taking pictures after the refrigerator compartment cooling operation, for example, even if dew condensation forms on the surface of food that has been newly placed in the storage compartment due to the temperature difference inside and outside the storage compartment, this condensation will disappear due to the cooling operation. . Therefore, the images taken by the cameras 35-1 and 35-2 are so clear that the characters written on the food packaging container can be read, which is preferable.

In addition, during the above operation, when the door 10 is closed (Y in step 1), the interior lighting lights 34-1 and 34-2 are turned off once, and the interior lighting lights 34-1 and 34 are turned off again before photographing. -2 is turned on and its illuminance is reduced, it is possible to suppress wasteful power consumption caused by the interior lighting lights 34-1 and 34-2 continuing to be turned on.

2-1-3. Photographing when the door is opened FIG. 16 is a flowchart showing the operation in the case of a single door, in which the food inside the storage is photographed when the door 10 is opened at a predetermined angle.

When the door 10 is opened to take food in and out (Y in step 1), the interior lighting lights 34-1 and 34-2 are turned on (step 2), and the opening angle of the door is adjusted to a predetermined angle, for example, as shown in FIG. When the indicated opening angle of 30 degrees is reached (Y in step 10), the illuminance of the interior lighting lights 34-1 and 34-2 is reduced (step 4), the cameras 35-1 and 35-2 are activated, and the refrigerator is opened. Photograph the food inside (Step 5). After that, the photographed image data is stored in the storage unit 60 of the refrigerator 51 (step 6), and when the door is closed (Y in step 11), the interior lights 34-1 and 34-2 are turned off (step 7). .

If the photograph is taken when the door 10 is opened at a predetermined angle as in this example, it is possible to photograph a wide range of the inside of the storage room, as already mentioned, and the image becomes natural without a sense of harmony, which is preferable. It is.

In addition, when photographing when the door is opened at a predetermined angle, photographing is performed at least twice: when the door opens and when the door closes. At this time, by storing the last captured image data (during the door closing operation) in the storage unit 60 during the operation time from one door opening to closing, the storage unit 60 stores the latest image data of the storage unit that has finished loading and unloading food. Food conditions can be photographed.

It goes without saying that in step 10 of detecting whether or not the opening angle of the door 10 reaches a predetermined angle, if the door 10 is closed before reaching the predetermined angle, the interior illumination light 34-1 is turned off. , 34-2, and waits for the door 10 to be opened again.

The photographing operations have been explained above by illustrating typical examples, but in addition to the above-mentioned photographing operations, for example, photographing is performed at a fixed time or after a predetermined time has elapsed since the previous photographing, or when a photographing instruction is given from the display terminal 15. Photographing operations such as .

2-1-4. Lighting for Photographing Food lighting for photographing, as already explained in FIGS. 14, 15, and 16, is performed by turning on the interior lighting lights 34-1 and 34-2 and then reducing the illuminance. irradiate it and take a picture. In some cases, it may be possible to turn on only the interior illumination lights 34-1 on both sides of the storage or the interior illumination 34-2 on the top of the storage to reduce the illuminance and take pictures. The reason why this illuminance is lowered and photographed will be described later.

Further, as another example of food irradiation during photographing, the following may be considered. That is, FIG. 17 is a flowchart showing the states of turning on and off the interior illumination lights 34-1 and 34-2 during photographing by the cameras 35-1 and 35-2, and is similar to those shown in FIGS. 14, 15, and 16 described above. This corresponds to the operation indicated by the broken line.

The above-mentioned photographs in FIGS. 14, 15, and 16 are examples in which the interior lighting lights 34-1 and 34-2 are turned on and the illuminance is lowered. In this example, both sides of the storage The food is illuminated and photographed with one side of the interior illumination lights 34-1 on each side of the storage chamber turned off, or in other words, only one of the interior illumination lights 34-1 on both sides of the storage chamber is turned on. In this case, the interior illumination light 34-2 on the top surface of the storage chamber may be turned on or off, but either one may be selected as appropriate so that the illumination inside the storage chamber is optimal for photographing.

In FIG. 17, before the cameras 35-1 and 35-2 start photographing, one of the interior lighting lights 34-1 on both sides of the storage compartment that is already on, for example, the right interior lighting light, is turned off ( Step 12), the photograph of the food illuminated by the left interior lighting lamp is photographed (Step 13) and stored (Step 14). Thereafter, the right interior lighting lamp is turned on (step 15), then the left interior lighting light is turned off (step 16), and the food illuminated by the right interior lighting lamp is photographed (step 17). ). The image data is stored (step 18), and the right chamber illumination light is turned off (step 19). The image data stored by irradiating the food from one side in this way is combined into a single image and stored.

The reason why the interior lighting lights 34-1 and 34-2 are turned on one by one to photograph the food and combine them into one image is as follows. In other words, as already mentioned, when taking pictures, the illuminance of the interior lighting lights 34-1 and 34-2 is lowered, but if the reduction in illuminance is too large, the image will become dark and lack sharpness. Therefore, there is a limit to the reduction in illuminance. Therefore, a certain degree of illuminance is maintained, but even at that illuminance, the influence of reflected light may remain if the internal volume of the storage is small. However, as explained in this example, if you turn on one side of the interior lighting lamp 34-1 and photograph the food, the influence of reflected light can be reduced, and the interior lighting lamp 34-1 turns on. The side facing the camera can be photographed with moderate brightness. Therefore, it is preferable to combine the left and right images of the food photographed with appropriate brightness, as the entire image will be clear.

2-2. Storage and Combination of Images Next, storage and combination of image data taken by each of the above-described operations will be explained.

2-2-1- Saving and compositing on the server FIG. 18 shows a case where image data is saved and synthesized on the server device 57. When the interior of the refrigerator is photographed by the operations shown in FIGS. 14 to 16 including FIG. , the refrigerator 51 connects to the Internet 56 via the wireless adapter 53, gateway device 54, and router device 55 (step 20), and sends the individual image data taken and stored by each camera to the server device 57 connected to the Internet 56. Send (step 21). The image data transmitted via the Internet 56 is subjected to a compositing process (step 22) in which the individual image data are combined by the server control unit 66 of the server device 57. The combined image data is stored in the data storage section 67 of the server device 57 (step 23).

If images are combined and stored in the server device 57 as in this example, the load on the control unit 59 and storage unit 60 on the refrigerator side can be reduced, making it possible to create an Internet-compatible refrigerator at a low cost. can be provided.

2-2-2- Saving and compositing on the refrigerator side Figure 19 shows a case where image data is saved and composited on the refrigerator side. , the control unit 59 of the refrigerator 51 takes out individual image data from each camera stored in the storage unit 60, and combines and synthesizes them (step 24). The synthesized image data is stored (step 25), and the refrigerator 51 is connected to the Internet 56 via the wireless adapter 53, the gateway device 54, and the router device 55 (step 20), and the stored image data is connected to the Internet 56. The information is transmitted to the server device 57 (step 21). The composite image data transmitted via the Internet 56 is stored in the data storage section 67 of the server device 57 (step 23).

In this example, the load on the control unit 59 and storage unit 60 on the refrigerator side increases, but image data can be sent and received to and from the display terminal 15 without going through the data storage unit 67 of the server device 57, reducing system operating costs. It can be done.

2-2-3. Storage and Composition on the Display Terminal Side FIG. 20 shows a case where image data is stored and composited on the display terminal 15 side. When the interior of the refrigerator is photographed by the operations shown in FIGS. connects to the Internet 56 from the wireless adapter 53 via the gateway device 54 and router device 55 (step 20), and transmits image data to the display terminal 15 instead of the server (step 26). The image data transmitted via the Internet 56 is subjected to compositing processing (step 27) to combine the image data by the display terminal 15. The combined image data is stored in the data storage section of the display terminal 15 (step 28).

Although the load on the display terminal 15 increases in this example as well, the load on the control unit 59 and storage unit 60 on the refrigerator side can be reduced, and an Internet-compatible refrigerator can be provided at a low cost.

Although typical image data transmission and compositing operations have been described above, the storage and compositing operations do not necessarily have to be performed at the same location, such as the server device 57; This can be set as desired depending on the system, such as combining the image data with the server device 57 and storing the image data with the server device 57.

2-3. Normal transmission and reception of image data 2-3-1. Transmission and reception of image data stored in server and refrigerator FIG. 21 is a flowchart showing transmission and reception of image data stored in server device 57 or refrigerator 51 and display terminal 15, and shows transmission and reception of image data stored in server device 57 or refrigerator 51. be.

First, the user presses a server connection button (not shown) displayed on the display of the display terminal 15 at a shopping place, etc., and requests reading of image data stored in the data storage section 67 of the server device 57 ( Step 31). It is detected whether image data has been imported from the server device 57 (step 32), and if the image data is not transferred, the display terminal 15 connects to the Internet 56 (step 33). The image data stored in the data storage section 67 of the server device 57 or the storage section 60 of the refrigerator is retrieved (step 34), and the image data is transmitted via the Internet 56 to the display terminal 15, such as a smartphone, which is the data calling source. (step 35). It is detected whether image data has been imported from the server device 57 (step 32), and when the image data has been imported, the image data is output to the display of the display terminal 15 (step 36).
Thereafter, it is detected whether an enlargement button (not shown) displayed on the display of the display terminal 15 has been pressed (step 37), and when it is detected that the enlargement button has been pressed, the Image data is created by enlarging the image data by 1.1 times or more through digital conversion. The enlarged and converted image data is output to the display of the display terminal 15 (step 36).

Furthermore, after that, it is detected whether a numeric keypad button (not shown) displayed on the display has been pressed (step 38), and when it is detected that the numeric keypad button has been pressed, the enlarged image data of the storage interior outputted to the display is displayed. The image data of the location specified by the numeric keypad is output to the display (step 36). Furthermore, it is detected whether a numeric keypad button has been pressed (step 38). Terminates if not detected.

Note that the enlargement button and the numeric keypad button for displaying image data at a predetermined location may be of a type that is slid with a finger on the surface of the display, and such a configuration improves usability.

As described above, according to this refrigerator system, it is possible to check the contents of the refrigerator 51 at a glance at a shopping place, it is possible to prevent forgetting to buy food, and it is possible to eliminate the waste of purchasing extra ingredients. . In addition, in a household where both parents work, the user can check the inside of the storage compartment 5 of the refrigerator 51 just before returning home and purchase foodstuffs etc. on the way home, which saves time for the user.

In addition, by enlarging the display or displaying a predetermined location, even on a display terminal 15 that has become more compact like the display terminal 15, the image can be enlarged and displayed, thereby ensuring that foods that are difficult to confirm on a small display terminal 15 are displayed. You can see it.

2-3-2. Transmission and reception of image data stored in the display terminal FIG. 22 shows a case where image data is already stored in the display terminal 15. In this case, as described in FIG. 24, the image data is sent to the display terminal 15 and stored, and when a request to read this image data is made (step 31), the image data is imported from the storage section of the display terminal 15. The image data is output to the display of the display terminal 15 (step 36). The enlarging display operation and local display thereafter are as explained in FIG. 25.

2-3-3. Switching and receiving image data The images taken by the cameras 35-1 and 35-2 are the individual images taken by each camera 35-1 and 35-2 (upper half or lower half of the storage), and the images taken by each of these images. An example will be shown in which a composite image (an image composed of the interior of the storage room from top to bottom in one composite image) is stored and can be switched, transmitted and received, and displayed on the display terminal 15.

FIG. 23 is a flowchart showing the operation in such a case.

First, the user presses an individual image/composite image call-up button (not shown) displayed on the display of the display terminal 15 (step 41), and calls up either the individual image or the composite image (step 42). ). Thereafter, according to the flow shown in FIG. 25 or 26, image data of either an individual image or a composite image is captured and displayed on the display of the display terminal 15.

If the image displayed on the display terminal 15 can be switched to either an individual image or a composite image as in this example, the status of the food in the storage room can usually be confirmed at a glance by calling in and displaying the composite image. However, if it is difficult to identify the food because it is in the corner of the storage, etc., switching to individual images can be used to reliably identify the food, improving usability.

2-4. Transmission and reception when requesting advance transmission of image data, etc. FIG. 24 shows a flowchart when there is a request for advance transmission of images from the display terminal 15.

First, the user operates the display terminal 15 to request an advance request signal (step 51). The server device 57 or the refrigerator 51 receives this request signal (step 52), reads the image data (individual image data or composite image data, the same applies hereinafter) stored in the data storage section 67 or the storage section 60, and transmits it (step 53). The display terminal 15 detects whether image data has been imported from the server device 57 or the refrigerator 51 (step 54), and if no image data has been transmitted, the display terminal 15 connects to the Internet 56 (step 55). The server devices 57 take out the image data stored in the refrigerator 51 (step 56), and transmit the image data to the display terminal 15 via the Internet 56 (step 57). It is detected whether image data has been taken in from the server device 57 (step 54), and if image data is detected, it waits until the image data is called (N in step 58). If image data is called up (Y in step 58), the image data is output to the display of the display terminal 15 (step 59).

As a result, when image data is requested on the display terminal 15, even if access to the server device 57 is concentrated and transmission of the image data from the server device 57 is delayed, the image data is sent in real time at the same time as the image data request. You can check the image by displaying it on the display. Therefore, it is possible to avoid a situation where the user has to wait for the image data to be displayed at the shopping destination, and the shopping time can be shortened.

Further, if there is a request for advance image transmission from the display terminal 15, the server device 57 is set to transmit image data to the display terminal 15 each time the door 10 of the refrigerator 51 is closed, and If the user who owns the image data is set to leave it alone once it is sent, they will be able to know this each time the door is opened or closed. Therefore, for example, if the user opens and closes the door many times and is likely to increase power consumption, he or she can call home and talk to children and others to refrain from using the refrigerator 51, thereby reducing energy savings. Convenience will also improve.

2-5. Switching of Transmission/Reception Path for Image Data, etc. FIG. 25 shows a display on the display terminal 15 when a communication path is selected. The flow of image data transmission and reception has already been described, but the communication route is changed to speed up the acquisition of image data.

Specifically, the user can select whether to connect the display terminal 15 directly to the gateway device 54 via the router device 55 or to the server device 57 via the Internet 56.

In FIG. 25, 70 and 71 are communication route selection buttons displayed on the display of the display terminal 15, and 70 is a "direct" selection button for directly connecting the display terminal 15 to the refrigerator 51 via the router device 55 and the gateway device 54. Button 71 is a "server" selection button that connects to the server device 57 or refrigerator 51 via the Internet 56. Reference numerals 72 and 73 are data selection buttons for data to be called, 72 is a button for food image data, and 73 is a button for energy saving related data such as the number of times the door is opened and closed. 74 is a send button.

When the "direct" selection button 70 is selected, "direct" communication is set for the display terminal 15. When the user touches the send button 74 while "direct" communication is set, an image data request signal from the display terminal 15 is transmitted directly from the display terminal 15 to the gateway device (relay device) 54 via the router device 55. Sent. Gateway device 54 transmits the received data request signal to wireless adapter 53 of refrigerator 51. Upon receiving the image data request signal, the wireless adapter 53 transmits it to the control unit 59 of the refrigerator 51. When the control unit 59 of the refrigerator 51 receives the image data request signal, it reads the image data stored in the storage unit 60 and transmits the image data to the display terminal 15 via the wireless adapter 53 and router device 55.

Therefore, when the user is located in residence A as shown by the broken line in FIG. The image data can be directly acquired from the storage unit 60 of the refrigerator 51 by communicating with the gateway device 54 via the gateway device 54 .

As a result, if there is a delay in the transmission response of image data to the user's operation on the display terminal 15 due to line congestion due to concentration of accesses to the Internet 56, the user can By switching to "Direct" communication, image data can be acquired without being affected by line congestion on the Internet 56. Therefore, the user can display the image on the display of the display terminal 15 in real time and check the food situation in the storage 5 without waiting time.

On the other hand, if the user is located outside residence A, as shown by the solid line in FIG. By touching 71, the display terminal 15 communicates with the server device 57 via the Internet 56 and reads the image data of the refrigerator 51. That is, when the user is outside residence A, by touching the "server" selection button 71 and setting "server" communication, the user can obtain the image data of the refrigerator using the display terminal 15 even in a remote location, and display the food. You can know the storage status of.

In this way, in the refrigerator system of this embodiment, the user can select either "direct" communication or "server" communication, and if the user is in residence A, "direct" communication is possible. You can view images comfortably without being affected by internet connection congestion by selecting a communication option.

Although the refrigerator and refrigerator system of the present invention have been described above by exemplifying specific configurations, the present invention is not limited to the above configurations.

For example, in the embodiment described above, the gateway device 54 is interposed in the communication path between the display terminal 15, the wireless adapter 53 of the refrigerator 51, and the Internet 56, but the gateway device 54 is not necessarily necessary, and the router device 55 It may be configured to directly communicate with the display terminal 15, the wireless adapter 53 of the refrigerator 51, and the Internet 56 via the Internet 56, and by doing so, the system configuration can be simplified.

Furthermore, in the embodiment described above, when the user is outside the residence A, the image data is acquired from the server device 57. The image data stored in the storage section 60 of the refrigerator 51 may be directly fetched from the refrigerator 51 by communicating with the refrigerator 54 . In this case and when performing "direct" communication, it is preferable to perform image combination processing in the storage unit 60 of the refrigerator 51 and to transmit the image data after this combination processing.

Furthermore, in this embodiment, a case is illustrated in which a "direct" selection button 70 and a "server" selection button 71 are displayed on the display of the display terminal 15, and the user selects between "direct" communication and "server" communication. However, this can be done by providing the display terminal 15 with a function that automatically switches to "direct" communication when it catches the radio wave from the router device 55, so that "direct" communication is automatically selected, which improves usability. can be improved.

2-6. Speech recognition operation of food names 2-6-1. Voice Recognition Operation When Door Opens FIG. 30 is a flowchart showing the operation when the voice recognition device is activated when the door 10 of the refrigerator 51 is opened.

When the door 10 is opened to check the food (Y in step 101), the voice recognition devices 100-1 and 100-2 are activated (step 102), and the interior lighting lights 34-1 and 34-2 are turned on. (Step 103). When the voice is recognized by the voice recognition device (Y in step 104), the voice input level of the voice recognition devices 100-1 and 100-2 is checked through microphone discrimination processing to determine which voice recognition data to adopt. (Step 105), the voice data adopted by voice analysis processing is analyzed to determine the name of the input food (Step 106), and the determined voice data is stored in the storage unit 60 of the refrigerator 51 (Step 107). . If the voice recognition device does not recognize the voice (N at step 104), the state of the door 10 is checked, and if the door 10 is not closed, the voice from the voice recognition device is checked again (N at step 108). If the door 10 is closed (Y in step 108), the interior lights 34-1 and 34-2 are turned off (step 109), and after a while, for example, several seconds, the voice recognition devices 100-1 and 100-2 are turned off. (step 110).

Here, by interlocking the voice recognition device with the door 10, it is possible to start the voice recognition device when the door opens and stop the voice recognition device when the door closes, which is preferable. That is, since it is not necessary to constantly operate the voice recognition device, it is possible to prevent erroneous voice recognition and to realize energy savings.

2-6-2. Voice Recognition Operation Based on Presence of Human Sensor FIG. 31 is a flowchart showing the operation when the voice recognition device is activated when the presence of a person in front of the refrigerator 51 is detected.

When the human sensor 101 detects that a person is present in front of the refrigerator (Y in step 111), the voice recognition devices 100-1 and 100-2 are activated (step 102). When the voice is recognized by the voice recognition device (Y in step 104), the voice input level of the voice recognition devices 100-1 and 100-2 is checked through microphone discrimination processing to determine which voice recognition data to adopt. (Step 105), the voice data adopted by voice analysis processing is analyzed to determine the name of the input food (Step 106), and the determined voice data is stored in the storage unit 60 of the refrigerator 51 (Step 107). . If the voice recognition device does not recognize the voice (N in step 104), check the state of the human sensor 101, and if a person is detected, check the voice of the voice recognition device again (N in step 112). . If no person is detected (Y in step 112), the voice recognition devices 100-1 and 100-2 are stopped after a while, for example, several seconds (step 110).

Here, by interlocking the voice recognition device with the human sensor 101, it is possible to start the voice recognition device when a person is detected and to stop the voice recognition device when no person is detected. That is, since it is not necessary to constantly operate the voice recognition device, it is possible to prevent erroneous voice recognition and to realize energy savings.

Although typical voice recognition operations have been described above, voice recognition operations other than the above-mentioned voice recognition operations, such as activation when a voice recognition activation instruction is received from the display terminal 15, are also conceivable.

2-6-3. Voice recognition operation when door is closed FIG. 32 is a flowchart showing the operation when the voice recognition device is stopped when the door 10 of the refrigerator 51 is closed.
If the door 10 is closed (Y in step 108), the interior lights 34-1 and 34-2 are turned off (step 109), and after a while, for example, several seconds, the voice recognition devices 100-1 and 100-2 are turned off. (step 110). If the door 10 is not closed, the state of the door is checked again (N in step 108).

Here, by interlocking the voice recognition device with the door 10, the voice recognition device can be stopped when the door is closed, which is preferable. That is, there is no need to manually stop the speech recognition device, and there is no need to forget to stop the speech recognition device, so it is possible to prevent erroneous recognition of speech and to save energy.

2-6-4. Voice recognition operation depending on presence or absence of human sensor FIG. 33 is a flowchart showing the operation when the voice recognition device is stopped when it is detected that no person is present in front of the refrigerator 51.
If the state of the human sensor 101 is checked and no person is detected (Y in step 112), the voice recognition devices 100-1 and 100-2 are stopped after a while, for example, after several seconds (step 110). If a person is detected, the state of the human sensor is checked again (N in step 112).

Here, by interlocking the voice recognition device with the human sensor 101, the voice recognition device can be stopped when no person is detected, which is preferable. That is, there is no need to manually stop the speech recognition device, and there is no need to forget to stop the speech recognition device, so it is possible to prevent erroneous recognition of speech and to save energy.

Although typical voice recognition operations have been described above, in addition to the voice recognition operations described above, voice recognition operations may also be performed, such as stopping when there is an instruction to stop voice recognition from the display terminal 15, for example.

2-7. Recipe Search for Food Recognized by Voice Next, recipe search using data recognized by voice by each of the operations described above will be explained.

2-7-1. Recipe Search on Server FIG. 34 shows a case where the server device 57 searches for a recipe according to voice recognition data. When voice recognition is performed by the operations shown in FIGS. The device 55 is connected to the Internet 56 (step 113), and the voice-recognized data is transmitted to the server device 57 connected to the Internet 56 (step 114). The voice recognition data transmitted via the Internet 56 is subjected to recipe search processing (step 115) using the voice recognition data by the server control unit 66 of the server device 57. The recipe search results are stored in the data storage section 67 of the server device 57 (step 116).

By searching and storing recipes on the server device 57 as in this example, the load on the control unit 59 and storage unit 60 on the refrigerator side can be reduced, making it possible to create an Internet-compatible refrigerator at a low cost. can be provided.

2-7-2. Recipe Search on Display Terminal Side FIG. 35 shows a case where recipe search is performed on the display terminal 15 side according to voice recognition data. When voice recognition is performed by the operations shown in FIGS. 54, connect to the Internet 56 via the router device 55 (step 113), and send the voice-recognized data to the display terminal 15 instead of the server (step 117). The voice recognition data transmitted via the Internet 56 is subjected to recipe search processing (step 118) using the voice recognition data by the display terminal 15. The recipe search results are stored in the data storage section of the display terminal 15 (step 119).

Although the load on the display terminal 15 increases in this example as well, the load on the control unit 59 and storage unit 60 on the refrigerator side can be reduced, and an Internet-compatible refrigerator can be provided at a low cost.

The transmission of voice-recognized data and recipe search operations have been explained above by illustrating typical examples, but the storage and recipe search do not necessarily have to be performed in the same place, for example, the server device 57. For example, the recipe search process may be performed on the display terminal 15 side, and the recipe search results may be saved on the server device 57, etc., which may be set arbitrarily depending on the system.

2-8. Recipe display on display terminal 2-8-1. Display of Recipe Search Results on Display Terminal FIG. 36 is a flowchart showing the transmission and reception between the server device 57 or the refrigerator 51 and the display terminal 15, and shows the transmission and reception of data stored in the server device 57 or the refrigerator 51.

If there is voice recognition, the display terminal 15 receives the voice recognition data identified by the voice analysis process of the refrigerator 51 and displays it on the display terminal 15 (step 120). After displaying the voice recognition data, press the search start button on the display terminal 15 or say a specific keyword to perform a recipe search on the server device 57 or the display terminal 15, and keep track of the recipe search status until the recipe search is completed. Continue checking (N in step 121). When the recipe search is completed (Y in step 121), the recipe search results are displayed on the display terminal 15 (step 122). Note that the recipe search results may be automatically adjusted to the optimal display position according to the user's height.

As described above, according to this refrigerator system, recipe search results can be displayed on the display terminal 15, and by removing the display terminal 15, there is no restriction on where the display terminal can be viewed, and the user can enjoy cooking while cooking. You can carry it up close and check it, improving cooking efficiency.

<3. Effect>
As described above with specific configurations, the refrigerator and refrigerator system according to the embodiments of the present invention achieve various effects, but the refrigerator that is the gist of the present invention, that is, the front side of the refrigerator door By installing a voice recognition device on the side of the inside of the refrigerator, voice recognition can be performed regardless of whether the refrigerator door is open or closed, making it possible to provide a refrigerator that allows recipes to be searched just by voice. can. Further, by interlocking the door opening/closing and the human sensor, the voice recognition means does not have to be operated all the time, so it is possible to provide a refrigerator that can prevent erroneous recognition and contribute to energy saving.

In addition, in the refrigerator of this embodiment, the cameras 35-1 and 35-2 photograph the food in the storage 5 from the front side of the storage 5, but at this time, the interior lighting lights 34-1 and 34-2 also 5, and irradiates the food in the storage 5 from the front side in the same direction as the photographing direction by the cameras 35-1 and 35-2. Therefore, the photographed image is not as dark or difficult to see as in the case where the interior illumination lights 34-1 and 34-2 are provided on the back wall of the storage 5, but is a clear image. Furthermore, there is no need to request the server device 57 to perform image processing to brighten and sharpen the photographed image.

Furthermore, since the cameras 35-1, 35-2 and the interior illumination lights 34-1, 34-2 are located forward of the front ends of the shelf boards 30-1 to 30-4, The images of the inside of the storage 5 taken by the cameras 35-1 and 35-2 are the same as what the user actually sees when the door 10 of the refrigerator 51 is opened. Therefore, when the user displays the image data on the display terminal 15, the user can check the food situation in the storage 5 without feeling any discomfort. Moreover, since the interior lighting lights 34-1 and 34-2 illuminate the food on the shelves from the front of the front ends of the shelves 30-1 to 30-4, Foods placed there can also be photographed brightly and clearly.

In addition, the cameras 35-1 and 35-2 are installed on the inner surface of the door 10 of the refrigerator 51, and the interior lighting lights 34-1 and 34-2 are installed on both side walls of the opening edge of the storage 5. 35-1 and 35-2 are first located approximately at the center of the left and right sides of the storage 5. Therefore, the images taken by the cameras 35-1 and 35-2 can be made to look more like what the user actually sees when opening the door 10 of the refrigerator 51. In addition, by increasing the distance between the cameras 35-1, 35-2 and the food on the shelves, it is possible to photograph a wide range inside the storage 5, and the display terminal displays a wider range of food storage conditions. 15, it is possible to confirm.

In addition, since the cameras 35-1 and 35-2 are located approximately in the center of the door pockets 33-1 and 33-2 on the inner surface of the door in the front and back direction, they can be used not only for the food inside the storage 5 but also for the door pockets. It is also possible to photograph a portion of food stored in 33-1 and 33-2, such as milk cartons. Since the user stored the item himself, he can recognize what it is even if only a portion of it is shown. Therefore, it is possible to photograph almost all the food in the storage.

Furthermore, if a camera 35-3 dedicated to the storage container is provided as shown in FIG. 9(a), the food inside the storage container 31 can also be clearly photographed. Furthermore, by making the camera 35-2 vertically movable as shown in FIG. 9(b) or vertically movable as shown in FIG. 9(c), the food in the storage container 31 can be viewed with two cameras, that is, It is possible to efficiently take pictures with a small number of cameras.

On the other hand, as is clear from FIG. 7, the interior lighting lights 34-1 and 34-2 are provided at positions farther back than the cameras 35-1 and 35-2. Since the light from the interior lighting lamps 34-1 and 34-2 does not directly enter the photographing sections of the cameras 35-1 and 35-2, It is possible to prevent the images of the cameras 35-1 and 35-2 from becoming poor, and it is possible to always obtain good images. This can be done by providing a light shielding part on the camera side of the interior lighting lights 35-1, 35-2, or by tilting the light source of the LEDs etc. that make up the interior lighting lights 34-1, 34-2 diagonally backward. Similar effects can be obtained by adopting a configuration such as arranging the

In addition, the illuminance inside the storage chamber of the above-mentioned interior lighting lights 34-1 and 34-2 is 150 lux to 200 lux, and if you take a picture with this illuminance, the food will look whitish as explained in the section of the photographing operation in Fig. 14. The image becomes difficult to distinguish. Therefore, in this embodiment, the configuration is such that the illuminance is lowered and the image is photographed. This enables clear photography even with a CCD camera or CMOS camera. The illuminance of the interior lighting lights 34-1 and 34-2 is controlled by the server control unit 66 so that the illuminance is lower than normal only when the door 10 is closed or the door 10 is opened to take a picture. By performing such control, it is possible to maintain the same illuminance during normal times when no photography is being performed, making it possible to easily see the inside of the refrigerator and to take clear photographs. The above-mentioned decrease in illuminance may be caused by, for example, turning off a part of one or both of the interior lighting lights 34-1 and 34-2, or turning off both interior lighting lights 34-1 and 34-2. Various methods can be considered, such as reducing the illuminance of the lights 34-1 and 34-2, and may be selected as appropriate.

Furthermore, the images inside the storage 5 taken by the cameras 35-1 and 35-2 are sent to the server device 57, or the refrigerator 51, or The images are combined and synthesized within the display terminal 15, and in this embodiment, the above-mentioned image synthesis process is facilitated.

That is, the cameras 35-1 and 35-2 are connected to the inner surface of the door corresponding to the front ends of the shelf boards 30-1 to 30-4, in this example, the upper shelf board 30-1 and the third shelf board 30-. Since the cameras 35-1 and 35-2 are provided on the inner surface of the door corresponding to the front end of the camera 3, the images taken by the respective cameras 35-1 and 35-2 are centered on the shelf boards 30-1 and 30-3 as shown by the line Z in FIG. I will be taking pictures.

Therefore, the image data in the storage 5 captured by the two cameras 35-1 and 35-2 can be combined at the second shelf board 30-2. That is, as shown in the "Raw Image" in FIG. 27, the shelf board 30-2 is commonly seen in the upper and lower images, and the curvature of this "Raw Image" is corrected to create a "after curvature correction process". After creating the image, if the common shelf boards 30-2 in the upper and lower images are matched, a photographed image of the entire storage room will be obtained. As a result, it is no longer necessary to perform the extremely difficult correction process of aligning and combining each food item placed on a shelf during the image merging process, making the image merging process easier. Become. Therefore, the processing program for image combination can be made simple, and the control part of the server device 57, refrigerator 51, or display terminal 15 for image combination processing does not need to have a large capacity. This can contribute to faster processing speed.

Furthermore, since the two cameras 35-1 and 35-2 are installed approximately in front of the shelf boards 30-1 and 30-3, which cannot be adjusted in height, Even if the height of the shelves 30-2 is changed, the image merging process can be easily performed by adjusting the range of the merging process according to the height of the shelf board 30-2 during the image merging process.

INDUSTRIAL APPLICATION This invention can provide the refrigerator and its refrigerator system which can search for the recipe related to the food in a storage only by voice, and can be applied to a household refrigerator.

1 Refrigerator body 2 Outer box 3 Inner box 4 Insulation material 5 Storage (cold room)
6 Switching room (storage)
7 Ice making room (storage)
8 Freezer room (storage)
9 Vegetable room (storage)
10 Door 10a Exterior panel 10a-1 Glass plate 10a-2 Magic mirror 10b Inner plate 10b-1, 10b-2 Raised wall portion 11 Center pillar 12a, 12b, 13, 14 Pull-out door 15 Display terminal 15a Display 16 Cooling room 17 Cooler 18 Blower fan 19 Compressor 20 Heat dissipation pipe 21 Capillary tube 22 Damper 23 Storage frame 24 Terminal holder 25 Touch panel 26 Gap 27 Frame 28 Recess 29, 29a Smoke sheet 30-1 to 30-4 Shelf board 31 Storage container 32-1 to 32-4 Side wall protrusion 33-1 to 33-8 Door pocket 34-1, 34-2 Interior lighting (lighting device)
35-1, 35-2, 35-3 Camera (photography device)
41 Camera module 50 Refrigerator system 51 Refrigerator 53 Wireless adapter (communication device)
53a “Connect” button 54 Gateway device (relay device)
55 Router device 56 Internet 57 Server device 58 Interface (I/F)
59 Controller (control unit)
60 Storage section 61 Door opening/closing detection section 62 Fan drive circuit 63 Compressor drive circuit 64 AC power supply 65 Server interface 66 Server control section 67 Data storage section 70 "Direct" selection button 71 "Server" selection button 72 Image data selection button 73 Energy saving Related data selection button 74 Send button 100-1, 100-2 Microphone (voice recognition device)
101 Human sensor A Residence

Claims (9)

In a system equipped with a refrigerator and a display terminal,
The refrigerator is
A refrigerator and
a door provided in the refrigerator compartment;
a first voice recognition means for recognizing voice, which is provided on the front side of the door;
a second voice recognition means for recognizing voice, which is provided on an inner side surface of the refrigerator compartment;
door opening/closing detection means for detecting the opening/closing state of the door;
control means;
When the open state of the door is detected by the door opening/closing detection means, the control means activates the first voice recognition means and the second voice recognition means,
When a voice is recognized by the first voice recognition means and the second voice recognition means, the control means is configured to recognize the first voice data recognized by the first voice recognition means and the second voice. Determining which audio data to adopt based on the audio input level among the second audio data recognized by the recognition means,
After the voice data is determined by the control means, the voice data is displayed on the display terminal,
Confirmation means is provided for determining whether to execute a recipe search using the voice data,
A system characterized in that when the confirmation means determines to execute a recipe search using the audio data, the result of the recipe search is displayed on the display terminal. The refrigerator further includes a person detection means for detecting the presence of a person in front of the refrigerator,
2. The control means activates the first voice recognition means and the second voice recognition means when the presence of a person in front of the refrigerator is detected by the person detection means. The system described in 1. The display terminal includes a selection means for selecting whether to execute a recipe search using the audio data,
3. The system according to claim 1, wherein when the selection means selects to execute a recipe search using the audio data, the result of the recipe search is displayed on the display terminal . When a keyword for executing the recipe search is recognized by the first voice recognition means or the second voice recognition means, the result of the recipe search is displayed on the display terminal. A system according to any one of claims 1 to 3. The refrigerator further includes lighting means provided in the refrigerator compartment for illuminating the inside of the refrigerator compartment,
When the open state of the door is detected by the door opening/closing detection means, lighting the illumination means;
When the closed state of the door is detected by the door opening/closing detection means, the lighting means is turned off, and after a predetermined time has elapsed since the lighting means is turned off, the first voice recognition means and the second voice recognition are performed. 5. System according to any one of claims 1 to 4, characterized in that the means is stopped. 6. The system according to claim 1, further comprising transmitting means for transmitting the first audio data or the second audio data to the outside. 7. The system according to claim 1, wherein the audio data is a food name. 8. The system according to claim 1, wherein the door is a double door. A refrigerator and
a door provided in the refrigerator compartment;
a first voice recognition means for recognizing voice, which is provided on the front side of the door;
a second voice recognition means for recognizing voice, which is provided on an inner side surface of the refrigerator compartment;
door opening/closing detection means for detecting the opening/closing state of the door;
control means;
and a display means,
When the open state of the door is detected by the door opening/closing detection means, the control means activates the first voice recognition means and the second voice recognition means,
When a voice is recognized by the first voice recognition means and the second voice recognition means, the control means is configured to recognize the first voice data recognized by the first voice recognition means and the second voice. Determining which audio data to adopt based on the audio input level among the second audio data recognized by the recognition means,
After the voice data is determined by the control means, the voice data is displayed on the display means,
Confirmation means is provided for determining whether to execute a recipe search using the voice data,
A refrigerator characterized in that when the confirmation means determines to execute a recipe search using the audio data, the result of the recipe search is displayed on the display means.