TW201804860A - Refrigerator, network system provided with same, living circumstances reporting method and living circumstance reporting program - Google Patents

Abstract

A refrigerator (1) used by a first user comprises: a storage chamber that stores food; a communication unit (23) that performs communications; an imaging device (27) that acquires an image of the interior of the storage chamber; an image management unit (101) that extracts food images from the storage chamber image acquired by the imaging device (27); a food status detection unit (103a) that detects, from food images during a certain period, a change in the status of food stored in the storage chamber; a use status acquisition unit (104) that acquires first information indicating the use status of the refrigerator (1) by the first user, from changes in the status of food detected by the food status detection unit (103a); and a life management unit (105) that reports second information indicating a living status to a terminal device (70) used by a second user via the communication unit (23), if it was determined from the first information that the living status of the first user is in a state that should be reported to the second user, who is different from the first user.

Description

Refrigerator, network system having the same, method for reporting living conditions, and program for reporting living conditions

The invention relates to a refrigerator with a communication function, a network system including the refrigerator, a method for reporting the living conditions of users of the refrigerator, and a program for reporting the living conditions that cause a computer to execute the reporting method.

In recent years, the proportion of elderly people (over 65 years old) in the total population in Japan has been increasing. In addition, there is a tendency that the number of people constituting one household decreases and the number of households (solitary living) increases. Therefore, it is necessary to confirm whether the elderly who live alone are safe and whether they need care. However, the living condition of a person can be inferred from the use condition of the refrigerator of the person (user). This is because the use of the refrigerator is closely related to the eating habits of a part of the life of the refrigerator user.

Patent Document 1 describes a technology in which a radio frequency tag is attached to a food, and based on information transmitted from the radio frequency tag and temperature information transmitted from a temperature sensor in a refrigerator, a storage period-storage temperature-quality tolerance (TTT) ) Theory to estimate the quality deterioration characteristics of food.

In addition, Patent Document 2 describes a refrigerator as an article storage library, which transmits image data obtained by taking pictures of the inside and outside of the refrigerator through a server to a mobile phone (mobile terminal), so that the inside of the refrigerator can be monitored from a remote place. And outside the refrigerator.

Advance technical literature

[Patent Literature]

Patent Document 1: Japanese Patent Laid-Open No. 2006-317109 (for example, paragraph 0027-0042, FIG. 5)

Patent Document 2: Japanese Patent Laid-Open No. 2007-46833 (for example, paragraphs 0021-0025, FIG. 1)

However, the technology described in Patent Document 1 has the following problem: In order to obtain information indicating the use status of a refrigerator, readers and writers such as radio frequency tags attached to foods are required. Dedicated device, which increases the overall cost of the system.

In addition, the refrigerator described in Patent Document 2 does not need a dedicated device such as a reader and a writer, and can transmit image data inside and outside the refrigerator to a user's mobile terminal, and display the image inside and outside the refrigerator on the user's Display of mobile terminal. However, there is a problem as follows: it cannot perform judgment based on the use status of the refrigerator (for example, judgment regarding the living conditions of users).

Furthermore, neither of Patent Documents 1 and 2 describes a technology for reporting the living conditions of the users of the refrigerator to a terminal device (second user terminal) used by other users (persons other than users, third parties).

The present invention is to solve the above-mentioned problems, and an object thereof is to provide a refrigerator, a network system including the refrigerator, a notification method of a living condition of a user of the refrigerator, and a notification program for causing a computer to execute the notification method of a living condition. The living conditions of the first user of the refrigerator obtained from the image information of the food stored in the storage room are notified to the terminal device used by the second user (a person other than the first user of the refrigerator).

One aspect of the present invention is a refrigerator including a storage room for storing food and a communication unit for performing communication. The refrigerator is used by a first user. The refrigerator includes: a photographing device for obtaining images in the storage room; An image management unit that extracts a food image from the image obtained by the photographing device; a food condition detection unit that detects a change in the condition of the food stored in the storage room from the food image in a certain period; a use condition acquisition unit To obtain the first information indicating the use status of the refrigerator by the first user from the change in the condition of the food detected by the food condition detection section; and the life management section judges and learns from the first information In the case where the living conditions of the first user should be notified to a second user different from the first user, the second information indicating the living conditions is transmitted to the second user through the communication unit. Terminal device notification.

According to another aspect of the present invention, a network system includes: the refrigerator; and the terminal device connected to the refrigerator in a communicable manner and used by the second user.

According to another aspect of the present invention, a method for reporting a living condition includes: a step of capturing a storage room of a refrigerator used by a first user to obtain an image of the storage room; and extracting a food image from the obtained image. A step of detecting a change in the condition of the food stored in the storage room from the food image in a certain period; obtaining a change indicating the change of the condition of the food from the detected first Step 1 of the information on the use of the refrigerator; And when it is judged from the first information that the living condition of the first user should be notified to a second user different from the first user, the second information indicating the living condition is used by the second user The procedure reported by the terminal device used by the user.

According to the present invention, it is possible to obtain the first information indicating the use state of the refrigerator by the first user from the change in the state of the food detected from the food image, and it can be judged from the first information that the living condition of the first user belongs to what should be When the situation is reported by the second user, the second information indicating the living situation is notified to the terminal device (other user terminal) used by the second user. Therefore, the second user who has received the second information indicating the living conditions of the first user can take appropriate measures to the first user.

1, 2, 3, 4‧‧‧ refrigerator

11‧‧‧ Refrigerator

12‧‧‧ Ice Making Room

13‧‧‧Switch Room

14‧‧‧Freezer

15‧‧‧ Fruit and Vegetable Room

16‧‧‧ Refrigerator Shelves

17‧‧‧cooling room

18‧‧‧ Cooling Box

19‧‧‧Cooling air path

20‧‧‧ Door

21, 21a‧‧‧ operation panel

22‧‧‧Opening and closing detection department

23‧‧‧ Ministry of Communications

24‧‧‧Memory Department

25‧‧‧Time Offer Department

27‧‧‧Photographic installation

62‧‧‧ Food list display screen in refrigerator

70‧‧‧Second user terminal (other user terminals, terminal devices)

71‧‧‧Display

91, 91a ~ 91e‧‧‧ have planned plan mark

92 (92a ~ 92g) ‧‧‧Mark of planned contents

93‧‧‧ Now

94, 94a, 94b ‧‧‧ Scheduled out of time zone

95, 95a, 95b ‧‧‧ Scheduled outing time zone

96‧‧‧ Bedtime zone

100, 200, 300‧‧‧ Control Department

101‧‧‧Image Management Department

102‧‧‧Change detection department

103‧‧‧Deterioration detection section

103a‧‧‧Food Condition Detection Department

104‧‧‧Usage status acquisition department

105‧‧‧Life Management Department

106‧‧‧Food Information Memory

107‧‧‧Food Identification Department

108‧‧‧Schedule Management Department

109‧‧‧ Schedule Setting Department

110, 210, 310‧‧‧ control board

500‧‧‧In-House System

501‧‧‧System Power

502‧‧‧Wire

503‧‧‧ Electricity measuring device

504‧‧‧Power measurement terminal

505‧‧‧In-House Controller

506‧‧‧ Broadband Router

515‧‧‧ Machine Communication Department

600‧‧‧out-of-house system

601‧‧‧Internet

602‧‧‧ Cloud Server

603‧‧‧Personal Certification Department

Fig. 1 is a front view showing a schematic structure of a refrigerator according to a first embodiment of the present invention.

Fig. 2 is a cross-sectional view showing a schematic structure of a refrigerator compartment of the refrigerator according to the first embodiment (the cross-sectional structure of Fig. 1 is cut by a line segment II-II).

Fig. 3 is a block diagram schematically showing a configuration of a control system of the refrigerator of the first embodiment.

Fig. 4 is a flowchart showing the operation of the refrigerator in the first embodiment.

5 (a) to (d) are diagrams showing examples of photographed images of the refrigerator compartment of the refrigerator according to the first embodiment, food images extracted from the photographed images, and additional information of the food images.

Figures 6 (a) and (b) are diagrams showing examples of display screens of food images captured from photographed images of the refrigerator compartment of the refrigerator in the first embodiment on the mobile terminal.

FIG. 7 is a diagram showing an example of a display screen of a food image captured from a photographed image of a refrigerator compartment of a refrigerator in Embodiment 1 on a mobile terminal.

8 (a) and 8 (b) are schematic diagrams illustrating a method for detecting a time change of a food image in a change detection unit of the refrigerator according to the first embodiment.

9 (a) and 9 (b) are diagrams showing examples of display screens of food images of the refrigerator in the mobile terminal of the first embodiment.

Fig. 10 is a sectional view showing a schematic structure of a refrigerator compartment of a refrigerator according to a second embodiment of the present invention (the sectional structure of Fig. 1 is cut by a line segment II-II).

Fig. 11 is a block diagram schematically showing a control system of a refrigerator according to a second embodiment.

Fig. 12 is a diagram showing an example of contents stored in a food information storage unit of a refrigerator according to a second embodiment.

Fig. 13 is a flowchart for explaining the operation of the refrigerator in the second embodiment.

Fig. 14 is a cross-sectional view showing a schematic structure of a refrigerator compartment of a refrigerator according to a third embodiment of the present invention (the cross-sectional structure of Fig. 1 is cut by a line segment II-II).

Fig. 15 is a block diagram schematically showing a configuration of a control system of a refrigerator according to a third embodiment.

Fig. 16 is a diagram showing an example of a schedule display screen (monthly display) on an operation panel (schedule setting unit) of the refrigerator according to the third embodiment.

Fig. 17 is a diagram showing an example of a schedule display screen (weekly display) on an operation panel (schedule setting unit) of the refrigerator according to the third embodiment.

Fig. 18 is a diagram showing an example of a schedule display screen (daily unit display) on an operation panel (schedule setting unit) of the refrigerator in the third embodiment.

Fig. 19 is a diagram showing the overall configuration of an in-house system and an out-house system to which a network system including a refrigerator according to a fourth embodiment of the present invention is applied.

《1》 Implementation

"1-1" composition

FIG. 1 is a front view showing a schematic structure of a refrigerator 1 according to Embodiment 1 of the present invention. As shown in FIG. 1, the refrigerator 1 according to the first embodiment includes a plurality of storage rooms for storing food. The plurality of storage compartments include, for example, a refrigerating compartment 11, an ice-making compartment 12, a switching compartment 13, a freezing compartment 14, and a fruit and vegetable compartment 15. In the example shown in FIG. 1, the refrigerator compartment 11, the ice-making compartment 12 and the switching compartment 13, the freezing compartment 14, and the fruit and vegetable compartment 15 are arranged in this order from the top. The ice making chamber 12 and the switching chamber 13 are arranged side by side in the horizontal direction. The refrigerator 1 has a cooling device which cools a storage room. The cooling device includes a refrigeration cycle including, for example, a compressor, a condenser, an expansion valve, and an evaporator. The plurality of storage compartments are cooled by a refrigeration cycle. The first embodiment is also referred to in the second embodiment (FIG. 10) and the third embodiment (FIG. 14). In Embodiments 2, 3, and 4, the refrigerator 1 is labeled as refrigerators 2, 3, and 4, respectively.

An operation panel 21 may be provided on the front (front) of the door piece (refrigerator door piece) 20 of the refrigerating compartment 11 and includes a display section (display screen) as a display device. Storage room information (inside the refrigerator, information about the refrigerator 1 such as the set temperature and current temperature of each storage room) may be displayed on the display screen of the operation panel 21. The operation panel 21 includes, for example, a touch panel that receives input from a user. The installation position of the operation panel 21 is not limited to the door sheet 20 of the refrigerating compartment 11, and may be provided outside the door sheet of other storage compartments or other positions such as the side of the refrigerator 1. In addition, for example, the ice-making compartment 12, the switching compartment 13, the freezing compartment 14, and the fruit and vegetable compartment 15 are configured to be pulled out and slide toward the near side of the refrigerator 1 together with a door panel provided in front of the ice-making compartment. FIG. 1 shows an example of a refrigerator to which the present invention is applicable, but the present invention The applicable refrigerator is not limited to the refrigerator shown in FIG.

Fig. 2 is a cross-sectional view showing a schematic structure of the refrigerator compartment 11 of the refrigerator 1 of the first embodiment (the cross-sectional structure of Fig. 1 is cut by a line segment II-II). As shown in FIG. 2, the refrigerator 1 includes a door piece 20 for opening and closing an opening portion on the front side (left side in FIG. 2) of the refrigerator compartment 11. The refrigerator compartment 11 is divided into a plurality of spaces by a plurality of shelf plates (shelf) 16 arranged inside. The space below the lowermost shelf plate 16 is a cooling chamber 17. A cooling box 18 is provided inside the cooling chamber 17 and can be drawn forward (left direction in the second figure) along a guide element (not shown) such as a rail. In addition, a cooling air passage 19 is provided on the back side (right side in FIG. 2) of the refrigerating compartment 11 as a passage for guiding the air cooled by the refrigeration cycle to the storage compartment.

The body of the refrigerator 1 connected to the door sheet 20 is provided with an opening / closing detection section 22 as an opening / closing detection device for detecting the opening and closing of the door sheet 20. The opening / closing detection unit 22 is a magnet-type device, and detects the approach of a magnet (permanent magnet) embedded in the door 20 by a pair of reed switches provided in the body of the refrigerator 1, for example. However, the opening / closing detection unit 22 may be a detection device using another detection method such as a mechanical switch or an optical switch.

The refrigerator 1 includes a control substrate 110 including a control circuit (control unit) for operating the refrigerator 1. The control substrate 110 is mounted on, for example, the rear surface of the refrigerator 1.

A photographing device 27 is provided on the inner side of the door piece 20. In the example shown in FIG. 2, the imaging device 27 is configured to use the range from the top surface of the refrigerating compartment 11 to the cooling compartment 17 as the field of view, and to photograph the refrigerating compartment from the door sheet 20 side on the same side as the user's viewpoint. Within 11. The installation position of the imaging device 27 is not limited to the inner surface of the door panel 20. The photography device 27 can also be set in other positions As long as the storage items such as foods placed on the shelf plate 16 of the refrigerator compartment 11 can be photographed in a wide range, it is sufficient. In addition, considering that the field of view will be blocked by the shelf plate 16, a plurality of photographing devices 27 may be provided so that more stored objects can be photographed. In addition, the imaging device 27 may be provided at a position where an image of a storage room other than the refrigerator compartment 11 is taken.

In addition, the refrigerator 1 is provided with a control substrate 110 having a control circuit for operating the refrigerator 1. The control substrate 110 is mounted on, for example, the rear surface of the refrigerator 1.

Fig. 3 is a block diagram schematically showing a configuration of a control system of the refrigerator 1 according to the first embodiment. For example, as shown in FIG. 3, the control system of the refrigerator 1 includes an operation panel 21, an opening / closing detection unit 22 that detects an open state and a closed state of the door sheet 20, and acts as an external device that can be wired or wirelessly A communication unit 23 of a communication device that communicates with a terminal (terminal device), a memory unit 24 such as a semiconductor memory that stores information, a time providing unit 25 that provides time information, and a control unit 100 that controls the overall operation of the refrigerator 1. In addition, the control system of the refrigerator 1 includes, for example, a compressor drive circuit 31 that drives a compressor 32 constituting a part of the refrigeration cycle, and a fan (motor) 34 that drives air to flow into the storage room of the refrigerator 1 Fan driving circuit 33. A circuit that is a main part of the control system of the refrigerator 1 may be provided on the control substrate 110. However, a part of the circuit in FIG. 3 may be provided on another control board (not shown).

The communication unit 23 includes a communication module such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The refrigerator 1 uses the communication unit 23 to pass, for example, at least one of a router (broadband router 506 shown in FIG. 19 to be described later) and an Internet (Internet 601 shown in FIG. 19 to be described later). And smartphones, portable tablets, mobile phones, personal computers, etc. inside or outside the house Terminal device for communication. For example, this terminal device includes a terminal device (hereinafter also referred to as a "first user terminal") used by a user of the refrigerator 1 (hereinafter also referred to as a "first user" or "supervised person") And terminal devices used by third-party users other than the first user (hereinafter also referred to as "second user" or "manager") (hereinafter also referred to as "second user terminal") Or "other user terminal") 70. The control unit 100 of the refrigerator 1 can transmit information for displaying various information about the refrigerator 1 on one or both of the display unit of the first user terminal and the display unit 71 of the second user device 70.

The time information provided by the time providing unit 25 is not only hours, minutes, and seconds, but also information of a date (month day or year month day). The time providing unit 25 may be, for example, a clock or a time receiving device that receives a time signal transmitted by a time signal provider.

The control unit 100 may be constituted by an electric circuit or a processor of an information processing device that executes software (program) stored in the memory unit 24. When the control unit 100 is configured by a processor, the functions of the control unit 100 are realized by executing programs stored in the storage unit 24. As described above, the function of the control unit 100 can be realized by a program (a notification method of a living condition or a notification program) executed by a processor in the refrigerator 1 (or 2 to 4). In addition, the function of the control unit 100 may be realized by a program (a method or a program for reporting a living condition) executed by a computer (also a server on the network) that can communicate with the refrigerator 1. The program for reporting the living conditions is a program for reporting the living conditions of the first user. The refrigerator 1 (or 2 to 4) used by the first user includes a storage room 11 for storing food, a communication section 23 for performing communication, and acquisition storage. The image capturing device 27 in the room 11 is a program that enables a computer (the control unit 100 or an external computer or a network An information processing device such as a server) to perform the processing described below: processing for the photographing device 27 to take pictures of the storage room 11 of the refrigerator 1 (or 2 to 4) to obtain the image in the storage room 11; to extract food images from the acquired images Processing of detecting a change in the condition of the food stored in the storage room 11 from a food image in a certain period; obtaining a change indicating the status of the food from the detected first food by the first user to the refrigerator Processing of the first information of the usage status of 1 (or 2 to 4); judging from the first information that the living condition of the first user belongs to a situation in which it is necessary to report to a second user different from the first user , A process of reporting the second information indicating the living situation to the terminal device used by the second user.

As shown in FIG. 3, the control unit 100 includes an image management unit 101, a change detection unit 102, a degradation detection unit 103, a use situation acquisition unit 104, and a life management unit 105. The control unit 100 may include a main control unit (not shown) that controls the overall operation of the control unit 100. The change detection unit 102 and the deterioration detection unit 103 constitute a food condition detection unit 103a. In addition, in FIG. 3, the memory unit 24 is shown as a separate structure from the video management unit 101, but the memory unit 24 may be a part of the video management unit 101 (or a part of the control unit 100), or may be other Make up part of it. For example, the memory unit 24 may be a memory device that is externally connected to the refrigerator 1 or a memory device that is a device (for example, a server) connected in a manner capable of communicating via a network.

The video management unit 101 is connected to the imaging device 27, the change detection unit 102, and the communication unit 23. For example, when the door sheet 20 is opened and closed, the image management unit 101 receives the entire image (image data) in the refrigerating room 11 obtained by capturing the interior of the refrigerating room 11 with the imaging device 27, and receives the received entire image (hereinafter also referred to as "Image in refrigerator" or "Image in storage room") Capture image of food part (food Product image). In addition, the image management unit 101 adds time information (for example, shooting time information) to the food image extracted from the obtained overall image as additional information, and stores it in a memory unit (may be It is a memory part different from the memory part 24, or may be the memory part 24). The image management unit 101 may also rearrange the food images in time series based on the time information and then store them. The image management unit 101 provides the food image and incidental information (which may include time information and storage and retrieval identification information) to the change detection unit 102 and the communication unit 23.

The change detection unit 102 is connected to the video management unit 101, the degradation detection unit 103, and the communication unit 23. The change detection unit 102 receives food images and supplementary information from the image management unit 101, detects changes in the color and size of the food image (corresponding to the actual food color and size), and stores them in the change detection unit. The memory unit in 102 (may be a memory unit different from the memory unit 24, or may be the memory unit 24). In addition, the change detection unit 102 provides information indicating the time change of the color and size of the detected food image to the deterioration detection unit 103 and the communication unit 23.

The degradation detection unit 103 is connected to the change detection unit 102, the usage status acquisition unit 104, and the communication unit 23. The deterioration detection unit 103 receives the time change of the color and size of the food image from the change detection unit 102, detects (estimates) the deterioration state of the food based on the time change, and provides the detection result of the deterioration state to the use situation Acquisition section 104 and communication section 23. Therefore, the food condition detection unit 103a including the change detection unit 102 and the deterioration detection unit 103 can detect the storage state of the stored food from the food image captured from the storage room image.

The use condition acquisition unit 104 is connected to the deterioration detection unit 103, the life management unit 105, and the communication unit 23 of the food condition detection unit 103a. Usage The acquisition unit 104 receives the deterioration state of the food unit of the food image from the deterioration detection unit 103 as the storage state of the food stored in the refrigerating compartment 11, and detects (estimates) from the information indicating the deterioration state that "now stored Food that has not deteriorated. "Or long-term use conditions such as" degraded food has been left for a few days. " As described above, the use status acquisition unit 104 detects the use status of the refrigerator 1 by the first user based on the deterioration state of the food in the storage room, and displays information indicating the use status (hereinafter also referred to as "first information"). ) To the life management section 105.

The life management unit 105 is connected to the use situation acquisition unit 104 and the communication unit 23. The life management unit 105 receives the first information indicating the use status of the first user based on the deterioration state of the food from the use status acquisition unit 104, and obtains the information indicating the life status of the first user from the first information (hereinafter (Also known as "second information"). For example, the second information indicating the living condition of the first user may be the tendency of the first user to be at home or away from the home of the first user, or the frequency of the first user's shopping. In addition, the life management unit 105 classifies information (to be notified) to be reported to the second user terminal 70 from the second information indicating the living situation, for example, information related to the safety of the first user of the refrigerator 1, Or, information about theft prevention of the first user's house is transmitted to the communication unit 23 as the second information indicating the reportable living conditions.

The communication unit 23 is connected to the video management unit 101, the change detection unit 102, the degradation detection unit 103, the usage status acquisition unit 104, and the life management unit 105, and is connected to the second user terminal 70 wirelessly or by wire. The communication unit 23 receives a food image and ancillary information from the image management unit 101, and receives information indicating a time change of the color and size of the food image from the change detection unit 102. Information, the information indicating the deterioration state of the food is received from the deterioration detection unit 103, the first information indicating the use situation of the first user is received from the use situation acquisition unit 104, and the second situation should be reported to the second situation among the living conditions received from the life management unit 105. Information reported by users (second information). Each piece of information can be transmitted to a second user terminal 70 used by a second user other than the first user using the refrigerator 1 by wireless communication. In addition, the second user terminal 70 as a transmission target of the second information may be registered in advance by, for example, a first user or the like in a memory section in the refrigerator 1 or a server serving as a management device on the network.

The second user terminal 70 is connected to the communication unit 23 so that wireless communication can be performed. The second user terminal 70 receives the food image and accompanying information transmitted from the communication unit 23, information indicating the time change of the color and size of the food image, information indicating the deterioration state of the food, and the refrigerator 1 indicating the first user. The information that should be notified to the second user among the first information of the usage status and the second information indicating the living conditions can be displayed on the display portion 71 of the second user terminal 70.

"1-2" action

Next, an example of the operation of the refrigerator 4 according to the first embodiment will be described with reference to FIGS. 2 to 4. FIG. 4 is a flowchart for explaining an operation performed when the door sheet 20 of the refrigerator 1 is opened and closed.

In the operation shown in Fig. 4, the opening and closing of the door panel 20 is started. When the opening / closing detection unit 22 detects the opening and closing of the door sheet 20 (YES in step S11), the imaging device 27 images the inside of the refrigerator compartment 11 (step S12). For example, the photographing device 27 may take an image while the door 20 is closed and the illumination in the refrigerator compartment 11 is still on. This is because when the opening and closing of the door sheet 20 is detected, the Food may be taken in and out.

The imaging device 27 supplies the obtained image of the storage room in the refrigerator compartment 11 (preferably the image of the entire storage room) to the image management unit 101 (step S13). Therefore, the image management unit 101 receives the storage room image from the photographing device 27 every time the door sheet 20 is opened and closed (that is, whenever the storage state in the refrigerator compartment 11 may change). In the following description, the time when the (most recent) door 20 is opened and closed is referred to as "this time the door is opened and closed", and the time when the last (one time ago) door 20 is opened and closed is referred to as "the last door When opening and closing. "

The image management unit 101 memorizes (for example, in the memory unit 24) the storage room image (image data) received from the photographing device 27 when the door was opened and closed last time, and re-memorizes (for example, in the memory unit 24) this time. The storage room image received from the imaging device 27 when the film is opened and closed. Next, the image management unit 101 calculates the difference between the image of the storage room when the door was opened and closed last time and the image of the storage room when the door was opened and closed this time, thereby capturing the image of the changed part of the storage room image as a food image (step S14). That is, the image of the changing part of the so-called storage room image is an image of the food placed in or taken out of the refrigerator.

For example, in the image of the storage room when the door was opened and closed last time, beer was placed on the shelf plate 16 of the refrigerator compartment 11. In the image of the storage room when the door was opened and closed this time, only the shelf without beer was placed at the same position. In the case of plate 16, the food image of beer is acquired from the difference between the images of the two storage rooms.

The image management unit 101 adds the time information of the current time (or the time of performing shooting) to the captured food image as additional information, and based on the time information, adds these food images and additional information The information is rearranged in time series and stored (step S15).

At this time, the image management unit 101 determines whether the food is stored in or out of the storage. Specifically, the image management unit 101 analyzes the changed part according to the difference when capturing the food image, such as the amount of edge change obtained after the edge processing, and the storage room image, food image, and The storage room images of the empty state (state where no food is stored in the refrigerating compartment 11 at all) are compared to determine whether the food is in or out of the storage. Subsequently, the image management unit 101 adds and stores the judgment result (incoming and outgoing identification information) of the food storage image to the incidental information of the food image.

In addition, the image management unit 101 transmits the food image and incidental information (time information and storage identification information) to the change detection unit 102 and the communication unit 23 (step S16).

In the next step S17, the change detection unit 102 receives the food image and additional information from the image management unit 101, and detects and stores the time change of the color and size of the food image, which will be described later. In steps S12 to S17, after obtaining the storage room image of the refrigerating compartment 11, a series of actions until the food image and additional information are provided to the food condition detection unit 103a are detected every time the door sheet 20 is detected in step S11. Implemented during opening and closing.

Figures 5 (a) to (d) show captured images 81a to 81d of the refrigerator compartment 11 of the refrigerator 1 of Embodiment 1, food images 82b, 82c, 82d1, 82d2, and food images captured from the captured images. Figures with examples of information 83d1, 84d1, 83d2, and 84d2. As shown in Figures 5 (a) to (d), when the door 20 is opened and closed, the shooting data (storage room image) obtained by the photographing device 27 and transmitted to the image management unit 101, and the image captured from the storage room image Data (food image), and its accompanying information (time information and inbound and outbound identification information).

FIG. 5 (a) shows that at time A, the imaging device 27 captures a storage room image 81 a which is acquired and transmitted to the image management unit 101. At time A, the door piece 20 is opened and closed, but no food is put in or taken out. There is no change in the storage room image at time A and before, so the image management unit 101 does not generate a differential image (food image).

FIG. 5 (b) shows that at time B after time A, the photographing device 27 captures a storage room image 81b acquired and transmitted to the image management unit 101, a food image 82b captured from the storage room image 81b, and With information 83b, 84b. At time B, the beer is taken out from the refrigerator compartment shelf plate on the uppermost floor of the refrigerator compartment 11. The image management unit 101 extracts the food image (beer) 82b based on the difference between the storeroom image at time A and the storeroom image at time B, and stores the food image and time information (time B) based on the result of the judgment of entering or leaving the warehouse. 83b and entry and exit identification information 84b for "out of stock".

Fig. 5 (c) shows that at time C after time B, the camera 27 captures a storage room image 81c acquired and transmitted to the image management unit 101, a food image 82c extracted from the storage room image 81c, and the accompanying Information 83c, 84c. At time C, the food (salad) in the container is taken out from the refrigerator compartment shelf plate on the lowermost floor of the refrigerator compartment 11. The image management unit 101 extracts a food image (salad) 82c based on the difference between the storage room image at time B and the storage room image at time C, and stores the food image and the time as additional information based on the result of the judgment of entering or leaving the warehouse. Information "time C" 83c and entry-exit identification information 84c of "out of stock".

Fig. 5 (d) shows that at time D after time C, the storage room image 81d, which was captured by the photographing device 27 and transmitted to the image management unit 101, the food images 82d1 and 82d2, which were captured from the storage room image 81d2, and Additional information 83d1, 84d1, 83d2, 84d2. At time D, the carton's milk is placed on the uppermost refrigerating compartment shelf plate of the refrigerator compartment 11, and the cake is placed on the lowermost refrigerating compartment shelf plate. The image management unit 101 extracts food images (milk and cake) 82d1 and 82d2 based on the difference between the storeroom image at time C and the storeroom image at time D, and based on the result of the judgment of entering or leaving the warehouse (here, the milk and cake are both ("Warehouse"), together with the food image, store time information "time D" 83d1, 83d2, which is supplementary information, and "warehouse" in and out identification information 84d1, 84d2.

As described above, the image management unit 101 stores the food image together with the time information and the storage and retrieval identification information. Therefore, the food storage time can be grasped, and the deterioration state can be detected based on the storage time. That is, the image management unit 101 grasps the foods (stocked foods) stored in the refrigerating compartment 11 based on the images, and holds information necessary for detecting the deterioration state of the stocked foods.

Here, the image data stored by the image management unit 101 may include only a food image (a changing part of a storage room image) extracted from a difference from the previous image data, and the latest storage room image (the difference reference) Overall image of the refrigerator compartment 11). That is, after capturing the food image, the image management unit 101 can save only the latest storage room image and delete other storage room images.

As described above, the image management unit 101 reduces the amount of image data stored in the image management unit 101. Therefore, the memory capacity of the memory unit and the data communication load can be reduced. In addition, the shooting by the photographing device 27, the capture of the food image by the image management unit 101, and the storage of the image data may be limited to only when the door 20 is opened and closed (possible to put or take out food), Only data necessary for the management of foods stored in the refrigerating compartment 11 are stored. Therefore, it is possible to (Photographing, image processing, and preservation) to obtain the storage state and deterioration state of the food in the refrigerator compartment 11.

Here, the so-called food storage state is information indicating which food is stored in the refrigerator compartment 11 at which point in time. The deterioration state of the food is information indicating how far the food has deteriorated (for example, whether it needs to be used as soon as possible).

The image management unit 101, through the communication unit 23, outputs the food image data stored together with the stored additional information (time information and storage identification information) according to time series or divided into storage or storage, and outputs it to the storage refrigerator 1 The display unit 71 of the first user terminal used by the first user and the second user terminal 70 used by the second user other than the first user. The second user (for example, a family member or a cohabitant of the first user who uses the refrigerator 1) can grasp the food stored in the refrigerator compartment 11 by viewing the display screen of the display section 71 of the second user terminal 70.

6 (a) and 6 (b) are diagrams showing examples of display screens of a food image captured from a captured image of the refrigerator 1 of the first embodiment on a mobile terminal. In the display screen (time axis display screen) shown in FIG. 6 (a), the food image 42 captured from the storage room image each time the door 20 is opened and closed, the time information 43 attached to the food image 42 and The inbound and outbound identification information 44 is displayed in a time series. In this example, at the center of the screen, the food images 42 are arranged from the top (from the newer shooting time information (time information)), the time information 43 is displayed on the left of each food image, and the information is displayed on the right. Outbound identification information 44. In addition, the target storage room (refrigerator compartment 11 in this example), its set temperature, and the current time are displayed at the top of the screen.

In the time axis display screen shown in FIG. 6 (a), the contents of the screen can be retrieved from top to bottom (for the target food retrospective time), and the expiration date or appreciation period can be confirmed visually. Therefore, the expiration date or taste period of food can be notified as information about eating and drinking to promote its consumption and use, thereby preventing food waste.

The display screen shown in FIG. 6 (b) (incoming / outgoing list display screen) is divided into left and right columns according to incoming and outgoing, and in each column, food images and time information are displayed in time series (list display). That is, in the left column, the food image and its time information (that is, the time of storage) 53 with the identification information 54 of the storage in and out of the storage are listed in order from the newest of the storage time 53 from top to bottom. . In the right column, the food image 52 and the time information (that is, the time of storage) 53 of the identification information 54 for the storage and removal are listed in order from the newest in time from the storage 53 to the bottom.

In the inbound and outbound list display screen shown in FIG. 6 (b), the current remaining amount of the food can be obtained from the inbound and outbound history of the target food, so it is possible to notify other user terminals 70 of the remaining food The amount is used as information related to eating and drinking, so that the user or the caregiver can confirm the display 71 of the other user terminal 70 at a remote place (for example, at a shopping place), which can prevent the purchase of useless ingredients, too many ingredients, and forgetting Buy ingredients.

FIG. 7 is a diagram showing another example of a display screen of a food image captured from a captured image of the refrigerator 1 of the first embodiment on a mobile terminal. The image management unit 101 determines from a food image (i.e., a food image captured every time the door 20 is opened and closed) and time information and storage identification information as additional information, and determines whether to store it at a certain point in time. In refrigerator compartment 11 (incoming and outgoing identification information is not "Out of stock"). In addition, the communication unit 23 displays a list of stock food images as shown in FIG. 7 as the food list display screen 62 in the refrigerator. Moreover, in the example of FIG. 7, unlike FIG. 6 (a) and (b), based on the time information, the food image is arranged in order from the top to the bottom from the beginning of the storage time.

In the refrigerator food list display screen 62 shown in FIG. 7, a list of stock food images at a certain time point is displayed, and the remaining amount of food is reported more clearly than the list display shown in FIG. 6 (b). As information about eating and drinking. Therefore, it is possible to grasp which foods are stored in the refrigerating compartment 11, and it is expected to prevent buying too much ingredients and forgetting to buy ingredients. In addition, food images are arranged from top to bottom in order from the beginning of the storage time, so it is easy to find foods that have been stored for a long time (that is, foods that must be used as soon as possible), which can reduce the number of food wastes that are discarded due to expiration. The situation happened.

In addition, it is explained here that the display section 71 of the second user terminal 70 displays the "acceptance / departure time axis" of Fig. 6 (a), the "acceptance / departure list" of Fig. 6 (b), and Fig. 7 Three types of food image display screens, such as the "stock food list", can continuously display any of these screens so that these screens can be displayed in response to the operation of the first user (for example, by touching the touch panel). In addition, the switching of the food image display screen may be performed by the operation of a button connected to each display screen, or may be performed by a horizontal sliding operation or the like.

In addition, the target storage room (the refrigerating room 11 in this example), its set temperature, and the current time are displayed above the screen of the display section 71 of the second user terminal 70, but may be displayed in addition to the set temperature. Measurement temperature in storage room.

Next, descriptions will be provided for the change detection section 102 and the deterioration detection section 103. deal with. In step S17 shown in FIG. 4, the change detection unit 102 detects and stores (for example, in the memory unit 24) the food image received from the image management unit 101 (that is, from the storage whenever the door 20 is opened and closed). The color and size of the food image taken from the room image are changed over time.

The food stored in the refrigerating compartment 11 will gradually deteriorate, and in particular, the color of the food or the size (or shape) of the food will change. For example, fresh foods such as meat and fish become brown due to oxidation (browning). For vegetables, the bright green color of cauliflower gradually fades and the yellow parts increase. If it is further deteriorated, it will become gray-brown due to decay or mold. Leaf vegetables, such as seaweed, will wither or shrink due to evapotranspiration, causing changes in size and size.

Therefore, the change detection unit 102 detects and stores the temporal change of the color and size of the food image received each time it receives a food image from the image management unit 101. The deterioration detection unit 103 detects (estimates) the deterioration state based on the temporal change of the color and size of the food image stored in the change detection unit 102.

For example, when the general color index L * a * b * color system is used as an index to detect the color change of food, the numerical value is: the oxidation of the fresh food mentioned above is reduced by a * value (red discoloration), Cauliflower discoloration is an increase in L * value or b * value (increased lightness and yellowness), and spoilage is a decrease in L * value (decreased lightness). Therefore, the change detection unit 102 detects and stores (for example, in the memory unit 24) a * value, b * value, and L * value each time it receives a food image from the image management unit 101.

In addition, the color index is not limited to the L * a * b * color system, and other color systems can also be used to quantify color changes (red fading, yellow increase, etc.) and brightness changes, such as the RGB color system ( The primary colors are R (red), G (green), B (blue) color system), or Mancell color system.

The change detection unit 102 may store a change in the color distribution in a memory unit (for example, the memory unit 24). In this case, the change detection section 102 stores a large number of color samples in a memory section (for example, the memory section 24) in advance as a database, and detects colors and color samples from the food image received from the image management section 101. Compare to calculate similarity. Then, the entire food image is classified according to each color corresponding to the most similar color sample, and the area occupancy rate of each color is calculated.

FIGS. 8 (a) and 8 (b) are diagrams showing an example of a temporal change of a food image in the change detection unit 102. Fig. 8 (a) shows the food image when the food (meat) is put in the refrigerator (warehouse) and the color distribution detected based on the food image. FIG. 8 (b) shows a food image after a time (T hours) sufficient for the food to oxidize, and a color distribution detected based on the food image. T is a value of 0 or more. In FIGS. 8 (a) and (b), the area occupation ratio (vertical axis) of each color (horizontal axis) corresponding to the color swatch in the food image is represented by a histogram. The change detection unit 102 calculates and stores data of the color distribution of a newly received food image each time a food image is received from the image management unit 101.

In addition, the change detection unit 102 stores, for example, a projection area, a peripheral length, and a length of the longest axis as indicators for detecting a change in size. In addition, the change detection unit 102 may extract a pattern from the outline shape of the food and store the pattern. Here, the change detection unit 102 detects and stores changes in color, size, and size of the food image, but may detect and store only changes in color, or only detect and store changes in size and size.

As described above, the deterioration detection unit 103 detects the deterioration of the food based on the time change of the color and size detected and stored by the change detection unit 102. state. The detection of the deterioration state of the food by the deterioration detection unit 103 is performed at regular intervals instead of every time the door flap 20 is opened and closed.

The deterioration state of food varies depending on the type of food (meat, fish, vegetables, etc.). Therefore, it is better for the deterioration detection unit 103 to determine the deterioration state of the food using a criterion suitable for the type of food. However, the deterioration detection unit 103 may determine the deterioration state of the food by using the same determination criterion regardless of the type of the food. Therefore, the deterioration detection unit 103 uses a plurality of indicators to judge a food image. When at least one indicator exceeds a reference value, it is judged as "must be used as soon as possible".

For example, when the L * a * b * color rendering system is used as the color index, the degradation detection unit 103 exceeds at least one of the a * value, b * value, and L * value stored by the change detection unit 102. At the reference value, the food corresponding to the food image is judged as "must be used as soon as possible".

In addition, when the color distribution shown in Figs. 8 (a) and (b) is used, according to the change in the color distribution (area occupation ratio of each color corresponding to the color sample) stored in the change detection unit 102, when When the area occupancy rate of the specific color exceeds the reference value, it is determined that the food corresponding to the target food image is, for example, "must be used as soon as possible".

In the example shown in Figures 8 (a) and (b), compared to Figure 8 (a) when the food was stored, in Figure 8 (b) after T hours, the color of the food (meat) is because It becomes oxidized and becomes thicker (actually becomes brown), and the area occupancy of the thicker color becomes larger. Brown is the color exhibited by degenerate myoglobin (Mctmyoglobin) due to oxidation (ie, browning).

Therefore, based on, for example, "Browning makes the color of No. 10 The knowledge (information) such as "increasing the color corresponding to the sample", "the color corresponding to the color sample No. 20 will increase when the corruption progresses", etc., the reference value of each color reflecting the deterioration is registered in the degradation detection unit 103 in advance.

Then, the deterioration detection unit 103 judges the food when the area occupancy rate of at least one of the plural colors (for example, colors corresponding to plural color samples such as No. 10 and No. 20) registered in advance exceeds a reference value. The food corresponding to the image is "Food degradation may be in progress."

In addition, the food may wither or shrink due to deterioration. Therefore, the deterioration state can be determined based on changes in the size of the food in the food image. In this case, the deterioration detection unit 103 changes the size of the food (for example, the projected area, the length of the surrounding area, or the length of the longest axis) based on the change in the size and size of the food detected and stored by the change detection unit 102. When the reduction rate relative to the time of storage exceeds the reference value, the food can be judged as "must be used as soon as possible".

Alternatively, the deterioration detection unit 103 performs a mode ratio of the pattern of the outline shape of the food at the time of storage and the pattern of the outline shape of the food at the time of detection based on the pattern of the shape of the food detected and stored by the change detection unit 102 Yes, if the error exceeds the benchmark, the food can be judged as "must be used as soon as possible".

In addition, in the judgment based on the color distribution described with reference to Figs. 8 (a) and (b), the absolute area value may be obtained instead of the area occupation ratio of each color corresponding to the excellent sample. , Its judgment can reflect changes in color and changes in size.

The deterioration detection unit 103 determines the deterioration state of each food image stored in the change detection unit 102. The display unit 71 of the second user terminal 70 displays a food image stored in the image management unit 101, The detection result of the deterioration state obtained by the chemical detection unit 103.

Figures 9 (a) and (b) are diagrams showing examples of display of the deterioration state of the food on the mobile terminal. In the display example shown in FIG. 9 (a), among the food images displayed on the food list display screen 62 in the refrigerator described in FIG. 7, the food detected by the deterioration detection unit 103 as “must be used as soon as possible” On the image, an alarm reminder (exclamation mark in this case) is displayed superimposed. As a result, not only the terminal device used by the first user, but the second user who sees the display portion 71 of the second user terminal 70, can also know that the food that displays the food image of the alarm reminder should be as soon as possible. Used food. That is, at a glance, it is known which foods among the plurality of foods stored in the refrigerating compartment 11 should be used as soon as possible. As a result, it is possible to prevent the food from being discarded because it has expired or expired.

Therefore, for example, in a case where foods that have expired or expired in the refrigerator compartment 11 are stored, the first user or the second user (for example, the caregiver or the first user of the first user) Family members, etc.) When hesitant to buy this food at the place of shopping, not only the storage condition in the refrigerating room 11 can be confirmed, but also the expiration date or taste period of the food. Therefore, it is possible to prevent foods that are forgotten to be purchased [although they are in the refrigerator 1 but have expired and cannot be used].

In the display example shown in FIG. 9 (a), the alarm reminder is superimposed on the food image, but it is not limited to this example, and other display methods may be used, as long as it can be based on the deterioration detection unit 103 The result of the detection is to notify the first user or the second user which foods must be used as soon as possible.

In addition, as shown in the schematic diagram of FIG. 9 (b), the display section 71 of the second user terminal 70 may display a display detected by the change detection section 102. Changes in color or size of a food image. In this case, for example, when one food image is selected on the food list display screen 63 in the refrigerator as shown in FIG. 7 (for example, selected by touching the screen with a finger), the food image is displayed to indicate that A food change display screen 64 showing changes in the color or size of time (in FIG. 9 (b), the food is assumed to be raw meat, so a cooling chamber is used as a display example). Thereby, the first user and the second user can grasp the deterioration state of the food from the video.

It is assumed that the door piece 20 is not opened or closed for a long period of time. Therefore, for example, at the latest point in time when the door panel 20 is opened and closed, even if there are only slight signs of food deterioration (for example, a change toward brown), the deterioration of the food progresses with time. Therefore, the deterioration detection unit 103 detects the deterioration state of the food based on the changes in the color, size, and size of each food image stored in the change detection unit 102 at regular intervals, and predicts based on the detection results. The first user or the second user is urged to consume the food (the notification time to the first user or the second user), and when the predicted time is reached, the user will display an alarm reminder on the display portion of the mobile terminal.

Next, processing performed by the use situation acquisition unit 104 and the life management unit 105 will be described.

The use condition acquisition unit 104 detects (obtains) the storage state of the food stored in the refrigerator 11 from the information on the deterioration state of the food image unit transmitted from the deterioration detection unit 103, that is, the first place where the food is stored. User usage. Specifically, if the stored food does not appear to be degraded at all, the usage status of [with frequent replacement and proper storage of food] is detected; if there is mobile food as a whole, but certain foods are degraded, the inspection is performed. Out [have forgotten the food Existence] of use; In addition, if a lot of food has been left in a deteriorated state for several days, the use of [Not in use of the refrigerating compartment 11] is detected.

Next, the life management unit 105 receives the first information indicating the usage status of the first user detected by the usage status acquisition unit 104, and detects the life status based on the first information indicating the usage status. For example, if the food is frequently replaced, the living conditions of [the first user frequents shopping and often eats at home] are detected; if the specific food in the storage room is forgotten, it may be detected [simply forgot its existence ], Or [A living situation where the food is not used (that is, the food life is biased). In addition, if the detected usage status is the status that [the refrigerating compartment 11 is not being used], it can be assumed that the first user is out for a long period of time, or there is any safety problem.

In addition, the life management unit 105 classifies the information determined to be notified to the second user from the detected second information indicating the living condition of the first user. In the above-mentioned example, there is no problem at all in that the food in the storage room is frequently replaced, or that the first user has a proper shopping and eating situation. However, judging the information about dietary life [the food life may be biased], or especially the usage status of [not using the refrigerator 11], is related to the health problems or crimes of the first user. The related status is reported to the second user terminal 70 held by the second user different from the first user using the refrigerator 1 through the communication unit 23 as safety-related information.

Here, the life management unit 105 sets in advance the conditions for determining that the second user should be notified among the second information indicating the living conditions of the detected first user. The above-mentioned condition of [the refrigerating compartment 11 is not used at all] has been set in advance for several hours, and the threshold value to be notified to the second user is in If you are away from home due to travel or other reasons, if you remove the threshold, you can prevent unnecessary notifications, and you can report safety-related information to the second user at an appropriate time, so that the second user can The user takes appropriate action.

In addition, the second information indicating the living situation notified to the second user and obtained by the life management unit 105 is not limited to safety-related information. For example, the second piece of information that was reported was that when someone replaced the food in the storage room during a trip or during work or school daytime (such as when there should be no people at home), it might be the first use. It is possible to report information about theft prevention to a second user by using the behavior of a person other than the intended person. In this case, the second user can promptly notify the police or the security company, and the crime can be prevented in advance. In addition, the situation where the degraded food is placed in the storage room may be a situation that the first user does not notice, and the use reminder may be notified to the second user as information about eating and drinking. Notifications like this are useful information for parents or wives who manage the health of children living alone or husbands working alone from afar.

The second user holding the second user terminal 70 may be a second user authenticated in advance. Although the above information related to eating and drinking is personal information, even if it is disclosed to an unspecified third person, it will not cause a major safety problem. However, safety-related information or anti-theft information includes the health information of the first user or the information of the first user at home or away, so it is best to set it as a family member or close relative, or the first user is In the case of an elderly person, there is a second user terminal 70 that includes a caregiver and the like and can receive second information indicating the living conditions of the first user. An authentication method of the second user who holds the second user terminal 70 will be described in the fourth embodiment.

"1-3" effect

As described above, the refrigerator 1 according to the first embodiment captures and manages food images from the images in the refrigerating compartment 11 captured by the photographing device 27, and manages the food images and information about the deterioration of the food (for example, information indicating the deterioration state). ) Are displayed together on the display section 71 of the second user terminal 70. Therefore, the first user and the second user can grasp the foods stored in the refrigerating compartment 11 even at a remote place (for example, a place for shopping), and can grasp the foods which are to be used as quickly as possible.

As a result, the second user is urged to consume the food before the expiration date, and the food can be prevented from being discarded in vain. In addition, it is possible to reduce the waste of space and energy caused by placing food in the refrigerator compartment 11 for a long period of time. In addition, in a state where the door sheet 20 is closed, it is also possible to confirm the food inventory status or deterioration state with a mobile terminal, and it is possible to suppress a temperature rise in the refrigerator compartment 11 caused by the door sheet 20 opening and closing, and to reduce power consumption.

In addition, the mobile terminal displays the food image and the storage and retrieval identification information and time information, so that the first user and the second user can easily grasp how long a certain food has been placed in the refrigerating compartment 11, especially If you check at the place where you buy something, you can judge the purchase of the food after considering the expiration date or taste period of the stocked food.

In addition, the food image is stored together with the storage identification information and time information, and the deterioration status of the food is detected based on this information. Therefore, it is possible to take into account changes in the food image and the time elapsed after storage. Deterioration of food.

In addition, each time the door 20 is opened and closed, the photographing device 27 captures an image of the storage room, and the image management unit 101 captures a food image from the image of the storage room. Like, therefore, the information is updated whenever the food in the refrigerator compartment 11 is likely to change. Therefore, the first user and the second user can more accurately understand the stocked food and its deterioration state.

In addition, the deterioration detection unit 103 detects the deterioration state of the food based on changes in the color or size (or both) of the food image, so that the accuracy of detection of deterioration such as oxidation (browning) and spoilage can be improved. .

In addition, the second information indicating the living conditions of the first user who owns the refrigerator 1 can be detected from the inventory status or deterioration state of the food in the refrigerator compartment 11, so the safety-related information and theft prevention of the first user can be detected. Relevant information, or information about eating habits, etc., is reported to the limited second user as their living conditions. Therefore, it is possible to confirm the safety status of elderly parents, the security status of the family when not at home, or the health status of family members who are not living together from a remote place, and appropriate measures can be taken separately.

"1-4" Modifications

In the above description, the image management unit 101 has identified the entry or exit of food, but may also recognize the movement of the food in the refrigerator compartment 11 in addition to the entry and exit of the food.

The movement of food can be identified by, for example, a method described later. That is, as described above, the image management unit 101 obtains a differential image of the storage room image at the previous time and the current time when the door is opened and closed. Furthermore, by comparison processing, it is determined whether the difference image of the storage room image at the last time and the last time when the door is opened and closed is consistent with the difference image of the storage room image at the last time and the time when the door is opened and closed. thing. As a result of the comparison process, among the differential images of the storage room images of the last time and the last time the door was opened and closed, there are those of the storage room images of the last time and the time when the door was opened and closed. When the difference images match, it is determined that the food has moved in the refrigerator compartment 11.

When the deterioration detection unit 103 calculates the elapsed time (that is, the inventory period) after the food is stored, it also calculates the inventory period before the food moves in the refrigerating compartment 11, so that it can improve the detection of the deterioration state of the food. The accuracy is reported and reported to the second user.

In addition, in the above description, various information (the image of the refrigerating compartment 11 or the extracted food image, the change in the color and size of the food image, the deterioration state of the food, and the first use of the food through the communication unit 23) When the first information of the usage status of the user is transmitted to the second user terminal 70, only the second information that the life management unit 105 determines to indicate the detected living condition of the first user, The above-mentioned various information is transmitted to the second user terminal 70 only when the living situation is a situation that should be notified to the second user. With this, it is possible to prevent the message from being transmitted in vain, and to reduce the burden on the second user.

In the above description, the case where the imaging device 27 is installed in the refrigerating compartment 11 has been described. However, it may be installed in a storage compartment other than the refrigerating compartment 11 or may be installed in a plurality of storage compartments at the same time. For example, if a photographing device 27 is provided in the fruit and vegetable room 15 that stores vegetables with a short purchase period and is susceptible to deterioration, it is possible to grasp the frequency of deterioration in a shorter period or the frequency of shopping in living conditions. If the imaging device 27 is also installed in the camera, the frozen meat or rice and the use status of the frozen food can be detected as information related to eating and drinking.

"2" Embodiment 2

"2-1" Composition

FIG. 10 is a cross-sectional view showing a schematic structure of a refrigerator compartment 11 of a refrigerator 2 according to Embodiment 2 of the present invention (the cross-sectional structure of FIG. 1 is cut by a line segment II-II). In the In FIG. 10, the same or corresponding components as those shown in FIG. 2 are denoted by the same symbols as those shown in FIG. Fig. 11 is a block diagram schematically showing a configuration of a control system of the refrigerator 2 of the second embodiment. In FIG. 11, the same or corresponding components as those shown in FIG. 3 are denoted by the same symbols as those shown in FIG. 3.

As shown in FIGS. 10 and 11, the refrigerator 2 according to the second embodiment is different from the refrigerator 1 according to the first embodiment in that the control unit 200 includes a food information storage unit 106 and a food identification unit 107. The control section 200 may be provided on the control substrate 210, for example. The food information memory unit 106 and the memory unit 24 may be separate memory devices, or may be different memory regions in the same memory device. In addition, the food information storage unit 106 may be a storage area in an information storage unit of an external device (for example, a server) that can communicate via a network.

As shown in FIG. 11, the food identification unit 107 is connected to the image management unit 101, the change detection unit 102, and the food information storage unit 106. The food identification unit 107 receives the food image captured by the image management unit 101, and recognizes the food (for example, the type and amount of food) stored in the storage room (here, the refrigerating room 11). The food identification unit 107 refers to the information stored in the food information storage unit 106 to identify the type of food and the like. The food identification unit 107 adds identification information (information about the type and amount of food) indicating the identification result to the additional information (time information and storage identification information) of each food image received from the image management unit 101, and adds the food The image and accompanying information to which identification information is added are provided to the change detection unit 102.

The food information storage unit 106 is connected to the food identification unit 107. The food information memory unit 106 stores, for each food type, the ability to associate the food type with The characteristic attributes (such as food shape, size, color, and packaging label) of other food types are used as reference information corresponding to each food type.

FIG. 12 is a diagram showing an example of specific information (food image database) stored in the food information storage unit 106. Here, the food information memory unit 106 memorizes the characteristic quantity of the food itself belonging to the food category or the characteristic quantity of the container containing the food. Specifically, the food information memory unit 106 stores information such as the shape (contour), the size of the long axis, the representative color (red, green, and blue (RGB) value), and the label (text) of each food type as characteristics. the amount.

In addition, the shape of each food type stored in the food information storage unit 106 may be a geometric shape (for example, a cylinder, a trapezoid, etc.) that is closest to the outer shape of the food without using a contour. In this case, the food recognition unit 107 compares the food shape of the food image obtained with the food shape and the food shape stored in the food information storage unit 106 by a method such as pattern matching. As for the color of the food image, the food recognition unit 107 may use other index values such as lightness and chroma in addition to the RGB values of the representative colors.

The control unit 200 may be configured by an electric circuit, or may be configured by a processor (an information processing device that executes software (program) stored in the memory unit 24). When the control unit 200 is configured by a processor, the functions of the control unit 200 are realized by executing programs stored in the storage unit 24.

"2-2" action

Next, an example of the operation of the refrigerator 2 according to the second embodiment will be described with reference to FIGS. 10 to 13. FIG. 13 is a flowchart for explaining an operation performed when the door sheet 20 of the refrigerator 2 is opened and closed. In FIG. 13, steps which are the same as or corresponding to the steps shown in FIG. 4 (Embodiment 1) are marked with the steps shown in FIG. 4 Step number with the same step number.

In the operation shown in FIG. 13, starting from the opening and closing of the door 20, the image management unit 101 calculates the difference between the image of the storage room last time and the time when the door 20 opens and closes, thereby capturing the image of the changing part of the storage room image As the food image, it is judged the information of the time when the capturing is performed, and the food is judged to be stored in or out of the food from the food image, and the identification information of the storage in and out is stored as incidental information in the image management unit 101 with the food image (step S15). The operation is the same as that of the first embodiment shown in FIG. In the first embodiment, the image management unit 101 transmits the captured food image and incidental information to the change detection unit 102. However, in the second embodiment, the image management unit 101 transmits the captured food image and incidental information. Go to the food identification part 107 (step S21).

The food identification unit 107 refers to the information stored in the food information storage unit 106 (for example, the information shown in FIG. 12), and identifies the food stored in the storage room (here, the refrigerating room) based on the food image received from the image management unit 101. 11) Type and amount of food in step (step S22). In this identification, the food identification unit 107 compares each food image captured by the image management unit 101 among the types of foods stored in the food information storage unit 106 to determine which feature type is consistent with the type of food.

Next, when a feature quantity (a feature quantity of the food type memorized in the food information memory unit 106) is stored in the food information memory unit 106, the feature quantity that matches the captured food image is determined by the image management unit 101. The type of food in the captured image is the type corresponding to the consistent feature amount. In addition, when the food information storage unit 106 does not have a feature quantity (a feature quantity of the type of food memorized in the food information storage unit 106) that matches the captured food image, the food The identification unit 107 may replace the food type of the image captured by the image management unit 101 with the most similar food type, or treat the food of the image as a storage item of unknown type and identify only its size.

The food identification unit 107 adds the information indicating the type and amount of food identified as described above as incidental information, and attaches to the food image received from the image management unit 101 and incidental information (time information and storage identification information) about each food image. ) (Step S23), and the food image and accompanying information are provided to the change detection unit 102 as in the first embodiment (step S24).

In the next step S24, as in the first embodiment, the change detection unit 102 is based on the food image and additional information (time information, information indicating the type and amount of food, and storage identification information) received from the food identification unit 107. ) To detect and store the change in color and size of food images over time.

Here, in Embodiment 1, the deterioration detection unit 103 uses the same judgment criterion regardless of the type of food, and detects the food based on the time variation of the color and size detected by the change detection unit 102. Degraded state. That is, the deterioration detection unit 103 uses a plurality of indicators to judge a food image. When at least one indicator exceeds a predetermined reference value, it determines that it must be used as soon as possible.

However, the state of deterioration of the food varies depending on the type of food (meat, fish, vegetables, etc.). For example, for meat and fish that are also fresh foods, even if the color changes from red to brown due to oxidation (browning), it is very serious for raw fish for sashimi. Degradation, but there is no problem at all for the meat that is usually heated and prepared; in addition, for vegetables or fruits, cauliflower will fade from bright green to yellow, but oranges will change from green It turns yellow to indicate that it is ready to eat. In addition, in terms of size, leafy vegetables will wither or shrink due to evapotranspiration to make them smaller, but the contents of containers such as milk or sauce will not change regardless of how much they are deteriorated. .

Therefore, in Embodiment 2, when the deterioration detection unit 103 detects the deterioration state of the food, the information indicating the type and amount of food recognized by the food identification unit 107 is reflected, and the reference value for determining deterioration is changed. This makes it possible to improve the detection accuracy of the deterioration state of the food.

For example, when using the color distribution shown in Figures 8 (a) and 8 (b), based on "browning will increase the color corresponding to the color sample of No. 10", and "corruption will make the No. 20 color "The color increase corresponding to the color sample" is registered in advance to determine the reference value for deterioration, but each reference value is changed according to the type of food. For fish that have the same change in color when browning or spoiled (for example, catfish) ), Set the reference value for color change to a larger value.

In addition, for the same food, for example, almost whole pieces of meat can be photographed, and only pieces of meat on the surface can be photographed. Even if the area share of browning or spoilage causes color changes, the meat pieces may be deteriorating inside Therefore, reflecting the information indicating its weight, and setting the reference value smaller, it is possible to detect deterioration beforehand.

In addition, the change in size over time reflects information indicating the type of food, and excludes the food contained in the container from the deterioration criterion, thereby improving the accuracy of detecting the deterioration of the food.

"2-3" effect

As described above, according to the refrigerator 2 of the second embodiment, the information indicating the type and amount of food is reflected to improve the quality of the food in the deterioration detection unit 103. Detection accuracy of the changed state. Therefore, based on the deterioration state of the food, it is possible to make the first information indicating the usage status of the refrigerator compartment 11 detected by the usage status acquisition unit 104 and the second information indicating the life status of the first user detected by the life management unit 105. The accuracy of the information has also improved.

For example, when a food that has been degraded is ignored, and if the food is a type of food that is routinely used like cow's milk or eggs, the life management unit 105 determines that the first user is in health or safety. If there is a problem, the communication unit 23 notifies the second user terminal 70 used by the second user of the safety-related information. On the other hand, if the degraded food is a seasoning or a hobby that is used only by a specific dish, it is likely that the first user simply forgot to use it, so the urgency is reduced, and unnecessary use can be suppressed. Notification.

"3" Embodiment 3

Composition of 3-1

FIG. 14 is a cross-sectional view showing a schematic structure of a refrigerator compartment 11 of a refrigerator 3 according to Embodiment 3 of the present invention (the cross-sectional structure of FIG. 1 is cut by a line segment II-II). In FIG. 14, the same or corresponding components as those shown in FIGS. 2 and 10 are denoted by the same symbols as those shown in FIGS. 2 and 10. Fig. 15 is a block diagram schematically showing a configuration of a control system of the refrigerator 3 according to the third embodiment. In FIG. 15, the same or corresponding components as those shown in FIGS. 3 and 11 are denoted by the same symbols as those shown in FIGS. 3 and 11.

As shown in FIGS. 14 and 15, the refrigerator 3 according to the third embodiment is different from the refrigerator 2 according to the second embodiment in that it includes a schedule management unit 108 and a schedule setting unit 109. More specifically, the refrigerator 3 of the third embodiment and the implementation The difference between the refrigerator 2 in the form 2 is that a built-in schedule setting unit 109 inputs and displays a schedule (ie, an action plan) of a first user (ie, a user who actually uses the refrigerator 3) on The operation panel 21 a (the operation portion or the operation display portion provided on the door 20 of the refrigerator compartment 11; the schedule management unit 108 is installed on the control board 310 (a control device mounted on the back of the refrigerator 3) to manage the schedule setting unit The schedule input by 109. The operation panel 21a is, for example, a touch panel having an image display function and an information input function. The schedule setting unit 109 includes, for example, a schedule input button displayed on the operation panel 21a. The schedule setting unit The schedule input button of 109 may be a display (touch panel) of a mobile terminal of a first user who can communicate with the refrigerator 3. The schedule management unit 108 displays the operation panel 21a using the entered schedule. Information that can be used for calendar management of the first user.

The first user of the refrigerator 3 inputs schedule information from the operation panel 21a, which is similar to, for example, the occupant of the house where the refrigerator 3 is installed (that is, the first user of the refrigerator 3) at home or away from the house. Scheduled (whether scheduled), lifestyle of the first user (e.g., wake-up time, bedtime, attendance time, home time, meal time, etc.), shopping mode of the first user (e.g., in a week Model of the shopping cycle, the type and amount of food to be added to the refrigerator). The schedule information is displayed on the operation panel 21a as shown in FIGS. 16 to 18 described later. In the refrigerator 3 according to the third embodiment, the schedule information of the first user of the refrigerator 3 is managed in the refrigerator 3. However, a part of the functions of the control unit 300 of the refrigerator 3 may be delivered to a server on the network (the cloud server 602 in FIG. 19 described later) for execution.

The schedule management unit 108 is connected to the schedule setting unit 109 and the life management unit 105 in the operation panel 21a, and receives the date input from the operation panel 21a. The schedule information of the first user of the refrigerator 3 of the schedule setting unit 109 causes the schedule setting unit 109 to display the schedule information in a specific form (shown in FIGS. 16 to 18 described later) and provide it To the life management department 105.

16 to 18 are diagrams showing examples of a schedule display screen showing schedule information set by the schedule setting unit 109 of the refrigerator 3 of the third embodiment. 16 to 18 show examples of screens in which the schedule information of the first user that has been entered is managed by the schedule management unit 108 and displayed on the operation panel 21a. Fig. 16 shows an example of a calendar display during a month, Fig. 17 shows an example of a calendar display during a week, and Fig. 18 shows an example of a calendar display for a day.

In the third embodiment, the first user of the refrigerator 3 is a family of four, that is, the father "TARO", the mother "HANAKO", the eldest daughter "TOMOMI", and the eldest boy "one husband" "(KAZUO). The schedule display for one month in FIG. 16 shows a scheduled plan mark 91 (91a to 91e). Having a scheduled plan mark 91a "ALL" indicates that everyone in the family has a scheduled plan, and having a scheduled plan mark 91b "Tai" (TA) indicates that "Taro" has a scheduled plan and a scheduled plan mark 91c " "HANA" means that "Hanako" has a planned plan, and there is a planned plan mark 91d. "TO" means that "Hami" has a planned plan, and there is a planned plan mark 91e. "Kazu" means "one husband" There are plans.

In addition, in the weekly schedule display of FIG. 17, the scheduled plan content mark 92 (92a to 92g) is displayed. For example, the scheduled plan content flag 92a indicates a scheduled travel plan, the scheduled plan content flag 92b indicates a scheduled travel plan, the scheduled plan content flag 92c indicates a scheduled golf plan, and the scheduled plan The drawing content mark 92d indicates that there is a plan for eating out. Scheduled plan content flag 92e indicates a scheduled plan with swimming (lesson), scheduled plan content flag 92f indicates a scheduled plan with piano (lesson), scheduled plan content flag 92g indicates a scheduled plan with football (lesson) .

In addition, in the calendar display of the day in FIG. 18 (the time on the horizontal axis is the time of one day), the current time 93 is displayed; and the scheduled out-of-off period is shown in the normal period (that is, the period in which the regular repeating action pattern is presented). (Time zone) 94a, 94b; scheduled out-of-office periods 95a, 95b in periods other than the ordinary period (that is, periods that do not present a regular repeating action pattern); and bedtime 96. For example, a planned out-going period 95a during periods other than the normal period indicates that there will be a (existing) business plan, and a scheduled out-going period 95b during periods other than the normal period indicates that (existing) food will be available Schedule. In addition, an out-going period 94a scheduled during the normal period indicates that there will be a (pre-existing) job plan, and an out-going period 94b scheduled during the normal period will indicate that an (existing) school plan will be available.

"3-2" action

Next, an example of the operation of the refrigerator 3 according to the third embodiment will be described with reference to FIGS. 14 and 16. The following description will focus on differences from the first and second embodiments.

As described in Embodiments 1 and 2, the life management unit 105 detects (acquires) the first user who uses the refrigerator 3 based on the use status of the refrigerating compartment 11 obtained from the food storage condition or the deteriorated condition in the storage room. Information on your living conditions. However, it is impossible to know the habits of the first user of the refrigerator 3 or a sudden change of a planned plan from the status of food in the storage room of the refrigerator alone. The life management unit 105 determines whether the first user 2 Information may be incorrect when it should be reported to a second user (That is, cannot be judged properly).

For example, foods that are used daily, such as milk or eggs, are left in the storage room in a degraded state, and if the first user has a long-term travel plan, there is no need to provide The second user is different from the first user who uses the refrigerator 3 to report safety-related information. In addition, when the expiration date of a hobby product such as cake is approaching, if the type of food is taken into consideration in the judgment, there is no urgency because it is a hobby product, so it is determined that it is not necessary to notify the second user .

Therefore, in the third embodiment, as shown in FIG. 14, the schedule management unit 108 manages the schedule information of the first user input by the operation panel 21 a (that is, the schedule setting unit 109 of the operation panel 21 a), The schedule setting unit 109 displays the schedule on the operation panel 21 a and provides schedule information indicating the schedule to the life management unit 105. Therefore, the life management unit 105 in the third embodiment can reflect the first information in addition to the first information indicating the use situation detected by the use situation acquisition unit 104 based on the deterioration state of the food described in the first and second embodiments. The user's schedule information detects and manages second information indicating the living conditions of the first user.

The specific method of reflecting schedule information is to control the schedule of the first user shown in FIG. 16 to FIG. 18 as a unit of several days. The following is an example of controlling the situation in a month: The calendar display and the calendar display in the week of Figure 17 show that from December 19 to December 21, all the first users (family members) were away from home because of travel. The storage in the storage room Even if food is degraded, it is left alone. Therefore, even if the use condition acquisition unit 104 detects In the case of using the refrigerating room 11], the life management section 105 reflects the schedule information of [Going to travel without being at home], and therefore does not report unnecessary safety-related information to a second user, such as a family member living far away.

In addition, an example of the schedule information reflecting the unit of one day is as follows: On the day of December 25 shown in the schedule display of one day in FIG. 18, except for the scheduled time out zone 94a, 94b in the ordinary period In addition, there are scheduled out-of-office periods 95a, 95b during periods other than the normal period, in which all the first users are absent from home for almost all day. In addition, according to the plan of business trip 95a and meal 95b, it can be estimated that nobody eats dinner at home. In this situation, if the usage status acquisition unit 104 detects that the food (for example, sashimi or cake) that is close to its expiration date has not been used, the life management unit 105 reflects all the first user tonight (the whole family) (Person), and schedule information about eating outside, and notify the second user who lives nearby, etc. [the refrigerator 3 has food that is about to expire], so that the second user can consume these foods. As described above, food can be prevented from being discarded.

"3-3" effect

As described above, according to the refrigerator 3 of Embodiment 3, the life management unit 105 uses the schedule information of the first user to determine whether the second information indicating the living condition of the first user belongs to the information that should be notified to the second user. Therefore, it is possible to prevent communication waste caused by reporting second information (that is, excessive reporting of the second information) that is information that does not need to be notified to the second user, and it is necessary to prevent the second user from being leaked. The reported second information is about food waste caused by notification (that is, insufficient notification).

In addition, when the living condition of the first user is detected based on the deterioration state of the food and the schedule information of the first user, the food in the storage room can also be estimated. Continuous storage mode, or the first user ’s shopping mode. For example, when the food deteriorates as it gets closer to the weekend, when the daytime schedule is almost filled with the scheduled outing periods 94a and 94b shown in Figure 18, it can be estimated that the first user is A family where both husband and wife go to work, and it is a family with a shopping model that buys enough food once on the weekend. In this case, in particular, if the scheduled schedule information is not input during periods other than the normal period but the deterioration state changes, it is possible to predict that the refrigerator 3 may malfunction in addition to the deterioration of the food itself. Therefore, the life management unit 105 can notify the second user (for example, an electrical appliance store that has been here for many years or a customer service staff of the electrical appliance store) different from the first user who has used the refrigerator 3 to indicate the first use of the refrigerator 3. The second information of the living situation of the person.

FIG. 15 shows that the schedule information of the first user, which is inputted to the schedule setting unit 109 of the operation panel 21a and organized in the schedule management unit 108 when the living situation is detected, is transmitted. To the life management section 105. However, the target of transmission of the schedule information may be the deterioration detection unit 5 or the use situation acquisition unit 104 arranged at a stage earlier than the life management unit 105. In either case, the schedule information can be reflected when the living condition of the first user is detected. In particular, when the schedule information is transmitted to the deterioration detection unit 103, the deterioration state of the food that has occurred can be judged first. Is it accidental or necessarily caused by a change in the schedule of the first user? Therefore, the second information indicating the living condition of the first user can be appropriately notified to the second user through the communication unit 23.

"4" Embodiment 4

In a fourth embodiment of the present invention, a network system including any one of the refrigerators of the first to third embodiments will be described. Figure 19 shows the network of the fourth embodiment. Diagram of the general structure of the system. In FIG. 19, the same components as those described in the first to third embodiments are denoted by the same symbols as those used in the first to third embodiments. As shown in FIG. 19, the network system of the fourth embodiment includes an in-house system 500 and an out-of-house system 600 that can communicate with the in-house system 500 via the Internet. The in-house system 500 includes a refrigerator 4. The refrigerator 4 is any one of the refrigerators 1, 2, and 3 of Embodiments 1, 2, and 3.

As shown in FIG. 19, the in-house system 500 includes home appliances such as a refrigerator 4, an indoor air conditioner, a water heater, an IH (Induction Heating) conditioning furnace, a microwave oven, and the like. These home appliances and the power measurement device 503 are connected to the system power supply 501 via a wire 502, and power is supplied from the system power supply 501. The in-house system 500 includes an electric power measuring device 503 and an electric power measuring terminal 504 for measuring the electric power supplied from the system power source 501 to each home appliance through a wire 502, an in-house controller 505, a broadband router 506, and a device as a communication device Communication Department 515.

The power measurement device 503 measures the power supplied to each home appliance (that is, the power consumption of each home appliance) and the total power supplied from the system power supply 501 to the entire home appliance and the power measurement device 503, and stores the measured power. Results (resume). The power measurement device 503 measures the supplied power using a power measurement terminal 504 such as CT (Current Transformer).

The home appliance and the power measuring device 503 each include a device communication unit 515 for communicating with the in-house controller 505 through a communication network. The communication network is, for example, a wired or wireless LAN (Local Area Network). Household appliances such as the refrigerator 4 and the power measuring device 503 can communicate with the in-house controller 505 in a wired or wireless manner through the device communication unit 515 provided in each of them. The device communication unit 515 connected to the refrigerator 4 has functions and embodiments 1 to The communication section 23 as the communication device described in 3 is the same. Therefore, in the case where the communication unit 23 of the refrigerator 4 can directly communicate with the in-house controller 505 in a wired or wireless manner, the refrigerator 4 does not need to be provided with a machine communication unit 515.

When the device communication unit 515 has a wired communication function, the device communication unit 515 has a wired communication module such as a serial port or a driver. When the device communication section 515 has a wireless communication function, the device communication section 515 includes a wireless communication module such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The device communication unit 515 may be built in the home appliance and the power measurement device 503, respectively, or may be provided outside the home appliance and the power measurement device 503.

The in-house controller 505 can obtain information on the power consumption of the home appliances measured by the power measurement device 503 and the power supplied by the system power source 501 from the power measurement device 503 through the communication network. In addition, the in-house controller 505 can obtain information on the operation status of the home appliance through a communication network. In addition, the in-house controller 505 manages the information on the power consumption, the supply power, and the operation status that it has acquired, and can transmit control change instructions (for example, operation status change instructions) to home appliances through a communication network when necessary. .

The in-home controller 505 is connected to the Internet 601 of the out-of-home system 600 via a broadband router 506.

The Internet 601 is connected to a cloud server 602 that stores various power information and environmental information such as weather forecasts. The Internet 601 is also connected to the second user terminal 70. The second user terminal 70 is a device different from the second user of the first user of the refrigerator 4 as described in the first embodiment. For example, the second user terminal 70 is a management of a second user (such as family members, relatives, friends, acquaintances, security companies, public agencies, or care companies). (Manager) A device with a communication function, such as a smart phone, tablet, mobile phone, or personal computer, which is authorized to use it.

The cloud server 602 is an external server connected to the refrigerator 4 and the second user terminal 70 in a communicable manner via the Internet 601. The cloud server 602 preferably has a personal authentication unit 603 to authenticate whether the second user terminal 70 has authority to connect to the refrigerator 4.

For example, the cloud server 602 memorizes the mobile terminal ID of the second user terminal 70 having the authority to connect to the refrigerator 4 for the ID (identifier) of the refrigerator 4 in a built-in or externally connected memory unit. Here, the ID of the refrigerator 4 is information that can uniquely identify each of the plurality of refrigerators that can be connected to the Internet 601. In addition, the mobile terminal ID is information that can uniquely identify each mobile terminal among a plurality of mobile terminals that can be connected to the Internet 601.

Next, the personal authentication unit 603 confirms whether the mobile terminal ID of the second user terminal 70 is memorized as having authority to connect to the refrigerator 4 with the refrigerator 4 (whether it is a terminal of a registered second user), thereby, It is verified whether the second user terminal 70 has authority to connect to the refrigerator 4.

The cloud server 602, which is an external server, permits the second user terminal 70 authenticated by the personal authentication unit 603 to have authority to connect to the refrigerator 4 to connect (communication) with the refrigerator 4 through the Internet 601. That is, the cloud server 602 prohibits mobile terminals that have not been authenticated by the personal authentication unit 603 from having authorization to connect to the refrigerator 4 to connect through the Internet 601 and the refrigerator 4.

The in-house controller 505 obtains, from the electric power measurement device 503, the power consumption of the home appliance or the power supplied by the system power source 501 in the in-house system 500, In addition, it obtains information on the operating status and the surrounding environment from the home appliance. In addition, the in-home controller 505 can be connected to the Internet 601 via the broadband router 506 and obtain various power information or various environmental information from the cloud server 602.

For example, the in-house controller 505 of the in-house system 500 can obtain the power supply and demand status of the area where the in-house system 500 is located from the cloud server 602 of the out-of-house system 600 as power information. When the power consumption in the area approaches the upper limit of the supplied power, the in-house controller 505 can transmit a power saving instruction to a home appliance having a particularly large power consumption. In this case, if the refrigerator 4 is judged to be over-cooled from the set temperature of the storage room and the actual temperature history, the in-house controller 505 may instruct to increase the set temperature.

The second user terminal 70 can also be directly or indirectly connected to the refrigerator 4 through the in-home controller 505 via the Internet 601. At this time, only the second user terminal 70 authenticated by the personal authentication unit 603 of the cloud server 602 as having the authority to connect to the refrigerator 4 is permitted to connect to the refrigerator 4.

Therefore, for example, when a family member or a caregiver is a manager, if the mobile terminal ID of the second user terminal 70 is registered in advance, the first user indicating the refrigerator 4 can be shared while maintaining security. Information on your living conditions. In addition, not only the first user, but also the second user can check the stock status of the food in the storage room of the refrigerator 4 at a shopping place such as a supermarket. Therefore, it is possible to prevent people who have repeatedly bought the same food or forgot to buy necessary food. Happening.

In the fourth embodiment, all or part of the reference information referred to by the food identification unit (element 107 in Fig. 11) may be stored in the cloud server 602 as an external server. By storing the reference information of the food information memory unit (structure 106 in FIG. 11) in the cloud server 602, for example, the first application Users and users other than the second user share a large amount of data on the type of food.

Furthermore, the personal authentication device 603 described above is a device that uses an ID to authenticate connection authority, but the authentication method is not limited to this. The personal authentication unit 603 may use a method described later instead of or in combination with an ID authentication method to authenticate connection authority: for example, biometric authentication of fingerprints, veins, eye iris, etc., or password authentication, or a combination of these authentication methods.

As described above, according to the network system of the fourth embodiment, in addition to the effects obtained by the refrigerators 1 to 3 of the first to third embodiments, it is possible to strictly authenticate whether the second terminal device 70 used by the second user is a A mobile terminal for those who have permission to communicate with the refrigerator. Therefore, it is possible to surely prevent the usage status of the refrigerator of the personal information of the first user and the living status of the first user from being leaked to an unauthorized person.

Claims (16)

A refrigerator includes a storage room for storing food and a communication unit for performing communication. The refrigerator is used by a first user. The refrigerator includes: a photographing device for obtaining images in the storage room; An image management unit that extracts a food image from the image; a food condition detection unit that detects a change in the condition of the food stored in the storage room from the food image during a certain period; a use condition acquisition unit The status change of the food detected by the status detection unit obtains the first information indicating the use status of the refrigerator by the first user; and the life management unit judges the first use from the first information. When a person's living condition should be notified to a second user different from the first user, the second information indicating the living condition should be notified to the terminal device used by the second user through the communication unit. For example, the refrigerator described in item 1 of the patent application scope further includes: a memory section for memorizing information; and a time providing section for providing time information; wherein: the image management section obtains time information from the time providing section, The time information is correspondingly stored in the storage unit. The refrigerator according to item 1 of the scope of patent application, wherein the food condition detection section includes a change detection section that detects a time change of the food image; and The deterioration detection unit detects a deterioration state of the food from a time change of the food image detected by the change detection unit. The refrigerator according to item 3 of the scope of patent application, wherein the change detection unit detects a time change of the color and size of the food image. For example, the refrigerator described in any one of items 1 to 4 of the scope of patent application, further includes: a food recognition unit that recognizes the type and amount of food corresponding to the food image; the use status acquisition unit is identified based on the food recognition unit The type and quantity of the food are used to detect the usage status of the first user. For example, the refrigerator described in item 5 of the scope of patent application further includes a food information memory unit, and at least one of the shape, size, color, and packaging label of the food corresponds to the type memory of the food, and the food identification The unit refers to the information stored in the food information memory unit to identify the type and amount of food corresponding to the food image. For example, the refrigerator described in any of items 1 to 4 of the scope of patent application, further includes: a schedule management section that manages the schedule of the first user; and the life management section based on the information obtained from the use status acquisition section. The first information and the schedule information of the first user received from the schedule management unit determine whether the living condition of the first user is a situation that should be notified to the second user. For example, the refrigerator described in any one of the items 1 to 4 of the scope of patent application, wherein the life management department continues to exceed the predetermined condition when it recognizes that the preservation state of the food or the use condition of the first user has not changed. During the period, the terminal is notified of safety and security related information through the communication department. For example, the refrigerator described in item 7 of the scope of patent application, wherein: When the life management department recognizes that the preservation state of the food or the use situation of the first user has changed at least once even though the first user is not at home, through the communication department Inform the terminal device of information related to theft prevention. For example, the refrigerator described in any one of items 1 to 4 of the scope of patent application, wherein the life management unit recognizes that there is a specific characteristic in the storage state of the food or the use condition of the first user, as the first (1) When the user's living conditions, the terminal is notified of information related to eating and drinking life through the communication unit. A network system includes: the refrigerator described in any one of items 1 to 10 of the scope of patent application; and the terminal device used by the second user to be communicably connected to the refrigerator. For example, the network system described in item 11 of the scope of patent application further includes a server that is communicably connected to the refrigerator and the terminal device through the Internet. The server includes a personal authentication section, which is used for The terminal device is authenticated as a terminal permitted to communicate with the refrigerator. For example, the network system according to item 12 of the scope of the patent application, wherein the server corresponds in advance at least one of the shape, size, color, and packaging label of the food to the type memory of the food, based on the storage in the storage room. The type and amount of food are obtained with the first information indicating the use status of the first user. For example, the network system described in item 12 of the scope of patent application, wherein the server: manages the schedule of the first user; Based on the first information indicating the use status of the refrigerator by the first user and the schedule information of the first user, it is determined whether the living condition of the first user is a status that should be notified to the second user. A method for reporting a living condition is a method for reporting a living condition of a first user who uses a refrigerator, and includes the steps of: photographing a storage room of the refrigerator to obtain an image of the storage room; and obtaining the image from the storage room. A step of extracting a food image; a step of detecting a change in the condition of the food stored in the storage room from the food image in a certain period; and acquiring and indicating the first Steps of the user for the first information on the use status of the refrigerator; and when judging from the first information that the living conditions of the first user should be notified to a second user different from the first user , A step of reporting the second information indicating the living situation to the terminal device used by the second user. A living status notification program is a living status notification program for a first user. The refrigerator used by the first user has a storage room for storing food, a communication section for performing communication, and a method for obtaining images in the storage room. The photographing device, the program for reporting the living condition causes the computer to perform the processing described later: processing for causing the photographing device to take pictures of the storage room of the refrigerator to obtain images in the storage room; processing for extracting food images from the obtained images; The food image in a certain period, the food stored in the storage room is detected The processing of the change of the condition of the product; the processing of obtaining the first information indicating the use status of the refrigerator by the first user from the detected change of the condition of the food; and judging that the first information When the first user's living situation should be notified to a second user different from the first user, the process of reporting the second information indicating the living situation to the terminal device used by the second user .