WO2018020541A1

WO2018020541A1 - Refrigerator, network system provided with same, living circumstances reporting method and living circumstance reporting program

Info

Publication number: WO2018020541A1
Application number: PCT/JP2016/071697
Authority: WO
Inventors: 毅内田; 永田　滋之; 伊藤　敬
Original assignee: 三菱電機株式会社
Priority date: 2016-07-25
Filing date: 2016-07-25
Publication date: 2018-02-01
Also published as: TW201804860A; TWI751977B; JP6576560B2; JPWO2018020541A1

Abstract

A refrigerator (1) used by a first user comprises: a storage chamber that stores food; a communication unit (23) that performs communications; an imaging device (27) that acquires an image of the interior of the storage chamber; an image management unit (101) that extracts food images from the storage chamber image acquired by the imaging device (27); a food status detection unit (103a) that detects, from food images during a certain period, a change in the status of food stored in the storage chamber; a use status acquisition unit (104) that acquires first information indicating the use status of the refrigerator (1) by the first user, from changes in the status of food detected by the food status detection unit (103a); and a life management unit (105) that reports second information indicating a living status to a terminal device (70) used by a second user via the communication unit (23), if it was determined from the first information that the living status of the first user is in a state that should be reported to the second user, who is different from the first user.

Description

Refrigerator, network system including the same, living situation notification method, and living situation notification program

The present invention relates to a refrigerator having a communication function, a network system including a refrigerator, a notification method of a living condition of a user of the refrigerator, and a notification program of a living condition that causes a computer to execute a notification method.

In recent years, the proportion of elderly people (over 65 years old) in the total population is increasing in Japan. In addition, the number of people constituting one household is decreasing, and there is a tendency for one household (living alone) to increase. For this reason, confirmation of the safety of elderly people living alone and confirmation of necessity of care are necessary. By the way, the living situation of a person can be estimated from the use situation of the refrigerator of the person (user). The reason is that the usage of the refrigerator is closely related to the eating habits that are a part of the life of the refrigerator user.

According to Patent Document 1, a wireless tag is attached to food, and based on the information sent from the wireless tag and the temperature information sent from the temperature sensor in the refrigerator, based on the storage period-storage temperature-quality tolerance (TTT) theory, Discloses a technique for estimating quality degradation characteristics.

Patent Document 2 transmits image data obtained by photographing inside and outside a warehouse to a mobile phone (mobile terminal) via a server, and monitors inside and outside the warehouse from a remote place. Describes a refrigerator as a storage for goods.

JP 2006-317109 A (for example, paragraphs 0027-0042, FIG. 5) JP 2007-046833 A (for example, paragraphs 0021-0025, FIG. 1)

However, the technique described in Patent Document 1 requires a dedicated device such as a reader and lighter for a wireless tag attached to food in order to obtain information indicating the usage status of the refrigerator. There is a problem that increases.

The refrigerator described in Patent Document 2 does not require a dedicated device such as a reader and a writer, transmits image data inside and outside the refrigerator to the user's portable terminal, and displays the inside and outside of the refrigerator on the display unit of the user's portable terminal. Images can be displayed. However, this refrigerator has a problem that it cannot make a determination based on the usage status of the refrigerator (for example, a determination regarding a user's life status).

Furthermore, both

patent documents

1 and 2 describe the technique which alert | reports the user's living condition of the refrigerator to the terminal device (2nd user terminal) which other users who are persons other than a user (third party) use. Not.

This invention is made | formed in order to solve the said subject, The person other than the 1st user of a refrigerator shows the living condition of the 1st user of the refrigerator acquired from the image information of the food preserve | saved in the storage chamber. A refrigerator capable of notifying the terminal device used by the second user, a network system including the refrigerator, a notification method of the living condition of the user of the refrigerator, and a notification program of the living condition for causing the computer to execute the notification method The purpose is to provide.

The refrigerator which concerns on 1 aspect of this invention is equipped with the storage room which accommodates food, and the communication part which communicates, It is a refrigerator used by a 1st user, Comprising: The imaging device which acquires the image in the said storage room, An image management unit that extracts a food image from the image acquired by the imaging device; and a food status detection unit that detects a change in the status of the food stored in the storage chamber from the food image in a certain period; A usage status acquisition unit that acquires first information indicating a usage status of the refrigerator by the first user from a change in the status of the food detected by the food status detection unit; and the first information from the first information. When it is determined that the user's living situation is a situation to be notified to a second user different from the first user, the second information indicating the living situation is transmitted to the second user via the communication unit. The is a life management unit notifies the terminal device to be used.

A network system according to another aspect of the present invention includes the refrigerator, and the terminal device connected to the refrigerator so as to be communicable and used by the second user.

According to another aspect of the present invention, there is provided a method for informing a living situation, a step of acquiring an image in the storage room by photographing the storage room of a refrigerator used by a first user, and a food image from the acquired image. Extracting a change in the status of the food stored in the storage chamber from the food image in a certain period, and detecting the change in the status of the food detected by the first user. When acquiring the first information indicating the usage status of the refrigerator, and determining from the first information that the first user's living status should be notified to a second user different from the first user And informing the terminal device used by the second user of the second information indicating the living situation.

According to this invention, the 1st information which shows the usage condition of the refrigerator by the 1st user from the change of the situation of the food detected from the food image is acquired, and the living situation of the 1st user is given to the 2nd user from the 1st information. When it is determined that the situation should be notified, the second information indicating the living situation can be notified to the terminal device (another user terminal) used by the second user. For this reason, the 2nd user who received the 2nd information which shows the living situation of the 1st user can take appropriate measures with respect to the 1st user.

It is a front view which shows schematic structure of the refrigerator which concerns on Embodiment 1 of this invention. 2 is a cross-sectional view showing a schematic structure (a cross-sectional structure taken along line II-II in FIG. 1) of the refrigerator compartment of the refrigerator according to Embodiment 1. FIG. 3 is a block diagram schematically showing a configuration of a control system of the refrigerator according to Embodiment 1. FIG. 3 is a flowchart showing the operation of the refrigerator according to the first embodiment. (A)-(d) is a figure which shows the example of the accompanying information of the picked-up image of the refrigerator compartment of the refrigerator which concerns on Embodiment 1, the food image extracted from the picked-up image, and a food image. (A) And (b) is a figure which shows the example of the display screen in a portable terminal of the food image extracted from the picked-up image of the refrigerator compartment of the refrigerator which concerns on Embodiment 1. FIG. It is a figure which shows an example of the display screen in a portable terminal of the food image extracted from the picked-up image of the refrigerator compartment of the refrigerator which concerns on Embodiment 1. FIG. (A) And (b) is a schematic diagram for demonstrating the time change detection method of the food image in the change detection part of the refrigerator which concerns on Embodiment 1. FIG. (A) And (b) is a figure which shows the example of the display screen in the portable terminal of the food image of the refrigerator which concerns on Embodiment 1. FIG. It is sectional drawing which shows schematic structure (cross-sectional structure which cuts FIG. 1 by line segment II-II) of the refrigerator compartment of the refrigerator which concerns on Embodiment 2 of this invention. It is a block diagram which shows roughly the structure of the control system of the refrigerator which concerns on Embodiment 2. FIG. It is a figure which shows the example of the content memorize | stored in the food information storage part of the refrigerator which concerns on Embodiment 2. FIG. 10 is a flowchart for explaining the operation of the refrigerator according to the second embodiment. It is sectional drawing which shows schematic structure (cross-sectional structure which cuts FIG. 1 in line segment II-II) of the refrigerator compartment of the refrigerator which concerns on Embodiment 3 of this invention. It is a block diagram which shows roughly the structure of the control system of the refrigerator which concerns on Embodiment 3. FIG. It is a figure which shows the example of the schedule display screen (1-month display) in the operation panel (schedule setting part) of the refrigerator which concerns on Embodiment 3. FIG. It is a figure which shows the example of the schedule display screen (1 week display) in the operation panel (schedule setting part) of the refrigerator which concerns on Embodiment 3. FIG. It is a figure which shows the example of the schedule display screen (1 day display) in the operation panel (schedule setting part) of the refrigerator which concerns on Embodiment 3. FIG. It is a figure which shows the whole structure of the in-home system and the outside system to which the network system provided with the refrigerator which concerns on Embodiment 4 of this invention was applied.

<< 1 >> Embodiment 1
<< 1-1 >> Configuration FIG. 1 is a front view showing a schematic structure of a refrigerator 1 according to Embodiment 1 of the present invention. As shown in FIG. 1, the refrigerator 1 according to Embodiment 1 includes a plurality of storage rooms for storing food. The plurality of storage rooms include, for example, a refrigerator room 11, an ice making room 12, a switching room 13, a freezing room 14, and a vegetable room 15. In the example shown in FIG. 1, the refrigerator compartment 11, the ice making room 12, the switching room 13, the freezer compartment 14, and the vegetable compartment 15 are arranged in this order from the top. Further, the ice making chamber 12 and the switching chamber 13 are arranged side by side in the horizontal direction. The refrigerator 1 includes a cooling device that cools the storage chamber. The cooling device includes, for example, a refrigeration cycle circuit having a compressor, a condenser, an expansion valve, and an evaporator. The plurality of storage rooms are cooled by a refrigeration cycle circuit. FIG. 1 is also referred to in the second embodiment (FIG. 10) and the third embodiment (FIG. 14). Refrigerator 1 is described as

refrigerators

2, 3, and 4 in the second, third, and fourth embodiments, respectively.

An operation panel 21 having a display unit (display screen) as a display device may be provided on the front surface (front surface) of the door (refrigeration chamber door) 20 of the refrigerator compartment 11. On the display screen of the operation panel 21, it is possible to display storage room information (internal information) that is information related to the refrigerator 1, such as the set temperature and the current temperature of each storage room. The operation panel 21 has, for example, a touch panel that receives user input. The installation position of the operation panel 21 is not limited to the door 20 of the refrigerator compartment 11, and may be installed at other positions such as the outer surface of the door of another storage room or the side surface of the refrigerator 1. In addition, the ice making room 12, the switching room 13, the freezing room 14, and the vegetable room 15 are configured so as to be able to slide out to the front side of the refrigerator 1 together with the doors provided on the front surfaces thereof, for example. Yes. FIG. 1 shows an example of a refrigerator to which the present invention is applicable, but the refrigerator to which the present invention is applicable is not limited to that shown in FIG.

FIG. 2 is a cross-sectional view showing a schematic structure of the refrigerator compartment 11 of the refrigerator 1 according to Embodiment 1 (cross-sectional structure taken along line II-II in FIG. 1). As shown in FIG. 2, the refrigerator 1 has a door 20 that opens and closes an opening on the front side (left side in FIG. 2) of the refrigerator compartment 11. The refrigerator compartment 11 is partitioned into a plurality of spaces by a plurality of shelf boards (shelves) 16 arranged inside. The space below the lowermost shelf 16 is a chilled chamber 17. A chilled case 18 that can be drawn forward (leftward in FIG. 2) along a guide member (not shown) such as a rail is installed inside the chilled chamber 17. A cooling air passage 19 is provided on the back side of the refrigerator compartment 11 (on the right side in FIG. 2) as a passage for guiding the air cooled by the refrigeration cycle circuit to the storage compartment.

The main body of the refrigerator 1 in contact with the door 20 is provided with an opening / closing detection unit 22 as an opening / closing detection device that detects opening / closing of the door 20. The open / close detection unit 22 is, for example, a magnet-type device that detects the proximity of a magnet (permanent magnet) embedded in the door 20 by a pair of reed switches installed in the main body of the refrigerator 1. However, the open / close detection unit 22 may be a detection device that employs another detection method such as a mechanical switch or an optical switch.

The refrigerator 1 also includes a control board 110 having a control circuit (control unit) for operating the refrigerator 1. The control board 110 is provided on the back surface of the refrigerator 1, for example.

The photographing device 27 is installed on the inner surface of the door 20. In the example of FIG. 2, the imaging device 27 can capture the inside of the refrigerator compartment 11 from the door 20 side, which is the same side as the user's viewpoint, with the visual field range from the ceiling surface of the refrigerator compartment 11 to the chilled chamber 17. It has a configuration. The installation position of the photographing device 27 is not limited to the inner surface of the door 20. The imaging device 27 may be installed at any position as long as it can capture a wide range of storage items such as foods placed on the shelf board 16. Further, in consideration of the fact that the shelf plate 16 blocks the field of view, a plurality of photographing devices 27 may be installed in order to photograph more stored items. Further, the photographing device 27 may be provided at a position for photographing a storage room other than the refrigerator compartment 11.

Further, the refrigerator 1 includes a control board 110 having a control circuit for operating the refrigerator 1. The control board 110 is attached to the back surface of the refrigerator 1, for example.

FIG. 3 is a block diagram schematically showing the configuration of the control system of the refrigerator 1 according to the first embodiment. As shown in FIG. 3, the control system of the refrigerator 1 includes, for example, an operation panel 21, an open / close detection unit 22 that detects an open state and a closed state of the door 20, and a mobile terminal (terminal device) as an external device. The communication unit 23 as a communication device capable of performing wired or wireless communication, a storage unit 24 such as a semiconductor memory for storing information, a time providing unit 25 for providing time information, and the overall operation of the refrigerator 1 The control part 100 which controls this is provided. Moreover, the control system of the refrigerator 1 includes, for example, a compressor drive circuit 31 that drives a compressor (compressor) 32 that constitutes a part of the refrigeration cycle circuit, and a fan (motor) 34 that flows air into the storage chamber of the refrigerator 1. And a fan driving circuit 33 to be driven. The circuit that is the main part of the control system of the refrigerator 1 can be provided on the control board 110. However, a part of the circuit of FIG. 3 may be provided on another control board (not shown).

The communication unit 23 includes a communication module such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). The refrigerator 1 uses the communication unit 23, for example, via at least one of a router (a broadband router 506 shown in FIG. 19 described later) and the Internet (the Internet 601 shown in FIG. 19 described later), It is possible to communicate with a terminal device such as a smartphone, a mobile tablet, a mobile phone, a personal computer, or the like that exists inside or outside the house. This terminal device is, for example, a terminal device (hereinafter also referred to as “first user terminal”) used by a user of the refrigerator 1 (hereinafter also referred to as “first user” or “managed person”) and a user other than the first user. And a terminal device (hereinafter also referred to as “second user terminal” or “another user terminal”) 70 used by a third party user (hereinafter also referred to as “second user” or “manager”). The control unit 100 of the refrigerator 1 transmits information for displaying various information related to the refrigerator 1 on either or both of the display unit of the first user terminal and the display unit 71 of the second user device 70. Can do.

The time information provided by the time providing unit 25 can include not only the hour, minute, and second but also the date (month / day or year / month / day). The time providing unit 25 may be, for example, a clock or a time receiving device that receives a time signal transmitted by a time signal provider.

The control unit 100 may be configured by an electric circuit, or may be configured by a processor as an information processing apparatus that executes a program as software stored in the storage unit 24. When the control unit 100 is configured by a processor, the function of the control unit 100 is realized by executing a program stored in the storage unit 24. As described above, the function of the control unit 100 can be realized by a program (informing method or informing program for living conditions) executed by the processor in the refrigerator 1 (or 2 to 4). Further, the function of the control unit 100 can also be realized by a program (living status notification method or notification program) executed by a computer (which may be a server on a network) that can communicate with the refrigerator 1. The living situation notification program uses the refrigerator 1 (or 2 to 4) including a storage room 11 for storing food, a communication unit 23 for communication, and a photographing device 27 for acquiring images in the storage room 11. A first user's life situation notification program, which is obtained by causing the photographing device 27 to photograph the inside of the storage room 11 of the refrigerator 1 (or 2 to 4) and acquiring an image in the storage room 11 From the process of extracting the food image from the captured image, the process of detecting the change in the status of the food stored in the storage chamber 11 from the food image in a certain period, and the change in the status of the detected food, the first A process for obtaining first information indicating the usage status of the refrigerator 1 (or 2 to 4) by the user, and a situation in which the living status of the first user should be notified to the second user different from the first user from the first information It is In such a case, the computer (the information processing device such as the control unit 100 or an external computer or a server on the network) is caused to execute processing for informing the terminal device used by the second user of the second information indicating the living situation. be able to.

As shown in FIG. 3, the control unit 100 includes an image management unit 101, a change detection unit 102, a deterioration detection unit 103, a usage status acquisition unit 104, and a life management unit 105. The control unit 100 may include a main control unit (not shown) that controls the overall operation of the control unit 100. Moreover, the change detection part 102 and the deterioration detection part 103 comprise the food condition detection part 103a. In FIG. 3, the storage unit 24 is shown as a separate configuration from the image management unit 101, but the storage unit 24 is a part of the image management unit 101 (or a part of the control unit 100). It may be a part of another configuration. For example, the storage unit 24 may be a storage device connected to the outside of the refrigerator 1 or a storage device connected to be communicable via a network (for example, a server).

The image management unit 101 is connected to the imaging device 27, the change detection unit 102, and the communication unit 23. For example, when the door 20 is opened and closed, the image management unit 101 receives an entire image (image data) in the refrigerator compartment 11 acquired by photographing in the refrigerator compartment 11 by the photographing device 27 and receives the received whole image (hereinafter referred to as “store”). The image of the food part (food image) is extracted from the “inner image” or “storage room image”. Furthermore, the image management unit 101 adds time information (for example, shooting time information) to the food image extracted from the acquired whole image as supplementary information, and stores the storage unit (the storage unit 24 and the storage unit 24 in the image management unit 101). It may be a different storage unit or may be the storage unit 24). The image management unit 101 may store the food images rearranged in time series based on the time information. The image management unit 101 gives the food image and accompanying information (which may include time information and entry / exit identification information) to the change detection unit 102 and the communication unit 23.

The change detection unit 102 is connected to the image management unit 101, the deterioration detection unit 103, and the communication unit 23. The change detection unit 102 receives the food image and supplementary information from the image management unit 101, detects a change in the color and size of the food image (corresponding to the color and size of the actual food), and detects the change. In the storage unit (may be a storage unit different from the storage unit 24 or may be the storage unit 24). In addition, the change detection unit 102 provides information indicating the temporal change in the color and size of the detected food image to the deterioration detection unit 103 and the communication unit 23.

The deterioration detection unit 103 is connected to the change detection unit 102, the usage status acquisition unit 104, and the communication unit 23. The deterioration detection unit 103 receives the time change of the color and size of the food image from the change detection unit 102, detects (estimates) the deterioration state of the food based on the time change, and obtains the use state of the detection result of the deterioration state To the unit 104 and the communication unit 23. Therefore, the food condition detection unit 103a including the change detection unit 102 and the deterioration detection unit 103 can detect the storage state of the stored food from the food image extracted from the storage room image.

The usage status acquisition unit 104 is connected to the deterioration detection unit 103, the life management unit 105, and the communication unit 23 of the food status detection unit 103a. The use state acquisition unit 104 receives the deterioration state of the food in units of food images as the storage state of the food stored in the refrigerator compartment 11 from the deterioration detection unit 103, and from the information indicating the deterioration state, for example, “Current Short-term use status such as “The stored food has not deteriorated”, or “Deteriorated food has been left for several days” in combination with time information such as date or date. Detect (estimate) long-term usage. As described above, the usage status acquisition unit 104 detects the usage status by the first user of the refrigerator 1 based on the deterioration state of the food in the storage room, and also indicates information indicating this usage status (hereinafter also referred to as “first information”). ) Is given to the life management unit 105.

The life management unit 105 is connected to the usage status acquisition unit 104 and the communication unit 23. The life management unit 105 receives first information indicating the usage status of the first user based on the deterioration state of the food from the usage status acquisition unit 104, and information indicating the first user's lifestyle status (hereinafter, “ Also referred to as “second information”). The second information indicating the living state of the first user is, for example, the tendency of the first user to stay at home or away from the home of the first user, or the frequency at which the first user goes shopping. Moreover, the life management part 105 is the information (notification information required) which should be alert | reported to the 2nd user terminal 70 in the 2nd information which shows a living condition, for example, the information regarding the safety of the 1st user of the refrigerator 1, Alternatively, information related to crime prevention at the first user's house is classified and sent to the communication unit 23 as second information indicating a living situation to be notified.

The communication unit 23 is connected to the image management unit 101, the change detection unit 102, the deterioration detection unit 103, the usage status acquisition unit 104, and the life management unit 105, and is connected to the second user terminal 70 wirelessly or by wire. The The communication unit 23 receives the food image and supplementary information from the image management unit 101, receives information indicating the time change of the color and size of the food image from the change detection unit 102, and indicates the deterioration state of the food from the deterioration detection unit 103. Receive information, receive first information indicating the usage status of the first user from the usage status acquisition unit 104, and send information (second information) to be notified to the second user in the lifestyle status from the life management unit 105. receive. Each information can be transmitted to the 2nd user terminal 70 which 2nd users other than the 1st user who uses refrigerator 1 use by wireless communication, for example. Note that the second user terminal 70 that is the transmission destination of the second information may be registered in advance in the storage unit in the refrigerator 1 or in a server as a management device on the network, for example, by the first user or the like. Good.

The second user terminal 70 is connected to the communication unit 23 so as to be capable of wireless communication. The second user terminal 70 transmits the food image and supplementary information transmitted from the communication unit 23, information indicating the color and size change of the food image, information indicating the deterioration state of the food, the first user's refrigerator 1 Information to be notified to the second user among the first information indicating the usage status and the second information indicating the living status can be received, and the received content can be displayed on the display unit 71 of the second user terminal 70.

<< 1-2 >> Operation Next, an example of the operation of the refrigerator 1 according to Embodiment 1 will be described with reference to FIGS. FIG. 4 is a flowchart for explaining an operation performed each time the door 20 of the refrigerator 1 is opened and closed.

4 starts when the door 20 opens and closes. When opening / closing of the door 20 is detected by the opening / closing detection unit 22 (YES in step S11), the photographing device 27 photographs the inside of the refrigerator compartment 11 (step S12). For example, after the door 20 is closed, the imaging device 27 performs imaging while the illumination in the refrigerator compartment 11 is turned on. This is because when the opening / closing of the door 20 is detected, the food in the refrigerator compartment 11 may have been put in and out.

The imaging device 27 provides the image management unit 101 with the storage room image (preferably the image of the entire storage room) in the refrigerator compartment 11 obtained by imaging (step S13). Therefore, the image management unit 101 receives the storage room image from the imaging device 27 every time the door 20 is opened and closed (that is, every time the storage state in the refrigerator compartment 11 may change). In the following description, the current (latest) door 20 opening / closing is referred to as “current door opening / closing”, and the previous (one-time previous) door 20 opening / closing is referred to as “previous door opening / closing”.

The image management unit 101 stores the storage room image (image data) received from the imaging device 27 at the previous opening / closing of the door (for example, in the storage unit 24), and the storage received from the imaging device 27 at the current opening / closing of the door. The room image is newly saved (for example, in the storage unit 24). And the image management part 101 extracts the image of the change part of a store room image as a food image by calculating the difference of the store room image at the time of the last door opening and closing, and the store room image at the time of this door opening and closing. (Step S14). That is, the image of the changed portion of the storage room image is an image of food that has been received or delivered.

For example, in the storage room image when the door is opened and closed the last time, beer is placed on the shelf 16 of the refrigerator compartment 11, and in the storage room image when the door is opened and closed this time, the shelf where no beer is placed in the same place. In the case of only 16, the food image of beer is extracted based on the difference between the two storage room images.

The image management unit 101 adds time information that is the current time (or time when the image was taken) as supplementary information to the food image extracted in this way, The incidental information is rearranged in a time series based on the time information and stored (step S15).

At this time, the image management unit 101 determines whether the food is in or out. Specifically, when the food image is extracted, the image management unit 101 analyzes a portion changed due to the difference, for example, an edge change amount obtained by edge processing, and a storage room image at the time of photographing. The food image is compared with the prestored storage state image in the empty state (the state where no food is stored in the refrigerator compartment 11), and the food in / out is determined. Then, the image management unit 101 adds and stores the determination result (entry / exit identification information) of warehousing or evacuation to the incidental information of the food image.

Further, the image management unit 101 gives the food image and the accompanying information (time information and entry / exit identification information) to the change detection unit 102 and the communication unit 23 (step S16).

In subsequent step S17, the change detection unit 102 receives the food image and the supplementary information from the image management unit 101, and detects and stores the time change of the color and size of the food image, which will be described later. A series of operations from acquiring the storage room image of the refrigerator compartment 11 in steps S12 to S17 to giving the food image and the supplementary information to the food condition detection unit 103a is detected by opening and closing of the door 20 in step S11. It is carried out every time.

5 (a) to 5 (d) show captured images 81a to 81d of the refrigerator compartment 11 of the refrigerator 1 according to the first embodiment,

food images

82b, 82c, 82d1, and 82d2 extracted from the captured images, and food images It is a figure which shows the example of incidental information 83d1, 84d1, 83d2, 84d2. 5 (a) to 5 (d), when the door 20 is opened and closed, the storage room image which is the shooting data acquired by the shooting by the shooting device 27 and sent to the image management unit 101 and the storage room image are extracted. A food image as image data and its accompanying information (time information and entry / exit identification information) are shown.

FIG. 5A shows a storage room image 81a that is acquired by photographing by the photographing device 27 and transmitted to the image management unit 101 at time A. FIG. At time A, the door 20 has been opened and closed, but no food is received or delivered. Since there is no change in the storage room image at time A and before that, the image management unit 101 does not generate a difference image (food image).

FIG. 5B shows a storage room image 81b that is acquired by photographing by the photographing device 27 and sent to the image management unit 101 at time B after time A, and a food image 82b extracted from the storage room image 81b. The accompanying

information

83b and 84b are shown. At time B, beer, which is a canned beverage, has been delivered from the uppermost refrigerator compartment shelf in the refrigerator compartment 11. The image management unit 101 extracts the food image (beer) 82b based on the difference between the storage room image at the time A and the storage room image at the time B, and includes the supplementary information together with the food image based on the determination result of entering or leaving. The time information “time B” 83b and the entry / exit identification information 84b “departure” are stored.

FIG. 5 (c) shows a storage room image 81c acquired by photographing by the photographing device 27 and transmitted to the image management unit 101 at time C after time B, and a food image 82c extracted from the storage room image 81c. And its accompanying

information

83c and 84c. At the time C, the salad which is the food in the container has been delivered from the refrigerator compartment shelf at the bottom of the refrigerator compartment 11. The image management unit 101 extracts the food image (salad) 82c based on the difference between the storage room image at time B and the storage room image at time C, and includes the supplementary information together with the food image based on the determination result of entering or leaving. The time information “time C” 83c and the warehousing / departure identification information 84c “departure” are stored.

FIG. 5D illustrates a storage room image 81d that is captured by the imaging device 27 and transmitted to the image management unit 101 at time D after time C, and food images 82d1 and 82d2 extracted from the storage room image 81d. And the accompanying information 83d1, 84d1, 83d2, and 84d2. At time D, the milk in the paper pack is stored on the uppermost refrigerator compartment shelf in the refrigerator compartment 11, and the cake is received on the lower refrigerator compartment shelf. The image management unit 101 extracts food images (milk and cake) 82d1 and 82d2 based on the difference between the storage room image at time C and the storage room image at time D, and the determination result of entering or leaving (in this case, On the basis of “receipt” for both milk and cake, the time information “time D” 83d1, 83d2 and the entry / exit identification information 84d1, 84d2 of “entry” are stored together with the food image.

Thus, since the image management unit 101 stores the food image together with the time information and the warehousing / departure identification information, the image management unit 101 can grasp the warehousing time of the food and detect the deterioration state based on the warehousing time. Can do. That is, the image management unit 101 grasps food (stock food) stored in the refrigerator compartment 11 as an image and holds information necessary for detecting the deterioration state of the stock food.

Here, the image data stored by the image management unit 101 includes the food image (changed portion of the storage room image) extracted based on the difference from the previous image data, and the latest storage room image (refrigerated) serving as a reference for the difference. Only the whole image of the room 11) may be used. That is, after extracting the food image, the image management unit 101 may store only the latest storage room image and delete other storage room images.

Thus, since the image management unit 101 reduces the amount of stored image data, the memory capacity of the storage unit and the data communication load can be reduced. Further, the photographing by the photographing device 27, the extraction of the food image in the image management unit 101, and the storage of the image data are performed only when the door 20 is opened and closed (when there is a possibility that the food has been put in and out). Alternatively, only data necessary for managing the food stored in the refrigerator compartment 11 is stored. For this reason, it is possible to acquire the storage state and the deterioration state of the food in the refrigerator compartment 11 with the minimum necessary operations (photographing, image processing and storage).

Here, the storage state of food means information indicating which food is stored in the refrigerator compartment 11 at which time. Moreover, the deterioration state of food means information indicating how much the deterioration of food has progressed (for example, whether or not it is necessary to consume the food at an early stage).

The image management unit 101 divides the food image data stored together with the stored supplementary information (time information and storage / entry identification information) in time series or storage / departure, via the communication unit 23. And it can output to the display part 71 of the 2nd user terminal 70 which not only the 1st user terminal which the 1st user who uses the refrigerator 1 uses but 2nd users other than a 1st user uses. A second user (for example, a family or a living person of the first user who uses the refrigerator 1) is housed in the refrigerator compartment 11 by viewing the display screen of the display unit 71 of the second user terminal 70. You can grasp the food.

6 (a) and 6 (b) are diagrams illustrating an example of a display screen on a mobile terminal of a food image extracted from a photographed image of the refrigerator 1 according to the first embodiment. In the display screen (timeline display screen) shown in FIG. 6A, the food image 42 extracted from the storage room image each time the door 20 is opened and closed, the time information 43 attached to the food image 42, and the input information. The shipping identification information 44 is displayed side by side in time series. In this example, food images 42 are arranged in the center of the screen from the top (in order from the latest shooting time information as time information), time information 43 on the left side of each food image, and entry / exit identification information 44 on the right side. Is displayed. Further, on the upper side of the screen, a target storage room (in this example, the refrigerating room 11), its set temperature, and the current time are also displayed.

In the timeline display screen shown in FIG. 6 (a), it is possible to check the expiry date or the expiry date by searching from the top to the bottom (going back in time for the target food). is there. For this reason, it is possible to prevent wasteful disposal of food expenses by informing the consumption period or best-before date of food as food-related information and urging consumption.

The display screen (incoming / outgoing list display screen) shown in FIG. 6B is divided into left and right fields for warehousing and warehousing, and food images and time information are displayed in chronological order in each field (list display). is doing. In other words, in the left column, the food images whose entry / exit identification information 54 is “Receive” and the time information (that is, the entry time) 53 are displayed side by side in order from the latest entry time 53. . In the right column, the food images 52 whose entry / exit identification information 54 is “exit” and the time information (that is, the exit time) 53 are displayed side by side in order from the latest exit time 53.

In the loading / unloading list display screen shown in FIG. 6 (b), the current remaining amount of the food can be acquired from the history of loading and unloading of the target food. By notifying the terminal 70 of the remaining amount of food, a family member or a helper confirms the display unit 71 of the other user terminal 70 at a remote place, for example, a shopping destination, so that excessive purchase of food or purchase of food is performed. It becomes possible to prevent forgetting.

FIG. 7 is a diagram illustrating another example of the display screen on the mobile terminal of the food image extracted from the photographed image of the refrigerator 1 according to the first embodiment. The image management unit 101 uses the stored food image (that is, the food image extracted every time the door 20 is opened / closed) and the time information and the warehousing / departure identification information as incidental information at a certain time. The food image stored in (the entry / exit identification information is not “exit”) is specified. Then, as shown in FIG. 7, a list of stock food images is displayed as a stock food list display screen 62 via the communication unit 23. In the example of FIG. 7, unlike FIGS. 6A and 6B, the food images are arranged from the top in order of the warehousing time based on the time information.

In the inventory food list display screen 62 shown in FIG. 7, since a list of inventory food images at a certain point in time is displayed, the food as food-related information is more clearly defined than the list display shown in FIG. The remaining amount list is notified. For this reason, it can grasp | ascertain which food is stored in the refrigerator compartment 11, and can expect further the effect of being able to prevent over buying of food and forgetting to buy food. In addition, since food images are arranged from the top in order of the warehousing time, it is easy to find old foods, that is, foods that "need to be consumed early", and suppress the disposal of foods due to expiration of consumption It becomes possible.

In addition, the display unit 71 of the second user terminal 70 is referred to as an “entrance / exit timeline” in FIG. 6A, an “entry / exit list” in FIG. 6B, and an “inventory food list” in FIG. Although three types of food image display screens have been described, any one of these may be displayed at all times, or may be displayed in response to a first user operation (for example, touching the touch panel). Also good. The food image display screen may be switched by operating a button linked to each display screen, or may be switched by a horizontal swipe operation.

Further, on the upper side of the screen of the display unit 71 of the second user terminal 70, the target storage room (in this example, the refrigerator room 11), its set temperature, and the current time are displayed. In addition, the measured temperature of the storage room may be displayed.

Next, processing of the change detection unit 102 and the deterioration detection unit 103 will be described. In step S <b> 17 shown in FIG. 4, the time of the color and size of the food image received by the change detection unit 102 from the image management unit 101 (that is, the food image extracted from the storage room image every time the door 20 is opened and closed). The change is detected (for example, stored in the storage unit 24) and saved.

The food stored in the refrigerator compartment 11 gradually deteriorates, and particularly changes in the color of food or the size (or shape) of food. For example, fresh foods such as meat and fish change color to brown due to oxidation (methation). As for vegetables, broccoli gradually fades from bright green color to increase the yellow part, and when it further deteriorates, it changes to grayish brown due to decay or mold. Leafy vegetables such as spinach are deflated or shrunken by transpiration, and the size also changes.

Therefore, every time a change detection unit 102 receives a food image from the image management unit 101, the change detection unit 102 detects and stores a time change in the color and size of the received food image. Based on the temporal change in the color and size of the food image stored in the change detection unit 102, the deterioration detection unit 103 detects (estimates) the deterioration state.

As an index for detecting a change in the color of food, for example, when the L ^* a ^* b ^* color system, which is a general color index, is used, the above-described oxidation of fresh food reduces the a ^* value (red Fading), broccoli fading is quantified as an increase in L ^* value or b ^* value (increased brightness and yellow), and decay is a decrease in L ^* value (decreasing brightness). Therefore, every time the food image is received from the image management unit 101, the change detection unit 102 detects and stores the a ^* value, the b ^* value, and the L ^* value (for example, in the storage unit 24).

In addition, the color index is not limited to the L ^* a ^* b ^* color system, but the RGB color system or Munsell table, which is a color system in which the primary colors are R (red), G (green), and B (blue). Even if another color system such as a color system is used, a color change (red fading, yellow increase, etc.) and a brightness change can be quantified.

The change detection unit 102 may store the change in color distribution in a storage unit (for example, the storage unit 24). In this case, the change detection unit 102 stores a large number of color samples in advance in a storage unit (for example, the storage unit 24) as a database, and compares the color detected from the food image received from the image management unit 101 with the color sample. Then, the similarity is calculated. Then, the entire food image is classified for each color corresponding to the most similar color sample, and the area occupancy for each color is calculated.

FIGS. 8A and 8B are diagrams illustrating an example of a time change of the food image in the change detection unit 102. FIG. FIG. 8A shows a food image at the time of receipt of food (meat) and a color distribution detected based on the food image. FIG. 8B shows a food image after a sufficient time (T time) for the food to oxidize and a color distribution detected based on the food image. T is a value of 0 or more. In FIGS. 8A and 8B, the area occupancy (vertical axis) for each color (horizontal axis) corresponding to the color sample in the food image is shown by a bar graph. Each time the change detection unit 102 receives a food image from the image management unit 101, the change detection unit 102 calculates and stores data of the color distribution of the newly received food image.

Also, the change detection unit 102 stores, for example, a projection area, a perimeter, a length of the longest axis, and the like as an index for detecting a change in size. Further, the change detection unit 102 may extract a pattern from the outline shape of the food and store the pattern. Here, the change detection unit 102 detects and stores changes in the color and size of the food image. However, only the change in color or only the change in size may be detected and stored.

In this way, the deterioration detection unit 103 detects the deterioration state of the food based on the temporal change in the color and size detected and stored by the change detection unit 102. The detection of the food deterioration state by the deterioration detection unit 103 is preferably performed not at every opening / closing of the door 20 but at every predetermined time.

The state of deterioration of food varies depending on the type of food (meat, fish, vegetables, etc.). Therefore, it is desirable that the deterioration detection unit 103 determines the deterioration state of the food using a determination criterion suitable for the type of food. However, the deterioration detection unit 103 may determine the deterioration state of the food using the same determination criterion regardless of the type of food. Therefore, the deterioration detection unit 103 makes a determination based on a plurality of indexes for one food image, and determines that “it is necessary to consume at an early stage” when at least one index exceeds a reference value.

For example, when the L ^* a ^* b ^* color system is used as the color index, the deterioration detection unit 103 uses at least one of the a ^* value, the b ^* value, and the L ^* value stored in the change detection unit 102. When one of the foods exceeds the reference value, it is determined that the food corresponding to the food image needs to be consumed at an early stage.

Further, when the color distribution shown in FIGS. 8A and 8B is used, based on the change in the color distribution held by the change detection unit 102 (area occupancy for each color corresponding to the color sample), When the area occupancy of the specific color exceeds the reference value, for example, it is determined that “the food needs to be consumed early” for the food corresponding to the target food image.

In the example shown in FIGS. 8A and 8B, the color of the food (meat) is oxidized due to oxidation in FIG. It changes dark (actually brown), and the area occupancy of the dark color increases. Brown is the color that metmyoglobin exhibits upon oxidation or methation.

Therefore, for example, based on knowledge (information) such as “the color corresponding to the number 10 color sample increases as the metrification proceeds” and “the color corresponding to the number 20 color sample increases as the decay proceeds”. The reference value is registered in advance in the deterioration detection unit 103 for each color that reflects the deterioration.

Then, the deterioration detection unit 103 determines that at least one of the area occupancy rates of a plurality of colors registered in advance (for example, colors corresponding to a plurality of color samples such as No. 10 and No. 20) exceeds a reference value. In addition, it is determined that the food corresponding to the food image “there is a possibility that the food has deteriorated”.

In addition, since the food is deflated or shrunken due to deterioration, the deterioration state can be determined based on a change in the size of the food included in the food image. In this case, based on the change in the size of the food detected and stored by the change detection unit 102 (for example, the projected area, the peripheral length, or the length of the longest axis), the deterioration detection unit 103 determines whether the size of the food is When the rate of decrease from the time of warehousing exceeds the reference value, it can be determined that the food needs to be consumed early.

Alternatively, based on the food contour pattern detected and stored by the change detection unit 102, the deterioration detection unit 103 performs pattern matching between the food contour shape pattern at the time of warehousing and the food contour shape pattern at the time of detection. If the error exceeds the reference time, it may be determined that the food needs to be consumed early.

Further, in the determination based on the color distribution described with reference to FIGS. 8A and 8B, if an absolute area value is obtained instead of obtaining the area occupancy for each color corresponding to the color sample, the color It is possible to make a judgment that reflects the change in size at the same time.

The deterioration detection unit 103 determines a deterioration state for each food image stored in the change detection unit 102. The display unit 71 of the second user terminal 70 displays the detection result of the deterioration state by the deterioration detection unit 103 together with the food image stored by the image management unit 101.

FIGS. 9A and 9B are diagrams showing display examples of the deterioration state of food on the mobile terminal. In the display example shown in FIG. 9A, among the food images displayed on the stock food list display screen 62 described with reference to FIG. 7, the food that the deterioration detection unit 103 has determined “need to be consumed early”. An alert (here, an exclamation point) is superimposed on the image. As a result, not only the terminal device used by the first user but also the second user who has seen the display unit 71 of the second user terminal 70 is a food that should be consumed at an early stage. I can understand that there is. That is, it becomes obvious at a glance which food should be consumed early among the plurality of foods stored in the refrigerator compartment 11. As a result, it is possible to suppress wastefully discarding food due to the expiration of the expiration date or the expiration date.

Therefore, for example, food in which the expiration date or expiration date has passed is stored in the refrigerator compartment 11, and the first user or the second user (for example, the first user's helper or the first user's family). When he / she decides whether or not to purchase the food at the shopping destination, not only the storage status in the refrigerator compartment 11 but also the expiration date or the expiration date of the target food can be confirmed. For this reason, it is possible to prevent the user from forgetting to purchase food that is stored in the refrigerator 1 but has expired and cannot be used.

In the display example shown in FIG. 9 (a), an alert is superimposed on the food image. However, the present invention is not limited to such an example, and based on the detection result of the deterioration detection unit 103, the food to be consumed early is displayed. Any other display may be used as long as it can inform one user or the second user.

The display unit 71 of the second user terminal 70 also displays a food image indicating a change in color or size of the food image detected by the change detection unit 102, as schematically shown in FIG. 9B. May be. In this case, for example, when one food image is selected on the stock food list display screen 63 as shown in FIG. 7 (for example, a finger is touched on the screen to select), the color of the food image for each time Or the food change display screen 64 which displays the change of a magnitude | size is displayed (In FIG.9 (b), since the raw meat is assumed as a foodstuff, the chilled room is taken as the example of a display). Thereby, the 1st user and the 2nd user can grasp | ascertain the deterioration state of foodstuff from an image.

It is assumed that the door 20 is not opened and closed for a long period. Therefore, for example, at the latest time point of opening and closing the door 20, even if there is a slight sign of deterioration of the food (for example, change to brown), the deterioration of the food may progress over time. Accordingly, the deterioration detection unit 103 detects the deterioration state of the food based on changes in the color and size of each food image stored in the change detection unit 102 at regular intervals, and based on the detection result, Predicting the time when one user or the second user should consume food (notifying time to the first user or the second user) and displaying an alert on the display unit of the mobile terminal when the predicted time is reached. desirable.

Next, processing of the usage status acquisition unit 104 and the life management unit 105 will be described.

The use status acquisition unit 104 determines the storage state of the food stored in the refrigerator compartment 11, that is, the first user storing the food from the information on the deterioration state of the food image unit transmitted from the deterioration detection unit 103. Detect (acquire) usage status. Specifically, if there is no deterioration in the currently stored food, it is detected that the food is always replaced and stored properly, and there is a movement of the food as a whole. The usage situation that the existence of the food is forgotten if the food has deteriorated, and the usage situation that the refrigerator compartment 11 itself is not used if many foods have been left for several days with deterioration. Is detected.

Next, the life management unit 105 receives the first information indicating the usage status of the first user detected by the usage status acquisition unit 104, and detects the living status based on the first information indicating the usage status. For example, if food is constantly being replaced, if the first user frequently goes shopping and eats at home, or if the presence of a particular food in the storage room is forgotten, simply In addition to forgetting the existence, it is also possible to detect a life in which the food is not used, that is, a living situation in which there is a possibility of having an unbalanced diet. Further, when a situation in which the refrigerator compartment 11 itself is not used is detected as the usage situation, there is a possibility that the first user has been out for a long period of time or that some safety problem has occurred. is assumed.

Also, the life management unit 105 classifies information determined to be notified to the second user from the detected second information indicating the life status of the first user. In the above example, there is no problem at all in the living situation where the food in the storage room is always replaced and the first user is assumed to be shopping and eating properly. However, eating habits related information that the eating habits may be biased, or a usage situation in which the refrigerator compartment 11 itself is not used is determined as a situation related to health problems or crimes of the first user. When it does, it notifies to the 2nd user terminal 70 which the 2nd user different from the 1st user who uses the refrigerator 1 has via the communication part 23 as safety related information.

Here, in the life management unit 105, conditions for determining that the second user should be notified in the second information indicating the detected life status of the first user are set in advance. When the above-described situation in which the refrigerator compartment 11 is not used at all continues for a certain number of hours, a threshold value for determining whether to notify the second user is set in advance. If it is made to cancel | release, useless alerting | reporting can be prevented, a safety related information can be alert | reported to a 2nd user at an appropriate timing, and a 2nd user can take an appropriate measure with respect to a 1st user. It is.

Further, the second information indicating the life situation acquired by the life management unit 105 and notified to the second user is not limited to the safety related information. For example, the second information to be notified is, for example, when a food change in the storage room occurs during a trip or during work or school on a weekday daytime (at a time when no one should be at home) In consideration of use by an unexpected person other than the first user, the crime prevention related information may be notified to the second user. In this case, the second user can promptly report to the police or security company, and can prevent crimes. Furthermore, in consideration of the situation in which the first user does not notice the situation in which the deteriorated food is left in the storage room, a usage warning can be notified to the second user as dietary related information. Such notification is useful information for parents or wives who manage the health status of a single child or a husband who lives alone from a remote location.

Further, the second user who owns the second user terminal 70 may be a second user who has been authenticated in advance. Although the above-mentioned dietary information is personal information, it is assumed that it will not be a big problem in terms of security even if it is disclosed to unspecified third parties. However, since the safety related information or the crime prevention related information includes the health information of the first user or the home or absence information of the first user, a helper or the like when the first user is an elderly person The 2nd user terminal 70 which can receive the 2nd information which shows the living situation of the 1st user beforehand is set up beforehand. Note that an authentication method for the second user who owns the second user terminal 70 will be described in a fourth embodiment.

<< 1-3 >> Effect As described above, according to the refrigerator 1 according to the first embodiment, the food image is extracted and managed from the image in the refrigerator compartment 11 photographed by the photographing device 27, and the second user terminal The display unit 71 displays a food image and information related to the deterioration of the food (for example, information indicating the deterioration state). Therefore, not only the first user but also the second user can grasp the food stored in the refrigerator compartment 11 even in a remote place (for example, a shopping destination), and can grasp the food to be consumed particularly quickly among them. it can.

As a result, the second user can be encouraged to consume the food by the expiration date, and wasteful disposal of the food can be suppressed. In addition, waste of space and energy due to leaving food in the refrigerator compartment 11 for a long period of time can be reduced. In addition, even when the door 20 is closed, the food stock status or deterioration state can be confirmed on the mobile terminal, so that the temperature rise in the refrigerator compartment 11 due to the opening and closing of the door 20 is suppressed, and the power consumption is reduced. can do.

In addition, by displaying the food entry / exit identification information and the time information together with the food image on the portable terminal, the first user and the second user can easily determine how long a certain food has been stored in the refrigerator compartment 11. In particular, in the case of confirmation from a shopping destination, the purchase of food can be determined in consideration of the expiration date or expiration date of the stock food.

In addition, the food image is stored together with the loading / unloading identification information and time information, and the deterioration state of the food is detected based on the information. Therefore, the food image is changed and the time elapsed since the loading is taken into consideration. Can be detected with high accuracy.

Further, every time the door 20 is opened and closed, the imaging device 27 captures a storage room image, and the image management unit 101 extracts a food image from the storage room image, so that the food in the refrigerator compartment 11 may change. Information is updated every time Therefore, the 1st user and the 2nd user can know stock food and its deterioration state more correctly.

Further, since the deterioration detection unit 103 detects the deterioration state of the food based on the change in the color and / or size (or both) of the food image, the detection accuracy of deterioration such as oxidation (methanization) and spoilage is increased. Can do.

Furthermore, the 2nd information which shows the living condition of the 1st user who owns the refrigerator 1 can be detected from the stock condition or the deterioration state of the foodstuffs in the refrigerator compartment 11, and the 1st user's safety relation is detected as the living condition. Information, crime prevention related information, eating habits related information or the like is notified to a limited second user. For this reason, it is possible to confirm the safety status of elderly parents, the crime prevention status of an answering house, the health status of a family living away from a remote location, etc., and it is possible to take appropriate countermeasures for each. It is done.

<< 1-4 >> Modifications In the above description, the image management unit 101 identifies food warehousing or warehousing, but in addition to food warehousing and warehousing, food movement in the refrigerator compartment 11 is identified. Also good.

Food movement is identified by the following method, for example. That is, as described above, the image management unit 101 obtains a difference image between the storage room images at the previous and current door opening and closing times. Furthermore, it is determined by matching processing whether there is a difference image of the storage room image at the time of opening and closing the previous door and the previous time and the difference image of the storage room image at the time of opening and closing the door of the previous time and this time. . As a result of the matching process, if any of the difference images of the storage room images at the previous and previous door opening / closing coincides with the difference images of the storage room images at the previous and current door opening / closing, the food is refrigerated. It is determined that it has moved in the chamber 11.

When the deterioration detection unit 103 calculates the elapsed time from the receipt of the food (that is, the inventory period), the inventory period before the food is moved in the refrigerator compartment 11 is also included, so that the deterioration state of the food can be detected. The accuracy of notification to two users can be improved.

Further, in the above description, various information (images of the refrigerator compartment 11 or extracted food images, changes in color and size of food images over time, food deterioration states, first user information indicating the usage status of the first user). ) Is transmitted to the second user terminal 70 via the communication unit 23. In the life management unit 105, the life status is detected in the second information indicating the detected life status of the first user. The various types of information may be transmitted to the second user terminal 70 only when it is determined that the situation should be notified to the second user. Thereby, useless transmission can be prevented and the burden on the second user can be reduced.

In the above description, the photographing device 27 is provided in the refrigerator compartment 11, but it may be provided in a storage room other than the refrigerator compartment 11, or may be installed in a plurality of storage rooms simultaneously. For example, if the photographing device 27 is installed in the vegetable room 15 in which vegetables that have a relatively short purchase cycle and are likely to deteriorate, the frequency of shopping can be grasped in a shorter period of time as the occurrence of deterioration or the living situation. Moreover, if it is additionally provided in the freezer compartment 14, it is possible to detect the use status of frozen meat or rice and frozen food as dietary related information.

<< 2 >> Embodiment 2
<< 2-1 >> Configuration FIG. 10 is a cross-sectional view showing a schematic structure (a cross-sectional structure taken along line II-II in FIG. 1) of refrigerator compartment 11 of refrigerator 2 according to Embodiment 2 of the present invention. 10, components that are the same as or correspond to the components shown in FIG. 2 are assigned the same reference numerals as those shown in FIG. FIG. 11 is a block diagram schematically showing the configuration of the control system of the refrigerator 2 according to the second embodiment. In FIG. 11, the same reference numerals as those shown in FIG. 3 are given to the same or corresponding elements as those shown in FIG.

As shown in FIGS. 10 and 11, the refrigerator 2 according to the second embodiment is different from the refrigerator 1 according to the first embodiment in that the control unit 200 includes a food information storage unit 106 and a food identification unit 107. Is different. The control unit 200 is provided on the control board 210, for example. The food information storage unit 106 and the storage unit 24 may be separate storage devices, or may be different storage areas of the same storage device. The food information storage unit 106 may be a storage area of an information storage unit of an external device (for example, a server) that can communicate via a network.

As shown in FIG. 11, the food identification unit 107 is connected to the image management unit 101, the change detection unit 102, and the food information storage unit 106. The food identification unit 107 receives the food image extracted by the image management unit 101, and identifies the food (for example, the type and amount of food) stored in the storage room (here, the refrigerator room 11). The food identification unit 107 refers to information stored in the food information storage unit 106 in order to identify the type of food. The food identification unit 107 adds identification information (information about the type and amount of food) indicating the identification result to each supplementary information (time information and entry / exit identification information) of the food image received from the image management unit 101. The change detection unit 102 is provided with the food image and the incidental information to which the identification information is added.

The food information storage unit 106 is connected to the food identification unit 107. The food information storage unit 106 has, for each type of food, a characteristic attribute for distinguishing the type of food from other types of food (for example, at least one of the shape, size, color, and exterior label of the food) Is stored as reference information associated with the type of food.

FIG. 12 is a diagram showing an example of the specific information (food image database) stored in the food information storage unit 106. Here, the food information storage unit 106 stores the characteristic amount of the food itself belonging to the type of food or the characteristic amount of the container containing the food in association with the type of food. Specifically, the food information storage unit 106 features information such as a shape (contour), a major axis size, a representative color (red green blue (RGB) value), and a label (character) for each type of food. I remember it as a quantity.

As the shape of each type of food stored in the food information storage unit 106, a geometric shape (for example, a cylinder, a trapezoid, etc.) closest to the outer shape of the food may be used instead of the contour. In this case, the food identification unit 107 collates the food shape of the acquired food image with the food shape stored in the food information storage unit 106, for example, by a method such as pattern matching. Moreover, you may make it the food identification part 107 use other index values, such as brightness or a saturation, for example other than the RGB value of a representative color about the color of a food image.

Further, the control unit 200 may be configured by an electric circuit, or may be configured by a processor as an information processing device that executes a program as software stored in the storage unit 24. When the control unit 200 is configured by a processor, the function of the control unit 200 is realized by executing a program stored in the storage unit 24.

<< 2-2 >> Operation Next, an example of the operation of the refrigerator 2 according to Embodiment 2 will be described with reference to FIGS. 10 to 13. FIG. 13 is a flowchart for explaining an operation performed each time the door 20 of the refrigerator 2 is opened and closed. In FIG. 13, steps that are the same as or correspond to the steps shown in FIG. 4 (Embodiment 1) are assigned the same step numbers as the step numbers shown in FIG.

In the operation shown in FIG. 13, starting from opening / closing of the door 20, the image management unit 101 calculates the difference between the storage room images when the door 20 is opened / closed last time and this time. The image is extracted as a food image, the time information when the extraction is performed, and the food image is determined based on the time of the extraction, and the food identification information is stored in the image management unit 101 as supplementary information together with the food image. Up to (Step S15), the operation is the same as that of the first embodiment shown in FIG. In the first embodiment, the image management unit 101 transmits the extracted food image and the accompanying information to the change detecting unit 102. However, in the second embodiment, the image management unit 101 uses the extracted food image and the accompanying information as the food. It gives to the identification part 107 (step S21).

The food identification unit 107 refers to the information stored in the food information storage unit 106 (for example, the information shown in FIG. 12), and based on the food image received from the image management unit 101, the storage room (here, The type and amount of food stored in the refrigerator compartment 11) are identified (step S22). In this identification, the food identification unit 107 selects the feature value of which food type among the types of food stored in the food information storage unit 106 for each food image extracted by the image management unit 101. Check for a match.

If the food information storage unit 106 includes a feature amount (feature type feature amount stored in the food information storage unit 106) that matches the extracted food image, the food identification unit 107 It is determined that the type of food in the image extracted by the management unit 101 is a type associated with the matched feature amount. If there is no feature amount (feature type feature amount stored in the food information storage unit 106) that matches the extracted food image in the food information storage unit 106, the food identification unit 107 The food type of the image extracted by the management unit 101 may be replaced with the most similar food type, or the food of this image may be configured to identify only the size as a stored item of unknown type. Good.

The food identification unit 107 indicates the type and amount of the food identified in this way in the food image received from the image management unit 101 and the incidental information (time information and entry / exit identification information) for each food image. Information is added as incidental information (step S23), and the food image and incidental information are given to the change detection unit 102 as in the case of the first embodiment (step S24).

In the subsequent step S24, the change is made in the same manner as in the first embodiment, based on the food image and incidental information (time information, information indicating the type and amount of food, and entry / exit identification information) received from the food identification unit 107. The detection unit 102 detects and stores temporal changes in the color and size of the food image.

Here, in the first embodiment, based on the time change of the color and size detected by the change detection unit 102, the deterioration detection unit 103 uses a uniform judgment criterion regardless of the type of food. A deterioration state was detected. That is, the deterioration detection unit 103 makes a determination based on a plurality of indices for one food image, and if the at least one index exceeds a predetermined reference value, the target food is “consumed early”. It is necessary ”.

However, the deterioration state of food varies depending on the type of food (meat, fish, vegetables, etc.). For example, meat and fish, which are the same fresh food, are generally cooked, even if the color changes from red to brown due to oxidation (metoization), even if the fish eaten raw as sashimi is severely degraded It is a level of degradation that is not a problem for meat, and in vegetables or fruits, in broccoli, bright green fades to yellow, but in tangerines, green turns yellow when it is ripe and ready to eat. It may indicate that. In addition, the leafy vegetables are deflated or shrunk by transpiration and become smaller, but the size of what is in the container, such as milk or dressing, does not change no matter how much it deteriorates.

Therefore, in the second embodiment, when the deterioration detection unit 103 detects the deterioration state of the food, the criterion for judging deterioration reflecting the information indicating the type and amount of the food identified by the food identification unit 107 is used. The value has been changed. Thereby, the detection accuracy of the deterioration state of food can be improved.

For example, when the color distributions shown in FIGS. 8A and 8B are used, “the color corresponding to the 10th color sample increases as metation progresses”, “the 20th when corruption progresses”. Based on the knowledge such as `` the color corresponding to the color sample increases '', the reference value for judging deterioration is registered in advance, but the reference value is changed depending on the type of food, and it is the same by methodization or decay The reference value for the color change is set larger than that of the fish whose color changes (for example, tuna).

In addition, the same food, for example, thin sliced meat that can be photographed almost entirely, and block meat that can only photograph the surface, even if the area occupancy that has changed color due to methodization or spoilage is the same, the block meat also undergoes internal deterioration Therefore, the deterioration can be detected in advance by setting the reference value to be small by reflecting the information indicating the amount.

In addition, the change in size over time reflects information indicating the type of food, and the food contained in the container is excluded from the criteria for judging deterioration, thereby improving the detection accuracy of the food deterioration state. be able to.

<< 2-3 >> Effect As described above, according to the refrigerator 2 according to the second embodiment, the detection accuracy of the deterioration state of the food in the deterioration detection unit 103 is reflected by reflecting the information indicating the type and amount of the food. Will improve. Therefore, based on the deterioration state of the food, the first information indicating the usage status of the refrigerator compartment 11 detected by the usage status acquisition unit 104 and the life status of the first user detected by the life management unit 105 are displayed. The precision of the 2nd information to show can be improved.

For example, even when a situation in which deteriorated food is left is detected, if the food is of a kind that is used on a daily basis, such as milk or egg, the life management unit 105 first It is determined that a health or safety problem has occurred in the user, and the safety related information is notified to the second user terminal 70 used by the second user via the communication unit 23. On the other hand, when the deteriorated food is a seasoning or a luxury product that is used only in a specific dish, it is highly likely that the first user has forgotten to use it, so the urgency is reduced, It is possible to suppress useless notification.

<< 3 >> Embodiment 3
<< 3-1 >> Configuration FIG. 14 is a cross-sectional view showing a schematic structure (a cross-sectional structure taken along line II-II in FIG. 1) of refrigerator compartment 11 of refrigerator 3 according to Embodiment 3 of the present invention. 14, components that are the same as or correspond to the components shown in FIGS. 2 and 10 are given the same reference numerals as those shown in FIGS. FIG. 15 is a block diagram schematically showing the configuration of the control system of the refrigerator 3 according to the third embodiment. 15, components that are the same as or correspond to the components shown in FIGS. 3 and 11 are given the same reference numerals as those shown in FIGS.

14 and 15, the refrigerator 3 according to the third embodiment is different from the refrigerator 2 according to the second embodiment in that a schedule management unit 108 and a schedule setting unit 109 are provided. More specifically, the refrigerator 3 according to the third embodiment is a user who actually uses the refrigerator 3 for an operation panel 21a as an operation unit or an operation display unit installed on the door 20 of the refrigerator compartment 11. A schedule setting unit 109 capable of inputting and displaying a schedule that is an action plan of the first user is built in, and is input to the control board 310 as a control device attached to the back of the refrigerator 3 by the schedule setting unit 109. This is different from the refrigerator 2 according to the second embodiment in that a schedule management unit 108 that manages the performed schedule is implemented. The operation panel 21a is, for example, a touch panel having an image display function and an information input function. The schedule setting unit 109 has, for example, a schedule input button displayed on the operation panel 21a. The schedule input button of the schedule setting unit 109 may be a display unit (touch panel) of the first user's mobile terminal that can communicate with the refrigerator 3. The schedule management unit 108 can display information useful for the schedule management of the first user on the operation panel 21a using the input schedule.

The first user of the refrigerator 3 schedules whether the resident of the residence where the refrigerator 3 is installed (that is, the first user of the refrigerator 3) is at home or away from the operation panel 21a (scheduled presence or absence). The first user's life pattern (for example, a pattern such as wake-up time, bedtime, work time, home time, meal time, etc.), the first user's shopping pattern (for example, a pattern of shopping cycles in one week, supplement to the refrigerator) Schedule information such as the type and amount of food to be made). The schedule information is displayed on the operation panel 21a as shown in FIGS. In the refrigerator 3 according to Embodiment 3, the schedule information of the first user of the refrigerator 3 is managed in the refrigerator 3. However, it is also possible to cause a server on the network (a cloud server 602 in FIG. 19 to be described later) to execute some of the functions in the control unit 300 of the refrigerator 3.

The schedule management unit 108 is connected to the schedule setting unit 109 and the life management unit 105 in the operation panel 21a, and receives schedule information of the first user of the refrigerator 3 input to the schedule setting unit 109 from the operation panel 21a. The schedule information is displayed on the schedule setting unit 109 in a predetermined format (shown in FIGS. 16 to 18 to be described later) and given to the life management unit 105.

16 to 18 are diagrams showing examples of a schedule display screen showing schedule information set by the schedule setting unit 109 of the refrigerator 3 according to the third embodiment. 16 to 18 show examples of screens in which the schedule information of the input first user is managed by the schedule management unit 108 and is displayed again on the operation panel 21a. 16 shows an example of a schedule display for one month, FIG. 17 shows an example of a schedule display for one week, and FIG. 18 shows an example of a schedule display for one day.

In the third embodiment, the first user of the refrigerator 3 is a family of four people, that is, the father “Taro” (TARO), the mother “Hanako” (HANAKO), the eldest daughter “Kazumi” (TOMOMI), and the eldest son “Kazuo” (KAZUO). ). In the schedule display for one month in FIG. 16, there are scheduled marks 91 (91a to 91e). The scheduled mark 91a “all” (ALL) indicates that the entire family has a schedule, the scheduled mark 91b “thick” (TA) indicates that “Taro” has a schedule, and the scheduled mark 91c “flower” "(HANA) indicates that there is a plan for" Hanako ", a scheduled mark 91d" Wa "(TOMO) indicates that there is a plan for" Kazumi ", and a scheduled mark 91e" One "(KAZU) indicates" "Ichio" shows that there is a plan.

In addition, a schedule content mark 92 (92a to 92g) is shown in the one week schedule display of FIG. As an example, the schedule content mark 92a indicates that the trip is scheduled, the schedule content mark 92b indicates that the trip is scheduled, the schedule content mark 92c indicates that the golf is scheduled, and the schedule content mark 92d is The schedule content mark 92e indicates that there is a plan for swimming (learning), the schedule content mark 92f indicates that there is a plan for piano (learning), and the schedule content mark 92g indicates soccer ( It shows that there is a schedule of learning.

In addition, in the daily schedule display of FIG. 18 (the horizontal axis is the time of the day), the current time 93, the outing period scheduled in the normal period (that is, the period in which a regularly repeated action pattern exists) (

Time periods

94a, 94b,

outing periods

95a, 95b and a sleeping period 96 scheduled in a period other than the normal period (that is, a period in which there is no regularly repeated action pattern) are shown. As an example, the outing period 95a scheduled in a period other than the normal period indicates that there is (or has been) a business trip, and the outing period 95b scheduled in a period other than the normal period has a plan to eat out. (Or was there). In addition, the outing period 94a scheduled in the normal period indicates that there is (or was) a work schedule, and the outing period 94b scheduled in the normal period indicates that the school is scheduled (or was). Show.

<3-2> Operation Next, an example of the operation of the refrigerator 3 according to the third embodiment will be described with reference to FIGS. 14 and 16 focusing on differences from the first and second embodiments.

As described in the description of the first and second embodiments, the life management unit 105 uses the refrigerator 3 based on the usage status of the refrigerator compartment 11 acquired from the storage status or the deterioration status of the food in the storage chamber. The second information indicating the life status of one user is detected (acquired). However, it is impossible to know the habit of the first user of the refrigerator 3 or a sudden change in schedule only from the state of the food in the refrigerator storage chamber, and the life management unit 105 indicates the living state of the first user. There is a possibility that the determination as to whether or not the second information is information to be notified to the second user is erroneous (that is, an appropriate determination cannot be made).

For example, even when foods that are used on a daily basis, such as milk and eggs, are left in the storage room in a deteriorated state, if the first user has a long-term trip plan, the refrigerator It is not necessary to notify safety-related information to a second user different from the first user who uses 3. In addition, when it is detected that the expiry date of a luxury item such as cake is approaching, if it is determined taking into consideration the type of food, it is a luxury item and there is no urgency, so the second user is notified. do not have to.

Therefore, in the third embodiment, as shown in FIG. 14, the schedule management unit 108 manages the schedule information of the first user input by the operation panel 21a (that is, the schedule setting unit 109 of the operation panel 21a), The schedule setting unit 109 displays a schedule on the operation panel 21 a and gives schedule information indicating the schedule to the life management unit 105. For this reason, the life management unit 105 in the third embodiment adds to the first information indicating the use state detected based on the deterioration state of the food in the use state acquisition unit 104 described in the first and second embodiments. It is possible to detect and manage the second information indicating the living status of the first user reflecting the schedule information of the first user.

As a specific method of reflecting the schedule information, with respect to the schedule of the first user shown in FIG. 16 to FIG. 18, first, as a control example in units of several days, the schedule display for one month in FIG. From the display of the weekly schedule, from December 19th to December 21st, all the first users (the whole family) are absent due to travel, and the food stored in the storage room is left unattended. It is certain to be done. For this reason, even if a situation in which the refrigerator compartment 11 is not used at all is detected in the usage status acquisition unit 104 from the deterioration state of the food, the life management unit 105 is absent during the trip. By reflecting the schedule information, unnecessary safety-related information can be prevented from being notified to a second user such as a family living in a distant place.

In addition, as an example of reflecting the daily schedule information, December 25 shown in the daily schedule display of FIG. 18 is not in the normal period in addition to the

outing time zones

94a and 94b scheduled in the normal period. There are

outing periods

95a and 95b scheduled in the period of No. 1, and all of the first users are not at home almost all day. Furthermore, it is assumed that there is no person who eats dinner at home due to the schedule of business trip 95a and restaurant 95b. In such a situation, when the usage status acquisition unit 104 detects a situation where a food item whose expiration date is approaching, such as sashimi or cake, is not used, the life management unit 105 is the first tonight. By reflecting the schedule information that all users (all family members) are eating out, the second user, who is a family living in the neighborhood, is notified that there is food in the refrigerator 3 that is about to expire. Can be used by the second user. In this way, wasteful disposal of food can be prevented.

<< 3-3 >> Effect As described above, according to the refrigerator 3 according to the third embodiment, the life management unit 105 uses the first user's schedule information to indicate the second user's life situation. Since it is determined whether or not the information is information to be notified to the second user, it is based on notification of the second information of information that does not need to be notified to the second user (that is, excessive notification of the second information). It is possible to prevent communication loss and food loss due to notification failure (that is, insufficient notification) of the second information that needs to be notified to the second user.

Moreover, when detecting the 1st user's living condition based on the deterioration state of a foodstuff, and a 1st user's schedule information, the continuous storage pattern of the foodstuff in a storage room, the pattern of a 1st user's shopping, etc. It can be assumed. For example, when the deterioration state of the food progresses as it approaches the weekend, and the weekday daytime schedule is almost filled with the

outing periods

94a and 94b scheduled in the normal period shown in FIG. It is assumed that the household has a shopping pattern in which food is bought at the weekend. At this time, in particular, when a change occurs in the deterioration state without inputting schedule information scheduled in a period other than the normal period, a possibility of failure of the refrigerator 3 is assumed in addition to the deterioration of the food itself. Is done. For this reason, the life management unit 105 sends a second user (for example, an electrical store with a long-standing transaction, or a service representative of the electrical store) to the second user different from the first user who uses the refrigerator 3. The second information indicating the living status of the first user can be notified.

In FIG. 15, in order to reflect the schedule information of the first user input by the schedule setting unit 109 of the operation panel 21 a and arranged by the schedule management unit 108 in the life management unit 105, The form of sending is shown. However, the transmission destination of the schedule information may be the deterioration detection unit 5 or the usage status acquisition unit 104 arranged in a stage preceding the life management unit 105. In any case, the schedule information can be reflected in the detection of the living condition of the first user, and particularly when the schedule information is sent to the deterioration detection unit 103, the deterioration state of the generated food is It is possible to determine in advance whether it is accidental or inevitable due to a change in the schedule of the first user. For this reason, the 2nd information which shows the 1st user's living condition can be appropriately notified to the 2nd user via communications department 23.

<< 4 >> Embodiment 4
In the fourth embodiment of the present invention, a network system including any one of the refrigerators according to the first to third embodiments will be described. FIG. 19 is a diagram illustrating a schematic configuration of a network system according to the fourth embodiment. In FIG. 19, the same components as those described in the first to third embodiments are denoted by the same reference numerals as those used in the first to third embodiments. As shown in FIG. 19, the network system according to the fourth embodiment includes a home system 500 and an outside system 600 that can communicate with the home system 500 via the Internet. Home system 500 includes refrigerator 4. The refrigerator 4 is any one of the

refrigerators

1, 2, and 3 of the first, second, and third embodiments.

As shown in FIG. 19, the in-home system 500 includes home appliances such as a refrigerator 4, a room air conditioner, a water heater, an IH (Induction Heating) cooking heater, and a microwave oven. These home appliances and power measuring device 503 are connected to the system power source 501 by the electric lamp line 502, and power is supplied from the system power source 501. The in-home system 500 includes a power measurement device 503 and a power measurement terminal 504 for measuring the power supplied from the system power source 501 to each home appliance through the power line 502, a home controller 505, a broadband router 506, and a communication device. Device communication unit 515.

The power measuring device 503 measures the power supplied to each home appliance (that is, the power consumption of each home appliance) and the total power supplied from the system power source 501 to the entire home appliance and the power measuring device 503. Measurement results (history) can be saved. The power measuring device 503 measures the supplied power using a power measuring terminal 504 such as a CT (Current Transformer).

Each of the home appliance and the power measuring device 503 is provided with a device communication unit 515 for communicating with the home controller 505 via a communication network. The communication network is, for example, a wired or wireless LAN (local area network). The home appliances such as the refrigerator 4 and the power measuring device 503 can bidirectionally communicate with the in-home controller 505 via the respective device communication units 515 in a wired or wireless manner. In addition, the apparatus communication part 515 connected to the refrigerator 4 plays the same role as the communication part 23 as a communication apparatus demonstrated by Embodiment 1-3. Therefore, when the communication unit 23 of the refrigerator 4 can directly communicate with the home controller 505 by wire or wireless, it is not necessary to include the device communication unit 515 for the refrigerator 4.

When the device communication unit 515 has a function of performing wired communication, the device communication unit 515 includes a wired communication module such as a serial interface or a driver, for example. When the device communication unit 515 has a wireless communication function, the device communication unit 515 includes a wireless communication module such as Wi-Fi (registered trademark) or Bluetooth (registered trademark). Note that the device communication unit 515 may be included in each of the home appliance and the power measurement device 503 or may be provided outside the home appliance and the power measurement device 503.

The home controller 505 can acquire information on the power consumption of the home appliance and the power supplied to the system power source 501 measured by the power measuring device 503 from the power measuring device 503 via the communication network. Further, the home controller 505 can acquire information on the operating state of the home appliance via the communication network. Then, the home controller 505 manages the acquired power consumption, supply power, and operating state information, and if necessary, controls change instructions (for example, operating state change instructions) to the home appliance via the communication network. ) Can be transmitted.

Also, the home controller 505 is connected to the Internet 601 of the outside system 600 via the broadband router 506.

The Internet 601 is connected to a cloud server 602 in which various power information and environmental information such as weather forecasts are stored. The Internet 601 is also connected to the second user terminal 70. The second user terminal 70 is a device used by a second user different from the first user of the refrigerator 4 as described in the first embodiment. The second user terminal 70 is, for example, a smart phone, a mobile tablet, a second user (a manager such as a family member, a relative, a friend, an acquaintance, a security company, a public institution, or a care company) that has been given usage rights. A device having a communication function such as a mobile phone or a personal computer.

The cloud server 602 is an external server that is communicably connected to the refrigerator 4 and the second user terminal 70 via the Internet 601. It is desirable that the cloud server 602 includes a personal authentication unit 603 that authenticates the second user terminal 70 with or without connection authority to the refrigerator 4.

For example, the cloud server 602 stores in advance a portable terminal ID of the second user terminal 70 having the authority to connect to the refrigerator 4 with respect to the ID (identifier) of the refrigerator 4 in a built-in or externally connected storage unit. ing. Here, the ID of the refrigerator 4 is information that can uniquely identify each of the plurality of refrigerators that can be connected to the Internet 601. The mobile terminal ID is information that can uniquely identify each of a plurality of mobile terminals connected to the Internet 601.

Then, the personal authentication unit 603 determines whether or not the portable terminal ID of the second user terminal 70 is stored as having authority to connect to the refrigerator 4 with respect to the refrigerator 4 (at the registered second user terminal). By checking whether or not there is an authority to connect to the refrigerator 4 with respect to the second user terminal 70.

The cloud server 602 as an external server connects (communications) the second user terminal 70 that has been authenticated by the personal authentication unit 603 to have the authority to connect to the refrigerator 4 to the refrigerator 4 via the Internet 601. to approve. That is, the cloud server 602 prohibits a mobile terminal that is not authenticated to have authority to connect to the refrigerator 4 by the personal authentication unit 603 from connecting to the refrigerator 4 via the Internet 601.

The home controller 505 obtains the power consumption of the home appliance or the power supplied from the system power source 501 from the power measuring device 503 in the home system 500, and obtains information on the operating state or the surrounding environment from the home appliance. Further, the home controller 505 is connected to the Internet 601 via the broadband router 506, and can obtain various power information or various environment information from the cloud server 602.

For example, the in-home controller 505 of the in-home system 500 can obtain the power supply / demand situation regarding the area where the in-home system 500 is installed as power information from the cloud server 602 of the outside system 600. When the local power consumption is approaching the upper limit value of the supplied power, the in-home controller 505 can transmit a power saving instruction to a home electric appliance with particularly high power consumption. In such a case, for the refrigerator 4, when it is determined that the refrigerator 4 is overcooled from the set temperature of the storage room and the actual temperature history, an increase in the set temperature can be instructed from the home controller 505.

Also, the second user terminal 70 can be connected to the refrigerator 4 directly or indirectly via the Internet 601 and via the home controller 505. At this time, only the second user terminal 70 that is authenticated by the personal authentication unit 603 of the cloud server 602 to be authorized to connect to the refrigerator 4 is permitted to connect to the refrigerator 4.

Therefore, for example, when a family member or a helper is an administrator, if the portable terminal ID of the second user terminal 70 is registered, the life status of the first user of the refrigerator 4 is shown while maintaining security. The second information can be shared. Further, not only the first user but also the second user can check the stock status of the food in the storage room of the refrigerator 4 at a shopping destination such as a supermarket. Or forgetting to buy necessary food.

In Embodiment 4, all or part of the reference information referred to by the food identification unit (configuration 107 in FIG. 11) may be stored in the cloud server 602 that is an external server. By storing the reference information of the food information storage unit (configuration 106 in FIG. 11) in the cloud server 602, huge data regarding the type of food is shared with users other than the first user and the second user, for example. Is possible.

Note that although the personal authentication unit 603 described above authenticates the connection authority using the ID, the authentication method is not limited to this. The personal authentication unit 603 replaces the authentication using the ID or in addition to the authentication using the ID, for example, biometric authentication such as a fingerprint, a vein, and an iris of the eye, or authentication using a password, or these authentication methods. The connection authority may be authenticated by a combination of.

As described above, according to the network system according to the fourth embodiment, in addition to the effects obtained by the refrigerators 1 to 3 according to the first to third embodiments, the second terminal device 70 used by the second user is Whether or not it is a portable terminal of a person who has authority to communicate with the refrigerator can be strictly authenticated. For this reason, it can prevent reliably that the usage condition of a refrigerator and the 1st user's living condition which are personal information of a 1st user leak to the person who does not have authority.

1, 2, 3, 4 Refrigerator, 11 cold room, 12 ice making room, 13 switching room, 14 freezer room, 15 vegetable room, 16 cold room shelf, 17 chilled room, 18 chilled case, 19 cooling airway, 20 door 21, 21a operation panel, 22 open / close detection unit, 23 communication unit, 24 storage unit, 25 time providing unit, 27 photographing device, 62 stock food list display screen, 70 second user terminal (separate user terminal, terminal device), 71 Display section, 91, 91a-91e Scheduled mark, 92, 92a-92g Scheduled content mark, 93 Current time, 94, 94a, 94b Outing time zone due to schedule other than normal period, 95, 95a, 95b Normal period schedule Going out time, 96 bedtime, 100, 200, 30 Control unit, 101 Image management unit, 102 Change detection unit, 103 Degradation detection unit, 103a Food status detection unit, 104 Usage status acquisition unit, 105 Life management unit, 106 Food information storage unit, 107 Food identification unit, 108 Schedule management unit 109 schedule setting unit, 110, 210, 310 control board, 500 in-home system, 501 system power supply, 502 power line, 503 power measurement device, 504 power measurement terminal, 505 in-home controller, 506 broadband router, 515 device communication unit, 600 Out-of-home system, 601 Internet, 602 cloud server, 603 personal authentication unit.

Claims

A refrigerator that includes a storage room for storing food and a communication unit that performs communication, and is used by a first user,
A photographing device for acquiring an image in the storage chamber;
An image management unit for extracting a food image from the image acquired by the imaging device;
A food status detection unit that detects a change in the status of the food stored in the storage room from the food image in a certain period;
From a change in the status of the food detected by the food status detection unit, a usage status acquisition unit that acquires first information indicating the usage status of the refrigerator by the first user;
When it is determined from the first information that the living status of the first user should be notified to a second user different from the first user, the second status indicating the living status via the communication unit A life management unit for notifying the terminal device used by the second user of information;
Refrigerator equipped with.
A storage unit for storing information;
A time providing unit that provides time information;
Further comprising
The refrigerator according to claim 1, wherein the image management unit acquires time information from the time providing unit, and stores the food image and the time information in the storage unit in association with each other.
The food status detection unit
A change detection unit for detecting a time change of the food image;
A deterioration detection unit that detects a deterioration state of the food from the time change of the food image detected by the change detection unit;
A refrigerator according to claim 1.
The refrigerator according to claim 3, wherein the change detection unit detects a temporal change in color and size of the food image.
A food identification unit for identifying the type and amount of food corresponding to the food image;
The refrigerator according to any one of claims 1 to 4, wherein the usage status acquisition unit detects the usage status of the first user based on the type and amount of the food identified by the food identification unit.
A food information storage unit that stores in advance, in association with at least one of the shape, size, color, and exterior label of the food, the type of food,
The refrigerator according to claim 5, wherein the food identification unit identifies the type and amount of food corresponding to the food image with reference to information stored in the food information storage unit.
A schedule management unit for managing the schedule of the first user;
The life management unit determines that the life status of the first user is the second user from the first information acquired by the usage status acquisition unit and the schedule information of the first user received by the schedule management unit. It is judged whether it is the situation which should be notified to. The refrigerator of any one of Claims 1-6.
When the life management unit identifies that the storage state of the food or the usage status of the first user has not changed for a predetermined period or longer, the life management unit passes the communication unit through the communication unit. The refrigerator according to any one of claims 1 to 7, wherein safety related information is reported to the terminal device.
The life management unit identifies that a situation that is assumed to have changed in the storage state of the food or the usage status of the first user has occurred at least once despite the absence of the first user. The refrigerator according to claim 7, wherein, when it is done, the terminal device is notified of crime prevention related information via the communication unit.
The life management department
When it is identified that the storage status of the food or the usage status of the first user has a predetermined characteristic as the living status of the first user,
The refrigerator according to any one of claims 1 to 7, wherein food-related information is notified to the terminal device via the communication unit.
The refrigerator according to any one of claims 1 to 10,
The terminal device that is communicably connected to the refrigerator and used by the second user;
Network system equipped with.
A server connected to the refrigerator and the terminal device via the Internet so as to be communicable;
The network system according to claim 11, wherein the server includes a personal authentication unit that authenticates that the terminal device is a terminal permitted to communicate with the refrigerator.
The server
Store in advance the food type in association with at least one of the shape, size, color, and exterior label of the food,
The network system according to claim 12, wherein the first information indicating the usage status of the first user is acquired based on a type and amount of food in the storage chamber.
The server
Managing the schedule of the first user;
Whether the first user's life situation should be notified to the second user from the first information indicating the use situation of the refrigerator by the first user and the schedule information of the first user. The network system according to claim 12 or 13.
A method for informing a living situation of a first user using a refrigerator,
Obtaining an image of the storage room by photographing the storage room of the refrigerator;
Extracting a food image from the acquired image;
Detecting a change in the status of food stored in the storage room from the food image in a certain period;
Obtaining a first information indicating a use status of the refrigerator by the first user from a change in the status of the detected food;
When it is determined from the first information that the life status of the first user should be notified to a second user different from the first user, the second information indicating the life status is given to the second user. Informing the terminal device used by
A method for notifying the living situation.
A program for notifying a living situation of a first user who uses a refrigerator including a storage room for storing food, a communication unit for performing communication, and a photographing device for acquiring an image in the storage room,
Processing to acquire an image in the storage room by photographing the storage room of the refrigerator by the imaging device;
A process of extracting a food image from the acquired image;
From the food image in a certain period, processing to detect a change in the status of food stored in the storage room;
From the detected change in the status of the food, a process of obtaining first information indicating the usage status of the refrigerator by the first user;
When it is determined from the first information that the life status of the first user should be notified to a second user different from the first user, the second information indicating the life status is given to the second user. Processing to notify the terminal device used by
Program for living conditions that causes a computer to execute.