JP2017089913A - Refrigerator and network system including the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a refrigerator and a network, for enabling a user to visually confirm the inside of the refrigerator and acquire information about deterioration of food products by using the existing portable terminal.SOLUTION: A refrigerator 1000 includes a storage chamber 100, a photographing device 2 for acquiring an image in the storage chamber 100 by photographing the inside of the storage chamber 100, a control device 1001 for extracting a food product image of food products stored in the storage chamber 100 from the image in the storage chamber 100 acquired by the photographing device 2, and a radio communication part 7 being a communication device for communicating with a portable terminal 8 having a display part 8a. The control device 1001 generates information on deterioration of the food products corresponding to a food product image from a temporal change in the food product image, and transmits the food product image and the information on the deterioration to the portable terminal 8 through the radio communication part 7. A network system includes the refrigerator 1000 and the portable terminal 8, and the portable terminal 8 displays at least one between the food product image and the information on the deterioration of the food products.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a refrigerator having a function of communicating with a mobile terminal, and a network system including the refrigerator and the mobile terminal.

  The diversification of eating habits and the increase in capacity of refrigerators make it difficult for users to accurately grasp the storage state (inventory state) and deterioration state of food items in the refrigerator. In addition, there is a case where food is redundantly purchased without noticing the food stored in the refrigerator. For these reasons, there are an increasing number of cases where foods whose expiration date has passed are discarded.

  Generally, in order to grasp the stock state and the deterioration state of the food stored in the refrigerator, the user opens the door of the refrigerator and visually checks the food in the refrigerator and the state and display of the food. However, if the frequency of opening and closing the door increases, the temperature in the refrigerator rises, so that electric power for cooling is consumed and this causes a quick deterioration of the food. Therefore, it is desirable to be able to check the stock status and the deterioration status of food without opening the refrigerator door.

  Patent Document 1 discloses a technique for photographing a storage room (storage room) of a refrigerator with a photographing device, comparing and analyzing a photographed image and a reference image, and calculating and displaying a free space (empty space) in the storage room. Disclosure. With this technology, the user can grasp the empty space without opening the refrigerator door.

  Moreover, patent document 2 attaches | subjects a radio | wireless tag to foodstuff, and based on the information sent from a radio | wireless tag, and the temperature information sent from the temperature sensor in a refrigerator, from preservation | save period-storage temperature-quality tolerance (TTT) theory Discloses a technique for estimating quality degradation characteristics of foods.

JP 2014-209048 A (for example, paragraphs 0041-0049 and FIGS. 7-8) JP 2006-317109 A (for example, paragraphs 0027-0042, FIG. 5)

  However, in the technique described in Patent Document 1, since the deterioration state of the food stored in the refrigerator cannot be known, there is a problem that the disposal of the food whose expiration date has expired cannot be sufficiently suppressed.

  In addition, the technique described in Patent Document 2 requires a dedicated device such as a reader and a writer for a wireless tag attached to food, and there is a problem that the cost of the entire system increases.

  The present invention has been made in order to solve the above-described problems of the prior art, and by using an existing portable terminal, the inside of the refrigerator can be visually confirmed, and information on food deterioration can be obtained. An object is to provide a refrigerator and a network system.

  The refrigerator according to the present invention includes a storage room for storing food, a photographing device for obtaining an image in the storage room by photographing the storage room, and an image in the storage room obtained by the photographing device, A control device for extracting a food image of the food stored in the storage room, and a communication device for communicating with a portable terminal having a display unit; Information regarding deterioration of food corresponding to the food image is generated, and the food image and information regarding deterioration are transmitted to the portable terminal via the communication device.

  The network system according to the present invention includes the refrigerator and a mobile terminal that includes the display unit and communicates with the refrigerator via wired or wireless communication via the communication device, and the mobile terminal includes the display unit. Further, at least one of the food image and the information on the deterioration is displayed.

  According to the present invention, the user can visually check the inside of the refrigerator using a mobile terminal without providing a dedicated device, and can obtain information on the deterioration of food using the mobile terminal. effective.

It is a front view which shows schematic structure of the refrigerator which concerns on Embodiment 1 of this invention. It is a figure which shows schematic structure (including the cross-sectional structure which cuts FIG. 1 by the line segment II-II, and the structure of a control apparatus), and a portable terminal of the refrigerator which concerns on Embodiment 1. FIG. 3 is a flowchart schematically showing the operation of the refrigerator according to the first embodiment. (A)-(d) is a figure which shows the example of the supplementary information of the picked-up image (storage room image) in the refrigerator compartment of the refrigerator which concerns on Embodiment 1, the food image extracted from the picked-up image, and a food image. . (A) And (b) is a figure which shows the example of the display screen in a portable terminal of the food image extracted from the picked-up image (storage room image) in the refrigerator compartment of the refrigerator which concerns on Embodiment 1. FIG. It is a figure which shows the example of the display screen in a portable terminal of the food image extracted from the picked-up image (storage room image) in the refrigerator compartment of the refrigerator which concerns on Embodiment 1. FIG. (A) And (b) is a schematic diagram which shows the time change detection method of the food image in the time change detection part of the refrigerator which concerns on Embodiment 1. FIG. (A) And (b) is a figure which shows the example of the display screen in a portable terminal about the image of the inventory food list of the refrigerator which concerns on Embodiment 1, and the image of a food change display. It is a figure which shows schematic structure (including the cross-sectional structure and the structure of a control apparatus which cut | disconnect FIG. 1 in line segment II-II), and a portable terminal of the refrigerator compartment which concerns on Embodiment 2 of this invention. It is a figure which shows the example of the content memorize | stored in the food image database which the food image identification part of the refrigerator which concerns on Embodiment 2 refers to in table format. 6 is a flowchart schematically showing the operation of the refrigerator according to the second embodiment. It is a figure which shows schematic structure (including the cross-sectional structure which cuts FIG. 1 by line segment XII-XII, and the structure of a control apparatus) and the portable terminal which concern on Embodiment 3 of this invention. (A)-(c) is a figure which shows the example of the display screen of the food image in the portable terminal which communicates with the refrigerator which concerns on Embodiment 3. FIG. (A) to (c) is a schematic diagram illustrating an example of an estimation method in a time limit estimation unit of a refrigerator according to Embodiment 3. FIG. It is a figure which shows schematically the structure of the network system which concerns on Embodiment 4 of this invention.

<< 1 >> Embodiment 1
<< 1-1 >> Configuration FIG. 1 is a front view showing a schematic configuration of a refrigerator 1000 according to Embodiment 1 of the present invention. As shown in FIG. 1, the refrigerator 1000 according to Embodiment 1 includes a plurality of storage rooms. The plurality of storage rooms include, for example, a refrigerator room 100, an ice making room 200, a switching room 300, a freezer room 400, and a vegetable room 500. In the example of FIG. 1, the plurality of storage rooms are arranged in the order of the refrigerator room 100, the ice making room 200, the switching room 300, the freezer room 400, and the vegetable room 500 from the top. The ice making chamber 200 and the switching chamber 300 are arranged side by side in the horizontal direction. The refrigerator 1000 has a refrigeration cycle circuit, and the plurality of storage rooms are cooled by the refrigeration cycle circuit. The refrigeration cycle circuit will be described in a refrigerator 1000b according to a third embodiment described later. FIG. 1 is also referred to in a second embodiment (FIG. 9) and a third embodiment (FIG. 12) described later. The refrigerator 1000 is described as a refrigerator 1000a in the second embodiment, and is expressed as a refrigerator 1000b in the third embodiment.

  An operation panel 1 having a display unit (display screen) may be provided on the front surface (front surface) of the door of the refrigerator compartment 100. On the display screen of the operation panel 1, information related to the refrigerator (in-house information) such as the set temperature of each storage room and the current temperature of each storage room can be displayed. The operation panel 1 has, for example, a touch panel that accepts user input. The installation position of the operation panel 1 is not limited to the door of the refrigerator compartment 100, and may be installed at other positions such as the outer surface of the door of another storage room or the side surface of the refrigerator 1000. In addition, the ice making room 200, the switching room 300, the freezer room 400, and the vegetable room 500 are configured to be slid out to the front side of the refrigerator 1000 together with the doors provided on the front surfaces thereof, for example. Yes. FIG. 1 shows an example of a refrigerator to which the present invention is applicable, but the refrigerator to which the present invention is applicable is not limited to that shown in FIG.

  2 is a diagram showing a schematic configuration (including a cross-sectional structure taken along line II-II in FIG. 1 and a configuration of the control device 1001) and the portable terminal 8 of the refrigerator compartment 100 of the refrigerator 1000 according to the first embodiment. It is. The refrigerator 1000 according to Embodiment 1 has, as main components, a refrigeration room 100 that is a storage room for storing food, and a photographed image (storage room image) in the refrigeration room 100 by photographing the inside of the refrigeration room 100. Extracting food images (food image data) of food stored in the refrigerator room 100 from the imaging device 2 such as a camera to be acquired and the storage room image (storage room image data) acquired by the imaging device 2 And a control device 1001 for managing (including storage). The control device 1001 includes a wireless communication unit 7 that performs communication with the mobile terminal 8 including the display unit 8a. However, the wireless communication unit 7 may be a part of the control device 1001, but may be configured separately from the control device 1001. The control device 1001 generates, for example, information on the deterioration of the food corresponding to the food image from the temporal change of the food image (change of the food image caused by the change of the food over time), and the food image and the food The information regarding the deterioration is transmitted to the portable terminal 8 via the wireless communication unit 7.

  Moreover, the refrigerator 1000 shown by FIG. 2 can be provided with the following structures. As shown in FIG. 2, the refrigerator compartment 100 is closed on the front side (left side in FIG. 2) by the refrigerator compartment door 101. The refrigerator compartment 100 can include a plurality of shelf boards 102 therein. The refrigerator compartment 100 is partitioned into a plurality of spaces (shelves) by the shelf plate 102. The space below the lowermost shelf 102 is, for example, a chilled chamber 110. Inside the chilled chamber 110, a chilled case 111 that can be slid forward (leftward in FIG. 2) and pulled out along a guide member (not shown) such as a rail is installed. Further, a cooling air passage 1010 is provided on the back side of the refrigerator compartment 100 (on the right side in FIG. 2). Details of the cooling air passage 1010 will be described in the third embodiment.

  The photographing device 2 is installed on the inner surface of the refrigerator compartment door 101. In the example of FIG. 2, the imaging device 2 can capture the inside of the refrigerator compartment 100 from the refrigerator compartment door 101 side, which is the same side as the user's viewpoint, with the visual field range from the ceiling surface of the refrigerator compartment 100 to the chilled chamber 110. It has a possible configuration. In addition, the installation position of the imaging device 2 is not limited to the inner surface of the refrigerator compartment door 101. The imaging device 2 may be installed at any position as long as it can capture a wide range of storage items such as food placed on the shelf board 102. Further, in consideration of the fact that the shelf plate 102 blocks the field of view, a plurality of imaging devices 2 may be installed in order to allow more items to be captured.

  The control device (control unit) 1001 can be configured by a control board having a control circuit (processing circuit). The control apparatus 1001 is attached to the back surface of the refrigerator 1000, for example. A control device 1001 shown in FIG. 2 includes a food image extraction unit 3, a food image management unit 4, a time change detection unit 5, a food deterioration state estimation unit 6, and a wireless communication unit 7 as a communication device. The communication device may be either a wireless communication unit or a wired communication unit. Moreover, all or part of the food image extraction unit 3, the food image management unit 4, the time change detection unit 5, and the food deterioration state estimation unit 6 is used as a storage device (storage unit) that stores a program as software. And a processor as an information processing apparatus that executes a program stored in the memory (for example, by a computer). The food image extraction unit 3, the food image management unit 4, the time change detection unit 5, and the food deterioration state estimation unit 6 include a memory that stores a program as software and a processor that executes the program stored in the memory, In addition, it may be realized by a combination of processing circuits.

  The wireless communication unit 7 includes a communication module such as Wi-Fi or Bluetooth (registered trademark), for example. The refrigerator 1000 uses the wireless communication unit 7, for example, via at least one of a router (broadband router 2006 shown in FIG. 15 described later) and the Internet (Internet 3001 shown in FIG. 15 described later). It is possible to communicate with a mobile terminal (user terminal) 8 such as a smart phone or a mobile tablet existing inside or outside the home. Therefore, the refrigerator 1000 can display various information related to the refrigerator 1000 on the display unit 8 a of the mobile terminal 8.

  The food image extraction unit 3 is communicably connected to the imaging device 2 and the food image management unit 4. The food image extraction unit 3 is, for example, a storage room image (for example, the inside of the refrigeration room 100) that is an image in the refrigeration room 100 photographed by the photographing device 2 when the refrigeration room door 101 is opened and closed (for example, immediately after opening and closing). And a food image which is an image portion of the food is extracted from the received storage room image. Furthermore, the food image extraction unit 3 adds time information (for example, shooting time information) to the food image extracted from the storage room image as supplementary information (that is, associates the food image with the corresponding supplementary information). ), And transmits these food images and supplementary information to the food image management unit 4. Here, the imaging device 2 and the food image extraction unit 3 constitute a food image detection unit that images the inside of the refrigerator compartment 100 and detects a food image. The time information can include not only the hour, minute, and second but also the date (month / day or year / month / day). The time information is acquired from, for example, a clock (not shown) as a time information acquisition unit provided in the control device 1001. However, the time information may be acquired from a time signal transmitted by a time signal provider. The food image extraction unit 3 may be configured to transmit the food image to the food image management unit 4 without performing the process of adding time information to the food image as supplementary information. In this case, the acquisition of time information is performed by a part other than the food image extraction unit 3 in the control device 1001.

  The food image management unit 4 is communicably connected to the food image extraction unit 3, the time change detection unit 5, and the wireless communication unit 7. The food image management unit 4 can receive the food image and the time information from the food image extraction unit 3, and can store information obtained by rearranging the food images in time series based on the time information in a storage unit (memory). . This storage unit may be either a storage unit in the food image management unit 4 or a storage unit outside the food image management unit 4. At this time, the food image management unit 4 can identify the receipt or delivery of the food based on the food image, and add the identification result (entry / exit identification information) to the supplementary information of the food image and store it in the storage unit. it can. Further, the food image management unit 4 can transmit the food image and accompanying information (time information and entry / exit identification information) to the time change detection unit 5 and the wireless communication unit 7.

  The time change detection unit 5 is communicably connected to the food image management unit 4, the food deterioration state estimation unit 6, and the wireless communication unit 7. The time change detection unit 5 can receive the food image and supplementary information from the food image management unit 4, detect temporal changes in the color and size of the food image, and store them in the storage unit (memory). This storage unit may be either a storage unit in the time change detection unit 5 or a storage unit outside the time change detection unit 5. Further, the time change detection unit 5 transmits the detected temporal change in the color and size of the food image to the food deterioration state estimation unit 6 and the wireless communication unit 7.

  The food deterioration state estimation unit 6 is communicably connected to the time change detection unit 5 and the wireless communication unit 7. The food deterioration state estimation unit 6 receives the temporal change of the food image from the time change detection unit 5, estimates the food deterioration state based on the temporal change, and transmits the estimation result to the wireless communication unit 7. To do. In other words, the food deterioration state estimation unit 6 generates information on the deterioration of food corresponding to the food image (for example, information indicating the deterioration state of food) from the temporal change of the food image, and the wireless communication unit 7 Via the mobile terminal 8.

  The wireless communication unit 7 is communicably connected to the food image management unit 4, the time change detection unit 5, and the food deterioration state estimation unit 6 in the control device 1001, and performs wireless communication with the mobile terminal 8. The wireless communication unit 7 changes the food image and accompanying information from the food image management unit 4 and the color and size of the food image detected from the time change detection unit 5 over time (the color and size of the food over time). Change in the color and size of the food image caused by the change in the image), and information indicating the deterioration state of the food estimated by the food deterioration state estimating unit 6, and the received information is carried by wireless communication. Transmit to terminal 8.

  The portable terminal 8 is connected to the wireless communication unit 7 wirelessly. The portable terminal 8 receives the food image and supplementary information, the detection result of the temporal change in the color and size of the food image, and the estimation result of the deterioration state of the food transmitted from the wireless communication unit 7 of the refrigerator 1000. The received information is displayed on the display unit 8 a of the mobile terminal 8.

<< 1-2 >> Operation Next, an example of the operation of the refrigerator 1000 will be described with reference to FIGS. 1, 2, and 3. FIG. 3 is a flowchart showing an operation performed every time the refrigerator compartment door 101 of the refrigerator 1000 is opened and closed (for example, immediately after being opened and closed).

  The operation shown in FIG. 3 is started by opening / closing the refrigerator compartment door 101. This is because when the refrigerator compartment door 101 is opened and closed, food in the refrigerator compartment 100 may be taken in and out. When the control device 1001 detects opening / closing of the refrigerator compartment door 101 based on a signal from a switch that detects opening / closing of the refrigerator compartment door 101 (step S11), the control device 1001 causes the photographing device 2 to photograph the inside of the refrigerator compartment 100 (step S12). . At this time, the switch that detects opening and closing of the refrigerator compartment door 101 is, for example, a magnet that detects the proximity of a magnet (magnet) embedded in the refrigerator compartment door 101 by a pair of reed switches installed on the refrigerator body side. Type sensor switch. In addition, after the refrigerator compartment door 101 is closed, the control device 1001 performs photographing by the photographing device 2 while the inside of the refrigerator compartment 100 is illuminated by the interior lighting device.

  The imaging device 2 transmits the image (storage room image) in the refrigerator compartment 100 obtained by imaging to the food image extraction unit 3 (step S13).

  The food image extraction unit 3 receives the storage room image from the imaging device 2 every time the refrigerator compartment door 101 is opened and closed (that is, every time the storage state in the refrigerator compartment 100 may change). In the following description, the opening / closing time of the current (latest) refrigerator compartment door 101 will be referred to as “current door opening / closing time”, and the previous time (one time before) opening / closing of the cold storage door 101 will be referred to as “previous door opening / closing time”. Called.

  The food image extraction unit 3 stores the storage room image received from the imaging device 2 when the door was opened / closed last time, and newly stores the storage room image received from the imaging device 2 when the door was opened / closed this time. And the food image extraction part 3 extracts the image of the change part of a store room image by calculating the difference of the store room image at the time of the last door opening and closing, and the store room image at the time of this door opening and closing (step) S14). The image of the changed portion of the storage room image is an image of food that has been received or delivered. For example, in the storage room image when the door was opened / closed last time, beer was placed on the shelf, and in this storage room image when the door was opened / closed, only the shelf where no beer was placed at the same location The food image of beer is extracted based on the difference between the two storage room images.

  The food image extraction unit 3 adds time information, which is the current time at the time of extraction, to the food image extracted in this way as supplementary information, and the food image management unit 4 adds these food images and supplementary information. (Step S15).

  The food image management unit 4 receives the food image and time information that is supplementary information from the food image extraction unit 3, and rearranges and stores the food images in time series based on the time information (step S16). At this time, the food image management unit 4 determines whether the food is in or out. Specifically, when the food image management unit 4 extracts a food image by the food image extraction unit 3, the food image management unit 4 analyzes a portion changed due to the difference, for example, an edge change amount obtained by edge processing, and Whether the food is in or out is determined by comparison with the storage room image stored in advance in an empty state (a state in which no food is stored in the refrigerator compartment 100). And the food image management part 4 preserve | saves by adding the judgment result (entrance / exit identification information) of the receipt or delivery of food to the incidental information of a food image.

  Further, the food image management unit 4 transmits the food image and accompanying information (time information and entry / exit identification information) to the time change detection unit 5 and the wireless communication unit 7.

  In the next step S17, the time change detection unit 5 receives the food image and the supplementary information from the food image management unit 4, detects the temporal change in the color and size of the food image, and stores it. Details of this processing will be described later.

  FIGS. 4A to 4D are diagrams showing examples of captured images in the refrigerator compartment 100 of the refrigerator 1000, food images extracted from the captured images, and supplementary information of the food images. 4 (a) to 4 (d), taken images (storage room images) 31a, 31b, 31c, 31d that are taken by the photographing device 2 and transmitted to the food image extraction unit 3 when the refrigerator door 101 is opened and closed. And food image 32b, 32c, 32d1, 32d2 extracted from the storage room image and supplementary information of the food image (time information 33b, 33c, 33d1, 33d2 and entry / exit identification information 34b, 34c, 34d1, 34d2) It is shown.

  FIG. 4A shows a storage room image 31 a that is captured by the imaging device 2 and transmitted to the food image extraction unit 3 at time A. At the time A, the refrigerator compartment door 101 is opened and closed, but no food is stored or delivered. Since there is no change in the storage room image 31a between time A and before time A, the food image extraction unit 3 does not generate a difference image (food image). Therefore, at time A, the food image extraction unit 3 does not transmit the food image to the food image management unit 4.

  FIG. 4B illustrates a storage room image 31b that is captured by the imaging device 2 and transmitted to the food image extraction unit 3 at time B after time A, and a food image 32b that is extracted from the storage room image 31b. The accompanying information of the food image 32b is shown. At time B, beer is left from the uppermost shelf in the refrigerator compartment 100. The food image extraction unit 3 extracts the food image (beer) 32b based on the difference between the storage room image 31a at time A and the storage room image 31b at time B, and converts the food image (beer) 32b to time information (time B) 33b. At the same time, it is transmitted to the food image management unit 4. The food image management unit 4 stores the warehousing / departure identification information 34b and the time information (time B) 33b that are associated with the food image 32b based on the determination result of the warehousing or warehousing. For convenience, the food image (difference image) shown in FIG. 4B also shows a shelf, but the actual food image does not have a refrigerator shelf and is only beer.

  FIG. 4C illustrates a storage room image 31c that is captured by the imaging device 2 and transmitted to the food image extraction unit 3 at time C after time B, and a food image 32c that is extracted from the storage room image 31c. The accompanying information of the food image 32c is shown. At time C, the salad is delivered from the bottom shelf in the refrigerator compartment 100. The food image extraction unit 3 extracts a food image (salad) 32c based on the difference between the storage room image 31b at time B and the storage room image 31c at time C, and converts the food image (salad) 32c to time information (time C) 33c. At the same time, it is transmitted to the food image management unit 4. The food image management unit 4 stores the entry / exit identification information 34c “time” and time information (time C) 33c in association with the food image 32c based on the determination result of entry or exit.

  FIG. 4D illustrates a storage room image 31d that is captured by the imaging device 2 and transmitted to the food image extraction unit 3 at time D after time C, and a food image 32d1, extracted from the storage room image 31d. 32d2 and accompanying information of the food images 32d1 and 32d2 are shown. At time D, milk is stored on the uppermost shelf of the refrigerator compartment 100, and cake is stored on the lowermost shelf. The food image extraction unit 3 extracts food images (milk and cake) 32d1 and 32d2 based on the difference between the storage room image 31c at time C and the storage room image 31d at time D, and food images (milk and cake) 32d1 and 32d2. Is transmitted to the food image management unit 4 together with the time information (time D) 33d1 and 33d2. The food image management unit 4 stores the entry / exit identification information 34d1, 34d2 and the time information (time D) 33d1, 33d2 “entry” in association with the food image based on the determination result of the entry or exit.

  As described above, the food image management unit 4 stores the food images 32b, 32c, 32d1, and 32d2 in association with the time information 33b, 33c, 33d1, and 33d2 and the entry / exit identification information 34b, 34c, 34d1, and 34d2, respectively. Therefore, the food warehousing time can be grasped, and the deterioration state can be estimated based on the warehousing time. That is, the food image management unit 4 grasps food (stock food) stored in the refrigerator compartment 100 as an image and holds information necessary for estimating the deterioration state of the food.

  Here, the image (image data) stored in the food image management unit 4 is only the food image (changed portion of the storage room image) extracted by the food image extraction unit 3. Moreover, the storage room image (for example, the whole image of the refrigerator compartment 100) memorize | stored in the food image extraction part 3 is also only the newest storage room image used as the reference | standard of a difference. That is, after extracting the food image, the food image extraction unit 3 can store only the latest storage room image and can erase the rest.

  As described above, since the food image extraction unit 3 and the food image management unit 4 reduce the amount of image data stored in each storage unit (memory), the storage capacity of the memory can be reduced, and Communication load can be reduced. The storage unit of the food image extraction unit 3 and the storage unit of the food image management unit 4 may be separate memories, but may be different storage areas of the same memory. In addition, the photographing by the photographing device 2, the extraction of the food image by the food image extraction unit 3, and the storage of the image data in the food image management unit 4 are performed when the refrigerator compartment door 101 is opened and closed (the possibility that the food has been put in and out). Only the data necessary for managing the food stored in the refrigerator compartment 100 is stored. Therefore, it is possible to grasp the storage state and the deterioration state of the food in the refrigerator 100 with the minimum necessary operations (photographing, image processing, and storage). However, when the refrigerator door 101 is not opened or closed (for example, every fixed time or every predetermined time), the control device 1001 causes the storage room image to be captured and the food deterioration state. It is also possible to grasp.

  Here, the storage state of food means which food is stored in the refrigerator compartment 100 at which point. In addition, the deterioration state of the food can include information indicating how much the deterioration of the food has progressed (for example, whether or not it is necessary to consume at an early stage).

  The food image management unit 4 divides the data of the food image stored in association with the accompanying information (time information and entry / exit identification information) in time series, entry or exit, and via the wireless communication unit 7 It can output to the display part 8a of the portable terminal 8. The user can grasp the food stored in the refrigerator compartment 100 by looking at the display screen of the display unit 8 a of the mobile terminal 8.

  FIGS. 5A and 5B are diagrams showing examples of display screens on the display unit 8a of the portable terminal 8 of the food image extracted from the captured image in the refrigerator compartment 100 of the refrigerator 1000 according to the first embodiment. It is. On the timeline display screen 21 shown in FIG. 5A, a food image 42 extracted from the storage room image every time the refrigerator compartment door 101 is opened and closed, time information 43 that is incidental information of the food image 42, and The entry / exit identification information 44 is displayed in chronological order from the top. In this example, food images 42 are arranged in order from the newest time indicated by the time information (photographing time information) in the horizontal center of the display screen, from the top to the bottom. The time information 43 is displayed on the right side and the entry / exit identification information 44 is displayed. In addition, on the upper side of the display screen, a target storage room (in this example, the refrigerating room 100), its set temperature 3 ° C., and time D are also displayed.

  On the timeline display screen 21 shown in FIG. 5A, for example, the target food can be searched retroactively to estimate the expiration date or the expiration date. Therefore, the food can be consumed before the expiration date or the expiration date, and the waste of food can be prevented.

  In the entry / exit list display screen 22 shown in FIG. 5 (b), the entry and exit are divided into left and right columns, and food images 52 and time information 53 are displayed in chronological order from the top (list display). )doing. That is, in the left column, the food image 52 and the time information 53 (that is, the warehousing time) in which the warehousing / department identification information 54 is “warehousing” are displayed from the top in order from the newest warehousing time. . In the right column, the food images 52 whose entry / exit identification information 54 is “exit” and the time information 53 (that is, the exit time) are displayed in order from the newest exit time.

  In the warehousing / listening display screen 22 shown in FIG. 5B, the current remaining amount of the food can be estimated from the history of warehousing and evacuation of the target food. By checking the display unit 8a of the mobile terminal 8 at the shopping destination, it is possible to prevent over-buying or forgetting to purchase useless food (for example, food).

  FIG. 6 is a diagram showing another example of the display screen of the food image 62 extracted from the photographed image in the portable terminal 8 of the refrigerator 1000 according to the first embodiment. The food image management unit 4 uses a food image stored in the refrigerator 1000 (here, a food image extracted every time the refrigerator compartment door 101 is opened / closed), time information and entry / exit identification information that are supplementary information. The food image 62 stored in the refrigerator compartment 100 at a certain point in time (the entry / exit identification information is not “exit”) is specified. Then, as shown in FIG. 6, a list of stock food images is displayed as a stock food list display screen 23 on the display unit 8 a of the mobile terminal 8 via the wireless communication unit 7. In the display example shown in FIG. 6, unlike FIGS. 5A and 5B, based on the time information, the food images 62 are displayed in order from the upper left, and the food images 62 from the upper left (for example, one line). Left, center, right of eyes, left, center, right of second row, left, center, right of third row, left, center, right of fourth row).

  In the stock food list display screen 23 shown in FIG. 6, since a list of stock food images at a certain point in time is displayed, it is stored in the refrigerator compartment 100 more clearly than the list display shown in FIG. Since it is possible to grasp the food that is being used, it is possible to prevent excessive purchase of useless food (for example, food) or forgetting to purchase necessary food. In addition, since food images are arranged in order from the oldest in the warehouse, the food images are arranged in order from the top, making it easier to find old foods, that is, foods that need to be consumed at an early stage, and suppressing the disposal of food. It becomes possible. However, the arrangement order of the display of the food images may be a method other than the method of arranging the food items in order from the oldest one, for example, the order of the new arrival times. This is because foods with an old warehousing time are often foods that can be stored for a long period of time, and new foods with a warehousing time that contain many foods that cannot be stored for a long time (for example, fresh ingredients such as meat or fish) ) Is a useful display method.

  Note that the display unit 8a of the portable terminal 8 includes three items, “entry / entry timeline” in FIG. 5A, “entry / exit list” in FIG. 5B, and “stock food list” in FIG. There are different types of food image display screens. Any one of these may be always displayed on the display unit 8a of the mobile terminal 8, or any one of them may be displayed in accordance with a user operation (for example, touching a touch panel). . Further, the food image display screen may be switched from a button linked to each display screen, or may be switched by a horizontal swipe operation or the like. Alternatively, these types of displays may be sequentially switched and displayed every predetermined time on the display unit 8a of the mobile terminal 8, or may be determined in advance according to time or day of the week. Select and display the display screen (for example, on Saturdays and Sundays where a large amount of food is often purchased, the list display of FIG. 6 is displayed, and from Monday to Friday when a small amount of food is often purchased. 5 (a) or (b) may be displayed).

  In addition, on the upper side of the screen of the display unit 8a of the mobile terminal 8, the target storage room (in this example, the refrigerator room 100), its set temperature, and the current time are displayed. In addition to the set temperature, You may display the measured temperature of each store room.

  Next, processing of the time change detection unit 5 and the food deterioration state estimation unit 6 will be described. In step S <b> 17 shown in FIG. 3, the color change of the food image received from the food image management unit 4 by the time change detection unit 5 (for example, a food image extracted from the storage room image every time the refrigerator door 101 is opened and closed) and Detect and store changes in size over time.

  Although there are differences depending on the type of food and the storage environment, the food stored in the refrigerator compartment 100 gradually deteriorates, and in particular, changes in color or size (or shape) appear. For example, fresh foods such as meat and fish change color to brown due to oxidation (methation). As for vegetables, broccoli gradually fades from bright green and increases in yellow, and when it further deteriorates, it changes to grayish brown due to spoilage or mold. Leafy vegetables such as spinach are deflated or shrunken due to transpiration, and their size changes.

  Therefore, every time a food image is received from the food image management unit 4, the time change detection unit 5 detects and stores temporal changes in the color and size of the received food image. Based on temporal changes in color and size stored in the time change detection unit 5, a food deterioration state estimation unit 6 described later estimates the deterioration state.

As an index for detecting a color change, for example, when the L ^* a ^* b ^* color system, which is a general color index, is used, the oxidation of fresh food described above reduces the a ^* value (red fading) Broccoli fading is quantified as an increase in L ^* value or b ^* value (brightness, yellow increase), and decay is a decrease in L ^* value (decrease brightness). Therefore, every time the food image is received from the food image management unit 4, the time change detection unit 5 detects and stores the a ^* value, the b ^* value, and the L ^* value.

In addition, the color index is not limited to the L ^* a ^* b ^* color system, but the RGB color system or Munsell table, which is a color system in which the primary colors are R (red), G (green), and B (blue). Even if another color system such as a color system is used, a color change (red fading, yellow increase, etc.) and a brightness change can be quantified.

  The time change detection unit 5 may store a change in color distribution. In this case, the time change detection unit 5 stores a large number of color samples in advance as a database, and calculates the similarity by comparing the color detected from the food image received from the food image management unit 4 with the color sample. Then, the entire food image is classified for each color corresponding to the most similar color sample, and the area occupancy for each color is calculated.

  FIGS. 7A and 7B are diagrams illustrating an example of a temporal change in the food image in the temporal change detection unit 5. FIG. 7A shows a food image when food (meat) is received and a color distribution detected based on the food image. FIG. 7B shows a food image after a sufficient time (T time) for the food to oxidize and a color distribution detected based on the food image. T is a value of 0 or more. In FIGS. 7A and 7B, the area occupancy (vertical axis) for each color (horizontal axis) corresponding to each color sample in the food image is shown by a bar graph. Every time the food image is received from the food image management unit 4, the time change detection unit 5 calculates and stores the color distribution data of the newly received food image.

  Moreover, the time change detection part 5 preserve | saves a projection area, a perimeter, the length of the longest axis etc. as a parameter | index which detects the change of a magnitude | size, for example. Moreover, a pattern may be extracted from the outline shape of food and the pattern may be stored. Here, the time change detection unit 5 detects and stores changes in the color and size of the food image, but may detect and store only the color change or only the size change. .

  In this way, the food deterioration state estimation unit 6 estimates the food deterioration state based on the temporal change in the color and size detected and stored by the time change detection unit 5. The estimation of the deterioration state of the food by the food deterioration state estimation unit 6 is preferably performed not at every opening / closing of the refrigerator compartment door 101 but at every predetermined time.

  The state of deterioration of the food varies depending on the type of food (meat, fish, vegetables, etc.), but here, the state of deterioration of the food is determined using a uniform criterion regardless of the type of food. Therefore, the food deterioration state estimation unit 6 performs determination based on a plurality of indices for one food image, and determines that “it is necessary to consume early” when at least one index exceeds a reference value. .

For example, when the L ^* a ^* b ^* color system is used as the color index, the food deterioration state estimation unit 6 includes the a ^* value, b ^* value, and L ^* value stored in the time change detection unit 5. If at least one of the values exceeds the reference value, it is determined that the food corresponding to the food image “needs to be consumed early”.

  Further, when the color distribution shown in FIGS. 7A and 7B is used, it is based on the change in the color distribution (area occupancy for each color corresponding to each color sample) stored in the time change detection unit 5. When the area occupancy of the specific color exceeds the reference value, the food corresponding to the food image is determined to be “need to be consumed early”.

  In the example shown in FIGS. 7 (a) and (b), the color of the food (meat) is oxidized due to oxidation in FIG. 7 (b) after T time compared to FIG. 7 (a) when the food is received. It changes dark (actually brown), and the area occupancy of the dark color increases. Brown is the color that metmyoglobin exhibits upon oxidation or methation.

  Therefore, for example, based on the knowledge such as “the color corresponding to the 10th color sample increases as the metrification progresses”, “the color corresponding to the 20th color sample increases as the decaying progresses”, etc. A reference value is registered in advance in the state estimation unit 6 for each color that reflects deterioration.

  Then, the food deterioration state estimation unit 6 has at least one of the area occupancy rates of a plurality of colors registered in advance (for example, colors corresponding to a plurality of color samples such as No. 10 and No. 20) exceeding a reference value. If it is determined that the food image corresponds to the food image, it is determined that “the food may be deteriorated”.

  In addition, since the food is deflated or shrunken due to deterioration, the deterioration state can be determined based on a change in the size of the food contained in the food image. In this case, based on a change in the size of the food detected and stored by the time change detection unit 5 (for example, the projected area, the perimeter, or the length of the long axis), the food deterioration state estimation unit 6 When the rate of decrease from the time of storage exceeds the reference value, it can be determined that the food needs to be consumed early.

  Alternatively, based on the contour shape pattern of the food detected and stored by the time change detection unit 5, the food deterioration state estimation unit 6 performs pattern matching with the contour shape pattern of the food at the time of warehousing, and the error is the reference. When the time is exceeded, it may be determined that the food needs to be consumed early.

  Further, in the determination based on the color distribution described with reference to FIGS. 7A and 7B, instead of obtaining the area occupancy for each color corresponding to each color sample, if an absolute area value is obtained, It is possible to make a judgment that reflects the change in size as well as the change in color.

  The food deterioration state estimation unit 6 determines the deterioration state of each food image stored in the time change detection unit 5. The display unit 8 a of the mobile terminal 8 displays the estimation result of the deterioration state by the food deterioration state estimation unit 6 along with the food image stored in the food image management unit 4.

  FIGS. 8A and 8B are diagrams illustrating display examples of the deterioration state of food on the mobile terminal 8. In the display example shown in FIG. 8A, the food deterioration state estimation unit 6 determines that “it is necessary to consume early” among the food images displayed on the stock food list display screen 23 described in FIG. An alert (in this case, an exclamation point) is superimposed on the food image. Thereby, the user who looked at the display part 8a of the portable terminal 8 can grasp | ascertain that the food of the food image in which the alert was displayed is a food which should be consumed at an early stage. In other words, it becomes obvious to the user which of the plurality of foods stored in the refrigerator compartment 100 should be consumed at an early stage. As a result, it is possible to suppress wastefully discarding food due to the expiration of the expiration date or the expiration date.

  Therefore, for example, when food whose expiration date or expiration date has passed is stored in the refrigerated room 100 and the user is wondering whether to purchase the food at a shopping destination, Since the expiration date or the expiration date of the target food can be confirmed on the display unit 8a of the portable terminal 8, it is possible to prevent forgetting to purchase food that is stored but cannot be used.

  In the display example shown in FIG. 8A, an alert is superimposed on the food image, but the display is not limited to such an example, and based on the estimation result of the food deterioration state estimation unit 6, it is consumed early. Other indications may be used as long as the user can be notified of the food to be used.

  The display unit 8a of the portable terminal 8 may also display a change in color or size of the food image detected by the time change detection unit 5 as schematically shown in FIG. 8B. In this case, for example, when one food image is selected on the inventory food list display screen 23 as shown in FIG. 6, the food change display screen 24 that displays a change in color or size of the food image for each time is displayed. Is displayed on the display unit 8a of the portable terminal 8 (in FIG. 8B, since raw meat is assumed as food, the chilled room is used as a display example). Thereby, the user can grasp | ascertain the degradation condition of foodstuff from an image.

  In addition, the case where the refrigerator compartment door 101 is not opened and closed over a long period of time is also assumed. Therefore, for example, even when the latest refrigeration room door 101 is opened and closed, there is a possibility that the deterioration of the food progresses over time even if the signs of the deterioration of the food (for example, change to brown) are slight. Therefore, the food deterioration state estimation unit 6 estimates the food deterioration state based on changes in the color and size of each food image stored in the time change detection unit 5 at regular intervals, and the estimation result It is desirable to predict a time (notification time to the user) that should prompt the user to consume food based on this, and display an alert on the display unit 8a of the mobile terminal 8 when the predicted time is reached.

<< 1-3 >> Effect As described above, according to the refrigerator 1000 according to the first embodiment, the food image is extracted from the storage room image in the storage room (for example, the refrigerator room 100) photographed by the photographing device 2. And displays a food image and information related to the deterioration of the food (for example, information indicating the deterioration state of the food) on the display unit 8a of the portable terminal 8. Therefore, even if the user is a place other than his home, for example, a shopping destination, the user can grasp the food stored in the storage room of the refrigerator 1000 and can easily know the food to be consumed particularly early in that. it can.

  As a result, the refrigerator 1000 can prompt the user to consume the food by the portable terminal 8 before the deadline (for example, the expiration date) regarding the food, and wasteful disposal of the food is suppressed. Further, it is possible to reduce the empty space in the storage room by leaving food in the storage room for a long period of time, and it is possible to reduce wasteful power consumption. Further, since the food stock status or the food degradation state can be confirmed on the display unit 8a of the mobile terminal 8 without opening the refrigerator compartment door 101, the temperature in the refrigerator compartment 100 due to the opening and closing of the refrigerator compartment door 101 can be confirmed. The increase can be suppressed and the power consumption can be reduced.

  In addition, by displaying the entry / exit identification information and the time information together with the food image on the display unit 8a of the portable terminal 8, the user can easily grasp how long a certain food has been stored in the refrigerator compartment 100. can do. In addition, when inventory food is confirmed from a shopping destination, purchase of the food can be determined in consideration of the expiration date or the expiration date of the inventory food.

  In addition, the food image is stored in association with the entry / exit identification information and the time information, and the deterioration state of the food is estimated based on these, so that the change in the food image and the passage of time since the entry are taken into account. The deterioration state of food can be estimated with high accuracy.

  Further, every time the refrigerator door 101 is opened and closed, the imaging device 2 captures a storage room image, and the food image extraction unit 3 extracts a food image from the storage room image, so that the food in the refrigerator room 100 changes. Information is updated whenever there is a possibility. Therefore, the user can know the stock food and its deterioration state more accurately.

  Moreover, since the food deterioration state estimation part 6 estimates the deterioration state of food based on the change of the color or the size (or both) of the food image, the detection accuracy of deterioration such as oxidation (meth) and rot is improved. Can be increased.

<< 1-4 >> Modifications In the above description, the food image management unit 4 identifies food warehousing or warehousing, but in addition to food warehousing and warehousing, food movement in the refrigerator compartment 100 is identified. May be. The movement of food is identified by the following method, for example. That is, as described above, the food image extraction unit 3 obtains a first difference image that is a difference image between the storage room image at the previous door opening and closing time and the current door opening and closing time. Further, whether or not there is a second differential image that is a difference image between the storage room image at the time of opening and closing the door two times before and the storage room image at the time of the previous door opening and closing coincides with the first difference image. Is determined by a matching process. As a result of the matching process, when there is a second differential image that matches the first differential image, it is estimated that the food has moved in the refrigerator compartment 100.

  When the food deterioration state estimation unit 6 calculates the elapsed time from the receipt of the food (that is, the stock period), the stock period before the food moves in the refrigerator compartment 100 is also included, so that the food deterioration state is calculated. The accuracy of estimation and notification to the user can be improved.

  In the above description, the imaging device 2 is provided in the refrigerator compartment 100, but the imaging device 2 (or other imaging device) may be provided in a storage room other than the refrigerator compartment 100. For example, you may equip the ceiling surface of the freezer compartment 400 or the vegetable compartment 500 with the imaging device 2 (or other imaging device). Also in this case, the food image extraction unit 3 extracts and extracts the food image from the storage room image (for example, the entire image in each storage room) of each storage room photographed by the imaging device 2 (or another imaging device). The food image is stored in the food image management unit 4 in association with the supplementary information, the deterioration state is estimated by the time change detection unit 5 and the food deterioration state estimation unit 6, and the food image and the food deterioration state are displayed on the operation panel 1. By displaying, the same effect as the case where the image capturing apparatus 2 is provided in the refrigerator compartment 100 can be obtained.

<< 2 >> Embodiment 2
<2-1> Configuration FIG. 9 includes a schematic configuration of the refrigerator compartment 100a of the refrigerator 1000a according to Embodiment 2 of the present invention (including a cross-sectional structure taken along line II-II in FIG. 1 and a configuration of the control device 1001a). ) And the portable terminal 8. 9, components that are the same as or correspond to the components shown in FIG. 2 are assigned the same reference numerals as those shown in FIG. Hereinafter, differences from the first embodiment will be mainly described.

  As shown in FIG. 9, in addition to the configuration of the control device 1001 of the first embodiment, the control device 1001a that is a control circuit (processing circuit) provided in the refrigerator 1000a according to the second embodiment has food The image identification unit 9 and a food information storage unit (food image database) 10 that stores in advance at least one of the shape, size, color, and exterior label of the food in association with the type of food. It has been. The control device 1001a can be configured by a control board having a processing circuit. The control device 1001a includes a food image extraction unit 3, a food image management unit 4, a time change detection unit 5, a food deterioration state estimation unit 6, and a wireless communication unit 7 as a communication device. The communication device may be either a wireless communication unit or a wired communication unit. Further, all or part of the food image extraction unit 3, the food image identification unit 9, the food image management unit 4, the time change detection unit 5, and the food deterioration state estimation unit 6 is stored in a program that stores a program as software. You may implement | achieve using the memory as an apparatus (memory | storage part) and the processor as an information processing apparatus which performs the program stored in this memory (for example, with a computer). The food image extraction unit 3, the food image identification unit 9, the food image management unit 4, the time change detection unit 5, and the food deterioration state estimation unit 6 are stored in a memory that stores a program as software and in this memory. It may be realized by a combination of a processor that executes a program and a processing circuit.

  The food image database 10 is communicably connected to the food image identification unit 9. For each food type, the food image database 10 has distinctive attributes (for example, the shape, size, color, and exterior label of the food) for distinguishing the food type from other food types. Are stored in association with each type of food.

  FIG. 10 is a table showing an example of contents stored in the food image database 10 referred to by the food image identification unit 9 of the refrigerator 1000a according to the second embodiment. The food image database 10 in FIG. 10 stores the food itself belonging to the type of food or the characteristic amount of the container containing the food in association with the type of the food. In the example of FIG. 10, the food image database 10 stores information such as shape (outline), size (long axis length), representative color (RGB value), and label (character) for each type of food. It is stored as a feature value.

  As the shape for each type of food stored in the food image database 10, instead of the shape (contour) shown in FIG. 10, or in addition to the shape (contour) shown in FIG. For example, a geometric shape closest to the outer shape of the food (for example, information on a columnar shape, a trapezoidal shape, a rectangular parallelepiped shape, a cubic shape, a triangular prism shape, etc.) may be used. In this case, in the identification by the food image identification unit 9, for example, matching with the shape stored in the food image database 10 is performed by a method such as pattern matching. As for colors, in addition to the RGB values of the representative colors, other index values such as brightness and saturation may be used.

<< 2-2 >> Operation Next, an example of the operation of the refrigerator 1000a according to the second embodiment will be described with reference to FIGS. 9, 10, and 11 focusing on differences from the first embodiment. FIG. 11 is a flowchart showing an operation performed each time the refrigerator compartment door 101 of the refrigerator 1000a according to the second embodiment is opened and closed. 11, the same processing steps as those shown in FIG. 3 are denoted by the same reference numerals as those shown in FIG.

  In the operation of the refrigerator 1000a shown in FIG. 11, the food image extraction unit 3 starts the process of detecting the opening / closing of the refrigerator door 101 (step S11), and the food image extracting unit 3 performs the previous door opening / closing of the storage room image. The operation up to the process of extracting the image of the changed portion of the storage room image by calculating the difference from the storage room image at the time (step S14) is the same as the operation of the control device 1001 of the first embodiment shown in FIG. It is. The food image extraction unit 3 adds the extracted food image and time information that is the current time at the time of the extraction as supplementary information, and in the first embodiment, the food image management includes the food image and the supplementary information. Although it transmitted to the part 4, in Embodiment 2, it transmits to the food image identification part 9 first (step S21).

  The food image identification unit 9 refers to the information stored in the food image database 10 shown in FIG. 10 and, based on the food image received from the food image extraction unit 3, stores the storage room (here, the refrigerator room 100). ) To identify the type and amount of food stored in () (step S22). In this identification, the food image identification unit 9 is the feature quantity of which food type among the types of food stored in the food image database 10 as individual food images extracted by the food image extraction unit 3. Is matched.

  When there is a feature quantity stored in the food image database 10 that matches an image of each food, the food image identification unit 9 corresponds to the feature quantity matched in the food image database 10. Judge that the type is attached. The food image identification unit 9 replaces the food type of the image with the most similar food type when there is no data that matches the feature amount in the food image database 10, or the food image of the image May identify only the size as an unknown storage.

  The food image identification unit 9 adds the type and amount information of the food identified in this way as supplementary information to the food image received from the food image extraction unit 3 and the time information (shooting time information) about each food image. It adds and transmits to the food image management part 4 (step S23).

  The food image management unit 4 receives the food image and incidental information (time information, food type and quantity information) from the food image identification unit 9, and as the incidental information from the first embodiment, the food type and quantity information is Similarly, the food images are rearranged in time series based on the time information and stored (step S16). In addition, the food image management unit 4 determines whether the food is stored or delivered as in the first embodiment, and stores the determination result (entry / exit identification information) as supplementary information of the food image. Further, the food image management unit 4 transmits the food image and accompanying information (time information, food type and quantity information, and entry / exit identification information) to the time change detection unit 5 and the wireless communication unit 7.

  In the next step S17, as in the first embodiment, the time change detection unit 5 receives the food image and the supplementary information from the food image management unit 4, and detects temporal changes in the color and size of the food image. And save.

  Here, in the first embodiment, based on the temporal change in the color and size detected by the time change detection unit 5, the food deterioration state estimation unit 6 determines a uniform determination criterion regardless of the type of food. Was used to estimate the state of food degradation. That is, the food deterioration state estimation unit 6 performs determination based on a plurality of indices for one food image, and when at least one index exceeds a reference value, the food corresponding to the food image is “early”. It is determined that it needs to be consumed.

  However, the deterioration state of food varies depending on the type of food (meat, fish, vegetables, etc.). For example, meat and fish, which are the same fresh food, are generally cooked, even if the color changes from red to brown due to oxidation (metoization), even if the fish eaten raw as sashimi is severely degraded It is a level of deterioration that is not a problem for meat, and in vegetables or fruits, in broccoli, bright green fades to yellow, but in tangerines, green becomes yellow when it is ripe and ready to eat. May indicate that Furthermore, the leafy vegetables are deflated or shrunk by transpiration and become smaller, but the size of what is in a container such as milk or dressing does not change due to deterioration.

  Therefore, in Embodiment 2, when the food deterioration state estimation unit 6 estimates the food deterioration state, the food type and amount information identified by the food image identification unit 9 is reflected (referred), The reference value for judging deterioration based on the type and amount information of food is changed. Thereby, the estimation precision of the deterioration state of foodstuffs can be improved.

  For example, when the food deterioration state estimation unit 6 uses the color distribution shown in FIGS. 7A and 7B, “the color corresponding to the number 10 color sample increases as metation progresses”, A reference value for judging deterioration is registered in advance based on knowledge such as “corresponding to the color sample No. 20 increases as corruption progresses”. The reference value is changed depending on the type of food, and the color change reference value is set to be larger than that of fish (for example, tuna) whose color changes in the same way due to methodization or decay.

<< 2-3 >> Effect As described above, according to the refrigerator 1000a according to the second embodiment, the food image is extracted and managed from the storage room image that is the image in the refrigerator room 100 photographed by the photographing device 2. Then, the food image and information regarding the deterioration of the food (for example, information indicating the deterioration state) are displayed together on the display unit 8a of the portable terminal 8. Therefore, the user can grasp the food stored in the refrigerator compartment 100 even at a place other than home (for example, a shopping destination), and can easily grasp the food that should be consumed particularly quickly.

  Also, for the same food (for example, sliced meat that can be photographed almost entirely and block meat that can only be photographed on the surface), even if the area occupancy that has changed color due to methodization or spoilage is the same, the block meat is inside Therefore, the deterioration can be estimated in advance by setting the reference value to a small value reflecting the amount information.

  In addition, the change in the size of food over time reflects information on the type of food, and the food contained in the container is excluded from the judgment criteria for deterioration, so that the accuracy of estimation of the deterioration state of food is estimated. Can be improved.

<< 3 >> Embodiment 3
<< 3-1 >> Configuration FIG. 12 includes a schematic configuration of a refrigerator 1000b according to Embodiment 3 of the present invention (including a cross-sectional structure of the refrigerator shown in FIG. 1 taken along line XII-XII and a configuration of the control device 1001b). ) And the portable terminal 8. 12, components that are the same as or correspond to the components shown in FIG. 9 are given the same reference numerals as those shown in FIG. Below, it demonstrates centering on difference with Embodiment 1 and 2. FIG.

  As shown in FIG. 12, the refrigerator 1000b according to Embodiment 3 cools the air supplied to each storage room (refrigeration room 100, chilled room 110, switching room 300, freezer room 400, and vegetable room 500). A refrigeration cycle circuit for supplying the air to each storage chamber with air cooled by the refrigeration cycle circuit.

  The refrigeration cycle circuit includes a compressor 1002, a condenser (not shown) that condenses the refrigerant discharged from the compressor 1002, a throttling device (not shown) that expands the refrigerant flowing out of the condenser, and a throttling device. And a cooler 1003 that cools the air supplied to each storage chamber by the refrigerant expanded in step (b). The compressor 1002 is arrange | positioned at the lower part of the back side of the refrigerator 1000b, for example. The cooler 1003 is provided on the upstream side of a cooling air passage 1010 described later. The cooling air passage 1010 is also provided with an air conveyance device 1004 for sending the air cooled by the cooler 1003 to each storage room (that is, for circulating the air in the refrigerator 1000b).

  The air path for supplying the air cooled by the refrigeration cycle circuit to each storage room is, for example, a cooling air path 1010, a return air path 1020, a refrigeration room return air path 103, and a vegetable room return air path 501. It consists of

  The cooling air passage 1010 is a ventilation passage through which the air cooled by the cooler 1003 is conveyed to each storage chamber, and is provided, for example, on the back surface of the refrigerator 1000b. The amount of cooling air flowing into each storage room is adjusted by one or a plurality of dampers (only the refrigerating room damper 104 for the refrigerating room 100 is shown) provided in the cooling air passage 1010.

  The return air passage 1020 is a ventilation passage through which the air that has cooled each storage chamber is conveyed to the cooler 1003. The refrigerator compartment return air passage 103 is a ventilation passage through which the air that has cooled the refrigerator compartment 100 and the chilled chamber 110 is conveyed to the vegetable compartment 500. The air that has cooled the refrigerator compartment 100 and the chilled chamber 110 merges with the air that has cooled the vegetable compartment 500 in the vegetable compartment return air passage 501, and is conveyed to the cooler 1003.

  The control device 1001b provided on the back surface of the refrigerator 1000b includes a time limit estimation unit 11, a set temperature calculation unit 12, and a cooling control unit 13 in addition to the components of the control device 1001a of the second embodiment. . In addition to the configurations of the first and second embodiments, the time limit setting unit is connected to the wireless communication unit 7 provided in the control device 1001b and can communicate with the refrigerator 1000b from inside or outside the house. 14 is provided. Of the food image extraction unit 3, the food image identification unit 9, the food image management unit 4, the time change detection unit 5, the food deterioration state estimation unit 6, the time limit estimation unit 11, the set temperature calculation unit 12, and the cooling control unit 13 In whole or in part, a memory as a storage device (storage unit) that stores a program as software and a processor as an information processing device that executes the program stored in the memory (for example, by a computer) It may be realized. The food image extraction unit 3, the food image identification unit 9, the food image management unit 4, the time change detection unit 5, the food deterioration state estimation unit 6, the expiration date estimation unit 11, the set temperature calculation unit 12, and the cooling control unit 13 It may be realized by a combination of a memory that stores a program as software, a processor that executes the program stored in the memory, and a processing circuit. The deadline setting unit 14 of the mobile terminal 8 may be realized using a memory that stores a program as software and a processor as an information processing apparatus that executes the program stored in the memory.

  The deadline setting unit 14 of the portable terminal 8 sets a deadline setting screen 25 for setting a deadline (for example, a consumption deadline or a best before date) for the user to consume food on the display unit 8a of the portable terminal 8. Is displayed on the display unit 8a. FIGS. 13A to 13C are diagrams showing examples of food image display screens on the display unit 8a of the mobile terminal 8 communicating with the refrigerator 1000b according to the third embodiment. FIG. 13A shows an example of the time limit setting screen 25. For example, when one food image is selected on the above-described stock food list display screen 23 (FIG. 6), the time limit setting screen 25 is displayed on the display unit 8 a of the mobile terminal 8. The time limit setting screen 25 is configured so that the user can input (set) a time limit (for example, a consumption time limit or a best before date) related to food for the selected food image.

  Further, when the food is received in the refrigerator compartment 100, the deadline setting unit 14 of the portable terminal 8 displays a display screen similar to the deadline setting screen 25 shown in FIG. 13A on the display unit 8a. The terminal 8 can accept an input of a user setting deadline as a deadline for food (for example, by a touch panel provided in the display unit 8a). The portable terminal 8 that has received the input of the user setting time limit can transmit information indicating the time limit regarding the input food to the refrigerator 1000b.

  The expiration date estimation unit 11 is connected to the portable terminal 8 via the food deterioration state estimation unit 6, the set temperature calculation unit 12, and the wireless communication unit 7 so as to be communicable. The deadline estimation unit 11 estimates, for example, a consumption deadline or a shelf life as a deadline for food recommended to consume food based on the estimation result of the food deterioration state in the food deterioration state estimation unit 6, The estimation result (the estimated time limit for the food) is displayed on the display unit 8a of the portable terminal 8, and the estimation result is transmitted to the set temperature calculation unit 12.

  In this case, for example, a deadline notification screen as shown in FIG. 13B is displayed on the display unit 8a of the portable terminal 8 for the food whose remaining period until the deadline related to the food estimated by the deadline estimation unit 11 is short. 26 is displayed. In the time limit notification screen 26 shown in FIG. 13B, the food image and the remaining period (for example, one day) until the time limit related to the food corresponding to the food image are displayed.

  The set temperature calculation unit 12 is communicably connected to the portable terminal 8 via the time limit estimation unit 11, the cooling control unit 13, and the wireless communication unit 7. At this time, the set temperature calculation unit 12 is communicably connected not only to the display unit 8 a of the mobile terminal 8 but also to the time limit setting unit 14 built in the mobile terminal 8. The set temperature calculation unit 12 is suitable for food based on the time limit (estimated time limit) related to food estimated by the time limit estimation unit 11 and the user set time limit that is a time limit related to food set by the user in the time limit setting unit 14. Calculate the set temperature (recommended set temperature).

  Then, the set temperature calculation unit 12 displays a recommended temperature notification screen 27 as shown in FIG. 13C, for example, on the display unit 8a of the portable terminal 8 in order to notify the user of the recommended set temperature. In addition, the set temperature calculation unit 12 transmits to the cooling control unit 13 the control content necessary for setting the temperature of the storage room to the calculated recommended set temperature.

  The cooling control unit 13 is connected to the compressor 1002, the air conveyance device 1004, the refrigerating room damper 104, and other storage room dampers (not shown) in addition to the set temperature calculation unit 12, and each of these devices is connected. Control. The cooling control unit 13 receives an input of the set temperature from the set temperature calculation unit 12, and sends a control signal to at least one of the compressor 1002, the air conveyance device 1004, the refrigerator compartment damper 104, and the dampers of other storage rooms. Send to control the cooling of the storage room (change the cooling capacity). The refrigerating room damper 104 and other storage room dampers constitute a “blowing air volume control device”.

<3-2> Operation Next, an example of the operation of the refrigerator 1000b according to the third embodiment will be described with reference to FIGS. 12 and 13 focusing on differences from the first and second embodiments.

  In FIG. 12, when the refrigerator compartment door 101 is opened and closed, the photographing apparatus 2 photographs the inside of the refrigerator compartment 100 (preferably, the entire refrigerator compartment 100), and the photographed image in the refrigerator compartment 100 obtained by photography. Is transmitted to the food image extraction unit 3. The food image extraction unit 3 extracts a food image based on the difference between the storage room image at the previous door opening / closing time and the current storage door image at the door opening / closing time, and adds time information (shooting time information) to the food image. And transmitted to the food image identification unit 9.

  The food image identification unit 9 refers to the information stored in the food image database 10 shown in FIG. 10 and based on the food image received from the food image extraction unit 3, the storage room (here, the refrigerator room 100). ) Identify the type and amount of food stored in the box. The food image identification unit 9 adds the identified food type and amount information as supplementary information to the food image received from the food image extraction unit 3 and the time information attached to each food image (shooting time information). The food image and the accompanying information are transmitted to the food image management unit 4.

  The food image management unit 4 receives food images and supplementary information (for example, time information, food type and quantity information) from the food image identification unit 9, and rearranges the food images in time series based on the time information. The rearranged food image and accompanying information are stored. In addition, the food image management unit 4 determines whether the food is stored or extracted, and adds the determination result (entry / exit identification information) to the supplementary information of the food image and stores it. Further, the food image management unit 4 transmits the food image and incidental information (time information, food type and quantity information, and entry / exit identification information) to the time change detection unit 5 and the display unit 8a of the portable terminal 8. (See FIGS. 5A, 5B and 6).

  Next, the time change detection unit 5 detects temporal changes in the color and size of the food image transmitted from the food image management unit 4 and stores the detection result. The food deterioration state estimation unit 6 reflects the food type and quantity information identified by the food image identification unit 9 based on the temporal change in color and size stored in the time variation detection unit 5. To estimate the state of food degradation. Furthermore, the food deterioration state estimation unit 6 displays the estimation result of the food deterioration state together with the food image on the display unit 8a of the mobile terminal 8 (see FIG. 8).

In the third embodiment, based on the food deterioration tendency indicating the progress of food deterioration (speed of progress of food deterioration) estimated by the food deterioration state estimation unit 6, the deadline estimation unit 11 Estimated deadlines (such as expiry date or expiry date) for which consumption is recommended. The tendency of food deterioration can be expressed by a value indicating the speed of progress of food deterioration, for example, the degree of food deterioration per day, and a curve (a ^* value curve) in FIG. ). And as shown in FIG.13 (b), the deadline estimation part 11 displays on the display part 8a of the portable terminal 8 the food image of the food which the estimated deadline is approaching, and notifies a user, It is intended to prevent wasteful consumption and prevent wasteful disposal of food.

FIGS. 14A to 14C are schematic diagrams showing a method for estimating the deadline (here, the expiration date) of the food in the deadline estimating unit 11. 14A to 14C schematically illustrate an example in which the time limit estimation unit 11 estimates the expiration date from the temporal color change (decrease in a ^* value) of the meat stored in the time change detection unit 5. Is shown. 14 (a) and 14 (b) are schematic diagrams showing food images at the time of warehousing (elapsed time 0) and elapsed time T [hour], and FIG. 14 (c) shows the temporal change of the a ^* value. It is a graph which shows. In FIG.14 (c), a vertical axis | shaft is a ^* value of a foodstuff (meat), and a horizontal axis is elapsed time (unit: time). The elapsed time 0 is when the food is received (see FIG. 7A), and the elapsed time T [hour] is an elapsed time sufficient for the food to be oxidized (see FIG. 7B). In FIG. 14C, a solid line (curve) indicates a change in the a ^* value, and a broken line (straight line) parallel to the time axis indicates the consumption limit of the a ^* value.

The change in the a ^* value shown in FIG. 14C is obtained by measuring the a ^* value every time the refrigerator door 101 is opened and closed, and obtaining it as a curve passing through each a ^* value. It is decreasing. and reduction characteristics of the a ^* value, and a detection value of a ^* value at the elapsed time T [time], it is possible to estimate the expiration date is time a ^* value reaches the consumption limit. The food deterioration state estimation unit 6 estimates the expiry date in this way, and before the expiry date is reached, the food that has a short period until the expiry date is displayed to the user via the display unit 8a of the portable terminal 8. Is notified. The notification to the user may be performed by a method of displaying an alert together with the food image (for example, FIG. 8A), or a method of displaying the remaining time until the expiration date (for example, FIG. 13B). It may be done at.

  Thereby, the user can know the time limit before reaching the food expiry date or the expiration date. Thereby, the user can consume the notified food early and prevent forgetting to consume it. For this reason, wasteful disposal of food can be suppressed.

Incidentally, consumption limit shown in FIG. 14 (c) may be a decline from an initial value of the absolute value or a ^* value preset a ^* value, or detected a ^* value The actual value reflecting the characteristic data of the time change may be used. If an appropriate deadline (for example, expiration date, best-before date, or other deadline) that matches the actual use state can be set as the deadline for food, the deadline information useful for the user can be notified, The amount of waste can be reduced.

Further, FIG. 14C shows an example in which the deadline for food is estimated from the decrease characteristic of the a ^* value that is a general color index, but the time change data to be processed is the example of FIG. 14C. It is not limited to. For example, as shown in the examples of FIGS. 7A and 7B, it is set by using the change in the area occupancy of various color samples or the deterioration tendency of food estimated from the increase or decrease in specific color samples. You may estimate the time limit (for example, expiration date) regarding a foodstuff by calculating the time (arrival time) estimated that the consumption limit will be reached.

  The deadline for food estimated by the deadline estimation unit 11 is displayed on the display unit 8 a of the mobile terminal 8, while being transmitted to the set temperature calculation unit 12. Further, the user setting time limit set by the user in the time limit setting unit 14 provided in the portable terminal 8 is also transmitted to the set temperature calculation unit 12. The set temperature calculation unit 12 determines the set temperature in the storage room by comparing the time limit related to the food estimated by the time limit estimation unit 11 and the user set time limit in the time limit setting unit 14.

  That is, when the deadline for food estimated by the deadline estimation unit 11 is longer than the deadline set by the user in the deadline setting unit 14, it is assumed that the quality of the food is maintained until the deadline desired by the user. There is no need to change the set temperature in the storage room.

On the other hand, when the deadline (estimated deadline) regarding the food estimated by the deadline estimating unit 11 is shorter than the user set deadline set in the deadline setting unit 14, the estimated deadline is brought closer to the user set deadline. Reduce the set temperature in the storage chamber. This is because if the temperature in the refrigerator compartment 100 is lowered, the speed of progress of the deterioration of the food (gradient of decrease in the a ^* value) becomes gentle.

  In the refrigerator 1000b, the air in the refrigerator cooled by the cooler 1003 is conveyed to each storage chamber by the air conveying device 1004 via the cooling air passage 1010. Then, the return air that has cooled each storage chamber returns to the cooler 1003 again via the return air passage 1020. At this time, the air (for example, −30 ° C. to −25 ° C.) cooled by the cooler 1003 is distributed to each storage room by opening and closing a plurality of dampers such as the refrigerator compartment damper 104. By adjusting the amount of cooling air flowing into each storage room by opening and closing the plurality of dampers, individual temperatures are set for each storage room.

  For example, the inflow damper of the freezing room 400 (for example, −22 ° C. to −16 ° C.) set to the lowest temperature is almost fully opened, and the inflow of the vegetable room 500 (for example, 3 ° C. to 9 ° C.) set to the highest temperature. The damper is almost fully closed. And for example, the vegetable room 500 is indirectly cooled with the return air which cooled the refrigerator compartment 100 (for example, 0 degreeC-6 degreeC) whose temperature setting is lower than the vegetable room 500, and the chilled room 110 (for example, 0 degreeC-2 degreeC). Thus, the temperature of each storage room is adjusted.

  Here, the set temperature of each storage room can be adjusted from about ± 2 ° C. to about ± 3 ° C. according to overcooling or undercooling. For example, the set temperature of the freezer compartment 400 can be changed within a range of about −25 ° C. to −13 ° C., and the set temperature of the refrigerator compartment 100 can be changed within a range of about −2 ° C. to 9 ° C.

  The control device 1001b controls the drive of the cooling chamber damper based on the detection value of a temperature sensor (not shown) that detects the temperature in each storage chamber, so that the inside temperature is within an appropriate temperature range. When lowering the temperature of a specific storage room that is under-cooled (for example, lowering the temperature of only the refrigerating room 100), by increasing the opening degree of the refrigerating room damper 104, Increase the inflow of cooling air. Moreover, when lowering the temperature of the plurality of storage rooms, it is necessary to increase the cooling capacity of the refrigeration cycle circuit by increasing the number of rotations of the compressor 1002 and the amount of air transported by the air transport device 1004. Therefore, when reducing the temperature of a some store room, the power consumption of the refrigerator 1000b increases.

  On the other hand, when raising the temperature of a specific storage room in an overcooled state, the opening degree of the damper that opens and closes the ventilation path leading to this specific storage room is reduced to reduce the cooling air to this specific storage room. Reduce the amount of inflow. Moreover, when raising the temperature of several storage chambers, the rotation speed of the compressor 1002 and the conveyance air volume by the air conveyance apparatus 1004 are decreased. In this case, the power consumption of the refrigerator 1000b decreases.

  Therefore, the cooling control unit 13 determines the opening degree of the damper of each storage room such as the refrigerator compartment damper 104, the rotation speed of the compressor 1002, and the air transfer device 1004 according to the set temperature calculated by the set temperature calculation unit 12. Controls the amount of air transported by At this time, particularly when it is desired to lower the temperature of a specific storage room, the number of rotations of the compressor 1002 and the amount of air transported by the air transport device 1004 are not increased as much as possible, and the ventilation path for sending the cooling air to the corresponding storage room. It is desirable to decrease the temperature by increasing the opening of the damper. Since the power consumption does not change if only the damper opening is changed, it is possible to satisfy the consumption expiration date or the expiration date desired by the user while eliminating waste of power consumption due to unnecessary cooling.

<< 3-3 >> Effect As described above, according to the refrigerator 1000b according to the third embodiment, the food deterioration state estimation unit 6 estimates the food deterioration tendency, and based on the estimation result, the time limit estimation unit 11 performs the estimation. A deadline (such as a consumption deadline or a best-before date) regarding the food is estimated, and the user is notified of the food that is close to the estimated deadline. For this reason, forgetting to consume food can be reduced, and wasteful disposal of food can be suppressed.

  Moreover, according to the refrigerator 1000b which concerns on Embodiment 3, it can be confirmed with the portable terminal 8 whether the food which the expiration date or expiration date passed in the refrigerator compartment 100 is accommodated in the refrigerator compartment 100 outside. When the user is wondering whether or not to purchase the target food at the shopping destination, not only whether the target food is in the storage room of the refrigerator 1000b (storage status) but also the expiration date of the target food. Or the expiration date can also be confirmed. In this way, food that is stored in the storage room but cannot be used (because it has expired) can be confirmed outside the home, so that it is possible to prevent forgetting to purchase necessary food.

  In addition, according to the refrigerator 1000b according to the third embodiment, unnecessary occupancy of storage space and waste of power due to continuous storage of food in the refrigerator compartment 100 that has passed a time limit related to food (for example, a time limit for consumption). Consumption can be reduced. For example, the cooling capacity assumed to have been wasted if food was left in the storage chamber is distributed to other food and other storage chambers, so that the rotation speed of the compressor 1002 and the conveyance by the air conveyance device 1004 The air volume can be reduced, and the amount of power consumed by the refrigerator 1000b can be reduced.

  Moreover, according to the refrigerator 1000b which concerns on Embodiment 3, the setting of the time limit by a user is received in the time limit setting part 14, This user setting time limit is compared with the time limit regarding the food estimated by the food deterioration state estimation part 6. The set temperature in the storage chamber is calculated. For this reason, when the period until the user setting deadline is long, a low set temperature at which the deterioration of the food hardly progresses is adopted, so that the quality of the food can be satisfactorily maintained until the user setting deadline. .

<< 4 >> Embodiment 4
<< 4-1 >> Configuration In the fourth embodiment, a network system including the refrigerator described in any of the first to third embodiments will be described. FIG. 15 is a diagram showing a schematic configuration of a network system according to Embodiment 4 of the present invention. In FIG. 15, the same components as those described in the first to third embodiments are denoted by the same reference numerals as those used in the first to third embodiments. As shown in FIG. 15, the network system according to the fourth embodiment includes a home system 2000 including a refrigerator 1000 (or 1000a or 1000b) and an outside system 3000 that can communicate with the home system 2000 via the Internet. ing. The refrigerator 1000 has the function of the refrigerator 1000 (or 1000a or 1000b) described in any of the first to third embodiments.

  As shown in FIG. 15, the in-home system 2000 includes home appliances such as a refrigerator 1000, a room air conditioner, a water heater, an IH (Induction Heating) cooking heater, a microwave oven, and the like. These home appliances and power measuring apparatus 2003 are connected to the system power supply 2001 by the electric power line 2002 and are supplied with power from the system power supply 2001. The in-home system 2000 includes a power measuring device 2003 and a power measuring terminal 2004 for measuring power supplied from the system power supply 2001 to each home appliance through the power line 2002, a home controller 2005, a broadband router 2006, and a communication device. Device communication unit 15.

  The power measuring device 2003 measures the power supplied to each home appliance (that is, the power consumption of each home appliance) and the total power supplied from the system power supply 2001 to the entire home appliance and the power measuring device 2003. Measurement results (history) can be saved. The power measuring apparatus 2003 measures the supplied power by a power measuring terminal 2004 such as a CT (Current Transformer).

  Each of the home appliances such as the refrigerator 1000 and the power measuring device 2003 is provided with a device communication unit 15 for communicating with the in-home controller 2005 via a communication network. The communication network is, for example, a wired or wireless LAN (local area network). Household appliances such as the refrigerator 1000 and the power measurement device 2003 can bidirectionally communicate with the in-home controller 2005 via the respective device communication units 15 in a wired or wireless manner. In addition, the apparatus communication part 15 connected to the refrigerator 1000 plays the same role as the radio | wireless communication part 7 demonstrated in Embodiment 1-3. Therefore, when the wireless communication unit 7 of the refrigerator 1000 can communicate directly with the in-home controller 2005 by wire or wireless, it is not necessary to include the device communication unit 15 for the refrigerator 1000.

  When the device communication unit 15 has a function of performing wired communication, the device communication unit 15 includes a wired communication module such as a serial interface or a driver, for example. When the device communication unit 15 has a function of performing wireless communication, the device communication unit 15 includes a wireless communication module such as Wi-Fi or Bluetooth (registered trademark). Note that the device communication unit 15 may be incorporated in each of the home appliance and the power measurement device 2003 or may be provided outside the home appliance and the power measurement device 2003.

  The in-home controller 2005 can acquire information on the power consumption of each home appliance and the power supplied to the system power supply 2001 measured by the power measurement device 2003 from the power measurement device 2003 via the communication network. Further, the home controller 2005 can acquire information on the operation state of each home appliance via a communication network. Then, the home controller 2005 manages the acquired power consumption, supply power, and operation state information, and if necessary, controls change instructions (for example, change of the operation state) to each home appliance via the communication network. (Instruction) can be transmitted.

  The in-home controller 2005 is connected to the Internet 3001 of the outside system 3000 via the broadband router 2006.

The Internet 3001 is connected to a cloud server 3002 that stores environmental information such as various power information and weather forecasts. The Internet 3001 is also connected to the mobile terminal 8. The portable terminal 8 is, for example, a smartphone, a portable tablet, or the like possessed by the user, as described in the first embodiment.
The cloud server 3002 includes an environment database related to weather information in a specific area. The control device of the refrigerator 1000 may include a food storage environment estimation unit that acquires weather information of a region where the refrigerator is placed or a food production area based on the environment database and estimates a food storage environment. The food deterioration state estimation unit 6 can also estimate the food deterioration state by reflecting the food storage environment estimated by the food storage environment estimation unit.

  The cloud server 3002 is an external server that is communicably connected to the refrigerator 1000 and the mobile terminal 8 via the Internet 3001. The cloud server 3002 preferably includes a personal authentication unit 3003 for authenticating whether the portable terminal 8 has the authority to connect to the refrigerator 1000.

  For example, the cloud server 3002 stores in advance the portable terminal ID of the portable terminal 8 that has the authority to connect to the refrigerator 1000 with respect to the ID (identifier) of the refrigerator 1000. Here, the ID of the refrigerator 1000 is information that can uniquely identify each of the plurality of refrigerators connected to the Internet 3001. The mobile terminal ID is information that can uniquely identify each of a plurality of mobile terminals connected to the Internet 3001.

  Then, the personal authentication unit 3003 confirms whether or not the mobile terminal ID of the mobile terminal 8 is stored as having authority to connect to the refrigerator 1000 with respect to the refrigerator 1000 of the refrigerator 1000. About 8, the presence or absence of the connection authority to the refrigerator 1000 is authenticated.

  The cloud server 3002 that is an external server permits the portable terminal 8 that has been authenticated by the personal authentication unit 3003 to have the authority to connect to the refrigerator 1000 to connect to the refrigerator 1000 via the Internet 3001. That is, the cloud server 3002 prohibits the mobile terminal 8 that is not authenticated to have authority to connect to the refrigerator 1000 by the personal authentication unit 3003 from connecting to the refrigerator 1000 via the Internet 3001.

  The in-home controller 2005 obtains the power consumption of each home appliance or the supply power of the system power supply 2001 from the power measuring device 2003 in the home system 2000, and obtains information on the operating state or the surrounding environment from each home appliance. Further, the home controller 2005 is connected to the Internet 3001 via the broadband router 2006, and can obtain various types of power information or environmental information from the cloud server 3002.

  For example, the in-home controller 2005 of the in-home system 2000 can obtain the power supply / demand situation regarding the area where the in-home system 2000 is placed as power information from the cloud server 3002 of the outside system 3000. When the local power consumption is approaching the upper limit value of the supplied power, the in-home controller 2005 can transmit a power saving instruction to a home electric appliance with particularly high power consumption. Further, in such a case, for the refrigerator 1000, when it is determined that the refrigerator 1000 is overcooled from the set temperature of the storage room and the actual temperature history, an increase in the set temperature can be instructed from the home controller 2005.

  In addition, the home controller 2005 can obtain, from the cloud server 3002, weather forecast information regarding the area where the home system 2000 is installed, for example, as environment information. And by utilizing this environmental information for the control of the refrigerator 1000, it becomes possible to cope with the fluctuation of the cooling load (environmental load) outside the refrigerator 1000. Therefore, the quality of food can be appropriately maintained.

  The environmental information obtained from the cloud server 3002 not only estimates the cooling load outside the refrigerator 1000 and uses it for control, but also reflects it in the estimation of the food deterioration state in the food deterioration state estimation unit 6 of the refrigerator 1000. it can. For example, the in-home controller 2005 obtains environmental information from the cloud server 3002, that is, external cooling load information, and information on the refrigerator internal temperature information (generally stored in the control device 1001 from the refrigerator 1000 in general) ( Obtain the set temperature and measurement temperature. Based on the information, the estimation accuracy can be improved by estimating the deterioration state of the food reflecting the past history and the future prediction environment for the food storage environment.

  In addition, the portable terminal 8 can be connected directly or indirectly to the refrigerator 1000 via the Internet 3001 and the home controller 2005. At this time, only the portable terminal 8 that is authenticated by the personal authentication unit 3003 of the cloud server 3002 to have the authority to connect to the refrigerator 1000 is permitted to connect to the refrigerator 1000. Therefore, for example, by registering the mobile terminal ID of the family mobile terminal 8 in the cloud server 3002 in advance, information such as the stock status and the deterioration status of the food in the storage room of the refrigerator 1000 can be maintained while maintaining security. It becomes possible to share with. For example, in the middle of shopping in a supermarket or the like, it becomes possible to check the stock status of the food in the storage room of the refrigerator 1000 and the deterioration status of the stock food, so the family can purchase the same ingredients redundantly, It can prevent you from forgetting to purchase necessary food.

  In the network system according to the fourth embodiment, all or part of the food image database 10 referred to by the food image identification unit 9 may be provided in the cloud server 3002 that is an external server. By storing the food image database 10 in the cloud server 3002, it is possible to share enormous data regarding the type of food with other users, for example.

  In addition, in the refrigerator 1000 of the network system according to Embodiment 4, other configurations and operations are the same as those of any of the refrigerators 1000, 1000a, and 1000b according to Embodiments 1 to 3.

  The personal authentication means 3003 authenticates the connection authority to the refrigerator 1000 using the ID, but the connection authority authentication method is not limited to this method. In addition to the ID, for example, authentication of the connection authority to the refrigerator 1000 may be performed by biometric authentication such as a user's fingerprint, vein, and iris of the eye, authentication by password, or a combination of these authentication methods. Good.

  For example, the in-home controller 2005 of the in-home system 2000 can obtain the power supply / demand situation regarding the area where the in-home system 2000 is placed as power information from the cloud server 3002 of the outside system 3000. When the local power consumption is approaching the upper limit value of the supplied power, the in-home controller 2005 can transmit a power saving instruction to a home electric appliance with particularly high power consumption. Further, in such a case, for the refrigerator 1000, when it is determined that the refrigerator 1000 is overcooled from the set temperature of the storage room and the actual temperature history, an increase in the set temperature can be instructed from the home controller 2005.

<< 4-2 >> Effects As described above, according to the network system according to the fourth embodiment, the following effects can be obtained in addition to the effects provided by the refrigerator described in the first to third embodiments.

  According to the network system according to the fourth embodiment, the cooling operation (inside the refrigerator) of the refrigerator 1000 (or 1000a, 1000b) according to the fluctuation of the cooling load (environmental load) outside the refrigerator 1000 (or 1000a, 1000b). Therefore, the quality of the food in the refrigerator can be maintained well for a long period of time.

  In addition, according to the network system according to the fourth embodiment, the deterioration state of the food in the storage chamber is estimated by referring to the past history (information) about the food storage environment and reflecting the future prediction environment. Therefore, it is possible to improve the estimation accuracy of the deterioration state of the food.

  Moreover, according to the network system which concerns on Embodiment 4, the stock status of the foodstuff in a storage room, the deterioration state of foodstuff, etc., maintaining security by authenticating the connection authority to the refrigerator 1000 (or 1000a, 1000b) Can be shared with family.

  In the network system according to the fourth embodiment, all or part of the food image database 10 may be provided in the cloud server 3002 that is an external server. For this reason, even if the storage capacity of the memory provided in the refrigerator 1000 (or 1000a, 1000b) is small, enormous data regarding the type of food can be shared with other users and the like.

  DESCRIPTION OF SYMBOLS 1 Operation panel, 2 Imaging device, 3 Food image extraction part, 4 Food image management part, 5 Time change detection part, 6 Food deterioration state estimation part, 7 Wireless communication part (communication apparatus), 8 Portable terminal, 8a Display part, DESCRIPTION OF SYMBOLS 9 Food image identification part, 10 Food image database, 11 Time limit estimation part, 12 Set temperature calculation part, 13 Cooling control part, 14 Time limit setting part, 15 Equipment communication part, 21 Timeline display screen, 22 Entry / exit list display screen, 23 Stock food list display screen, 24 Food change display screen, 25 Time limit setting screen, 26 Time limit notification screen, 27 Recommended temperature notification screen, 100 Refrigeration room (storage room), 101 Refrigeration room door, 102 Shelf, 103 Refrigeration room return Air channel, 104 cold room damper, 110 chilled room, 111 chilled case, 200 ice making room (storage Room), 300 switching room (storage room), 400 freezing room (storage room), 500 vegetable room (storage room), 501 vegetable room return air path, 1000, 1000a, 1000b refrigerator, 1001, 1001a, 1001b control device, 1002 Compressor, 1003 cooler, 1004 air conveying device, 1010 cooling air passage, 1020 return air passage, 2000 in-home system, 2001 system power supply, 2002 power line, 2003 power measuring device, 2004 power measuring terminal, 2005 in-home controller, 2006 broadband Router, 3000 Outside system, 3001 Internet, 3002 Cloud server, 3003 Personal authentication means.

Claims (20)

A storage room for storing food;
A photographing device for obtaining an image in the storage room by photographing the storage room;
A control device for extracting a food image of the food stored in the storage room from the image in the storage room acquired by the imaging device;
A communication device for communicating with a portable terminal having a display unit;
With
The controller is
Generate information on deterioration of food corresponding to the food image from the temporal change of the food image,
The refrigerator is characterized in that the food image and information relating to the deterioration are transmitted to the portable terminal via the communication device.   The control device, together with the information on the food image and the deterioration, stores the warehousing identification information indicating the warehousing of the food corresponding to the food image into and out of the storage room via the communication device. The refrigerator according to claim 1, wherein the refrigerator is transmitted to the portable terminal.   The control device transmits time information indicating a photographing time by the photographing device to the portable terminal via the communication device together with the food image, the information regarding the deterioration, and the warehouse identification information. The refrigerator according to claim 2. The controller is
A temporal change detection unit for detecting temporal changes in the food image;
Based on the temporal change of the food image detected by the time change detection unit, as information about the deterioration, a food deterioration state estimation unit that estimates a deterioration state indicating the degree of deterioration of the current food,
The refrigerator according to claim 3.   The refrigerator according to claim 4, wherein the temporal change detection unit detects temporal changes in color and size of the food image. A food image identification unit that identifies the type and amount of food corresponding to the food image and obtains identification information of the type and amount of the food image,
The refrigerator according to claim 4 or 5, wherein the food deterioration state estimation unit reflects identification information of the type and amount of the food image in the estimation of the deterioration state of the food. A food information storage unit that stores in advance the food type in association with at least one of the shape, size, color, and exterior label of the food;
The food image identification unit refers to the information stored in the food information storage unit, identifies the type and amount of food corresponding to the food image, and acquires the identification information of the type and amount of the food image The refrigerator according to claim 6. The food deterioration state estimation unit, based on the temporal change of the food image detected by the time change detection unit, as the information on the deterioration, the food deterioration tendency indicating the speed of progress of the food deterioration Estimate
The refrigerator according to claim 6 or 7, wherein the control device further includes a time limit estimating unit that estimates a time limit related to the food based on the deterioration tendency of the food estimated by the food deterioration state estimating unit. . Regarding the food image managed by the control device, a time limit setting unit in which a user sets a setting time limit,
The refrigerator according to claim 8, further comprising: a set temperature calculation unit that calculates a set temperature in the storage room, wherein a time limit related to the food estimated by the time limit estimation unit may be after the set time limit. . The refrigerator according to claim 9, wherein the time limit setting unit is provided in the portable terminal. A compressor that compresses the refrigerant, a condenser that condenses the refrigerant discharged from the compressor, a throttle device that expands the refrigerant that has flowed out of the condenser, and a refrigerant that is expanded by the throttle device is supplied to the storage chamber. A refrigeration cycle circuit having a cooler for cooling the air to be
An air conveyance device for conveying the air cooled by the cooler to the storage chamber;
A blowout air volume control device for adjusting an inflow amount of air supplied to the storage room;
According to the set temperature calculated by the set temperature calculation unit, the cooling capacity for the storage chamber is changed by controlling at least one of the compressor, the air conveying device, and the blown air volume control device. The refrigerator according to claim 9 or 10, further comprising a cooling control unit. The control device includes, in addition to the food image, identification information of the type and amount of the food image identified by the food image identification unit, a time limit related to the food estimated by the time limit estimation unit, and the set temperature calculation The refrigerator according to any one of claims 9 to 11, wherein at least one of the set temperatures calculated by a unit is transmitted to the portable terminal via the communication device. The refrigerator according to any one of claims 1 to 12,
A portable terminal having a display unit and communicating with the refrigerator by wired or wireless communication via the communication device;
With
The mobile terminal displays at least one of the food image and information on the deterioration on the display unit. The refrigerator according to claim 2 or 3,
A portable terminal having a display unit and communicating with the refrigerator by wired or wireless communication via the communication device;
With
The network system, wherein the portable terminal causes the display unit to display at least one of the food image, the information related to the deterioration, and the warehouse entry / exit identification information. A refrigerator according to claim 3;
A portable terminal having a display unit and communicating with the refrigerator by wired or wireless communication via the communication device;
With
The mobile terminal causes the display unit to display at least one of the food image, information about the deterioration, the entry / exit identification information, and the time information. The refrigerator according to claim 6 or 7,
A portable terminal having a display unit and communicating with the refrigerator by wired or wireless communication via the communication device;
With
In the portable terminal, the display unit includes the food image, the information related to the deterioration, the entry / exit identification information, the time information, and the type and amount identification information of the food image identified by the food image identification unit. A network system characterized in that at least one of them is displayed. A refrigerator according to claim 8;
A portable terminal having a display unit and communicating with the refrigerator by wired or wireless communication via the communication device;
With
The portable terminal includes a time limit setting unit for a user to set a time limit for setting the food image managed by the control device,
The network system, wherein the refrigerator further includes a set temperature calculation unit that calculates a set temperature in the storage chamber in which a time limit related to the food estimated by the time limit estimation unit may be after the set time limit. The refrigerator according to any one of claims 1 to 12,
A portable terminal having a display unit and communicating with the refrigerator by wired or wireless communication via the communication device;
With
The portable terminal and the refrigerator are connected to an external server via the Internet,
Communication between the refrigerator and the mobile terminal is permitted when the mobile terminal is a mobile terminal authenticated by the external server.
A network system characterized by this. A refrigerator according to claim 6;
A portable terminal having a display unit and communicating with the refrigerator by wired or wireless communication via the communication device;
With
The portable terminal and the refrigerator are connected to an external server via the Internet,
Communication between the refrigerator and the mobile terminal is permitted when the mobile terminal is a mobile terminal authenticated by the external server,
The external server includes a food information storage unit that stores in advance, in association with at least one of the shape, size, color, and exterior label of the food, the type of food,
The food image identification unit refers to the information stored in the food information storage unit via the communication device, identifies the type and amount of food corresponding to the food image,
The network system, wherein the food deterioration state estimation unit estimates the food deterioration state by reflecting identification information of the type and amount of the food image identified by the food image identification unit. The external server includes an environmental database related to weather information in a specific area,
The control device includes a food storage environment estimation unit that obtains weather information of an area where the refrigerator is placed or a food production area based on the environment database, and estimates a food storage environment.
The network system according to claim 19, wherein the food deterioration state estimation unit estimates a food deterioration state reflecting the food storage environment estimated by the food storage environment estimation unit.