JP2017106671A - Refrigerator, image management system in refrigerator and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make the data capacity of a refrigerator interior image to be transmitted small by shortening a time until transmitting the refrigerator interior image to the outside in a refrigerator with a refrigerator interior photographing function.SOLUTION: A control device 1 detects an opened/closed state of a door 20 for closing a front opening part of a storage chamber 21 of a refrigerator 2, and specifies an area where a stored article is taken into or out of the storage chamber 21 on the basis of a change in a detection result of detection devices 4a to 4i. The control device 1 controls refrigerator interior imaging devices 5a to 5c so as to photograph the area where the stored article is taken in or out when detecting a closed state of the door 20, and stores refrigerator interior image information showing refrigerator interior images photographed by the refrigerator interior imaging devices in association with photographing position information showing the area where the stored article is taken in or out in a storage part. When controlling the refrigerator interior imaging devices 5a to 5c, the control device 1 replaces the refrigerator interior image information corresponding to the area where the stored article is taken in or out in the refrigerator interior information stored in the storage part with the refrigerator interior image information showing the refrigerator interior image photographed by the refrigerator interior imaging devices.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a refrigerator including an in-compartment imaging device that photographs the interior, an in-fridge image management system, and a program.

  2. Description of the Related Art In recent years, imaging devices are provided inside and outside a refrigerator, and those equipped with a function that allows users to view images inside and outside the refrigerator on an information terminal of a user have been put into practical use.

  In Patent Document 1, it is possible to confirm who used the image capturing apparatus for photographing outside the warehouse, and to determine whether there is any change in the contents stored in the warehouse using the image capturing apparatus for photographing inside the warehouse. The refrigerator which can transmit the image data which specified the change area to the outside of the warehouse is disclosed.

JP 2007-046833 A

  In the configuration of Patent Document 1, when it is determined that there is a change in the storage item in the warehouse, a difference image of the imaging data in the warehouse is calculated, and image data of the entire warehouse in which the change area is specified is transmitted to the outside. However, there is a problem that it takes time to transmit an image and the transmission data takes a long time because the transmission data is large.

  The present invention has been made in view of the above circumstances, and in a refrigerator with an in-compartment photographing function, the time required to transmit an in-compartment image to the outside is shortened, and the data capacity of the in-compartment image to be transmitted is reduced. The purpose is to do.

  In order to achieve the above object, a refrigerator according to the present invention includes a storage room, a door, an open / close determination unit, an in-compartment imaging device, an imaging control unit, a detection device, and a position specifying unit. The storage room stores storage items. The door closes the front opening of the storage chamber. The open / close determining unit detects the open / closed state of the door. The internal imaging device images the inside of the storage room. When the open / close determination unit detects the closed state of the door, the imaging control unit controls the internal imaging device to take an image of the inside of the storage room. The detection device detects a stored item inside the storage chamber. The position specifying unit specifies a region where the stored item is put in and out of the storage room based on a change in the detection result of the detection device while the open / close determining unit detects the open state of the door. The imaging control unit controls the in-compartment imaging device so as to take an image of an area where the storage item specified by the position specifying unit has been taken in and out.

  According to the present invention, in the refrigerator with an in-compartment photographing function, by specifying and photographing the region where the stored item changes, the difference image calculation of the entire interior after the photographing can be omitted, and the inside of the limited range of the interior Since only the image is transmitted, it is possible to shorten the time until the internal image is transmitted to the outside, and to reduce the data capacity of the internal image to be transmitted.

It is a figure which shows the structural example of the image management system in a refrigerator which concerns on Embodiment 1 of this invention. 6 is a diagram illustrating another arrangement example of the detection devices according to Embodiment 1. FIG. FIG. 3 is a diagram illustrating a functional configuration example of a control device according to the first embodiment. It is a figure which shows the function structural example of the warehouse image server which concerns on Embodiment 1. FIG. It is a figure which shows an example of the authentication table which the internal image server which concerns on Embodiment 1 hold | maintains. 4 is a flowchart illustrating an example of an operation of the control device according to the first embodiment. 6 is a flowchart illustrating an example of an operation of a control device according to a modification of the first embodiment. 4 is a flowchart illustrating an example of operations of the control device and the in-compartment image server according to the first embodiment. It is a flowchart which shows an example of operation | movement with the warehouse image server which concerns on Embodiment 1, and a user terminal. It is a figure which shows the structural example of the image management system in the refrigerator which concerns on Embodiment 2 of this invention. 6 is a diagram illustrating a functional configuration example of a control device according to Embodiment 2. FIG. 6 is a flowchart illustrating an example of the operation of the control device according to the second embodiment. It is a figure which shows the structural example of the image management system in the refrigerator which concerns on Embodiment 3 of this invention. FIG. 10 is a diagram illustrating a functional configuration example of a control device according to a third embodiment. 10 is a flowchart illustrating an example of the operation of the control device according to the third embodiment. It is a figure which shows an example of the hardware constitutions of each apparatus which concerns on embodiment of this invention.

  EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated in detail with reference to drawings. In addition, the same code | symbol is attached | subjected to the part which is the same or it corresponds in a figure.

(Embodiment 1)
FIG. 1 is a diagram illustrating a configuration example of an in-fridge image management system according to Embodiment 1 of the present invention. The in-refrigerator image management system 100 includes a control device 1, a refrigerator 2, a user terminal 7, and an in-compartment image server 8. The refrigerator 2 includes, for example, a compressor, an evaporator, a circulation fan, a temperature sensor, and the like (all not shown), and cools articles in the warehouse. The refrigerator 2 includes a storage chamber 21 having a double door 20, a drawer-type storage chamber 22, and a storage chamber 23. The door 20 that closes the front opening of the storage chamber 21 is not limited to the double door opening of the two doors, and may be one right opening door or left opening door. In FIG. 1, the left door 20 is not shown for convenience.

  The refrigerator 2 includes an opening / closing sensor 3 that detects opening / closing of the door 20. The refrigerator 2 includes in-compartment imaging devices 5a to 5c (hereinafter, collectively referred to as the in-compartment imaging device 5) that captures images in the interior. The refrigerator 2 includes detection devices 4a to 4i (hereinafter, collectively referred to as the detection device 4) that detect the presence or absence of stored items in the warehouse. The open / close sensor 3, the in-compartment imaging devices 5 a to 5 c, and the detection devices 4 a to 4 i are connected to the control device 1. The opening / closing sensor 3 is, for example, a contact output type sensor, embedded in the opening peripheral edge of the storage chamber 21, and detects the open state and the closed state of the door 20. The open / close sensor 3 transmits an open / close signal indicating the detected open / close state of the door 20 to the control device 1. The detection devices 4a to 4i are, for example, an infrared sensor or an optical sensor that detects an object. The detection devices 4a to 4i are arranged inside the storage chamber 21 and detect the presence / absence of storage items in the detection range depending on whether or not the irradiated light beam is blocked. Each of the detection devices 4 a to 4 i transmits a detection signal indicating the presence or absence of the detected stored item to the control device 1. Although the example which arrange | positions the detection apparatuses 4a-4i in the rear surface inside the storage chamber 21 is shown in FIG. 1, it is not restricted to this.

  FIG. 2 is a diagram illustrating another arrangement example of the detection devices according to the first embodiment. As shown in FIG. 2, the detection devices 4 j to 4 o may be arranged on the side surface inside the storage chamber 21. Moreover, you may arrange | position to both the rear surface and the side surface inside the storage chamber 21. FIG. Each of the detection devices 4j to 4o transmits a detection signal indicating the presence or absence of the detected stored item to the control device 1. In FIG. 2, the door 20 is not shown for convenience.

  The in-compartment imaging devices 5 a to 5 c in FIG. 1 are juxtaposed at a predetermined interval at the inner end of the right door 20 and shoot a predetermined imaging range in accordance with an instruction from the control device 1. Each of the in-compartment imaging devices 5 a to 5 c transmits the captured in-compartment image to the control device 1. In the example of FIG. 1, the in-compartment imaging device 5a can photograph the detection ranges of the detection devices 4a to 4c. The internal imaging device 5b can photograph the detection ranges of the detection devices 4d to 4f. The internal imaging device 5c can shoot the detection ranges of the detection devices 4g to 4i. In the example of FIG. 2, the in-compartment imaging device 5a can photograph the detection ranges of the detection devices 4j and 4k. The in-compartment imaging device 5b can photograph the detection ranges of the detection devices 4l and 4m. The internal imaging device 5c can shoot the detection ranges of the detection devices 4n and 4o. The number and position of the detection device 4 and the internal imaging device 5 are not limited to the number and position shown in FIGS. 1 and 2, and may be determined according to the number of shelves and the volume of the storage chamber 21.

  The control device 1 determines whether or not the detection signal from the detection device 4 has changed. The control device 1 specifies that the stored item has been taken in and out within the detection range of the detection device 4 in which the detection signal has changed. The control device 1 controls the in-compartment imaging devices 5a to 5c corresponding to the detection device 4 in which the detection signal has changed, and obtains an in-compartment image of the region in which the position of the stored item has changed. When there are a plurality of detection devices 4 in which the detection signal has changed, the internal image pickup devices 5 corresponding to the detection devices 4 are controlled to obtain a plurality of internal images. The control device 1 is connected to an external network 9 via the router 6, and transmits in-store image information indicating the in-store image of the refrigerator 2 to the in-store image server 8. The user terminal 7 is a terminal such as a smartphone, a tablet terminal, or a personal computer that is used by the user, and is a terminal that is connected to the network 9 and can browse the image inside the refrigerator 2. The in-compartment image server 8 stores the in-compartment image information of the refrigerator 2 and transmits the in-compartment image information to the user terminal 7 when there is a request from the user terminal 7. The internal image server 8 is an external server in the present invention.

  FIG. 3 is a diagram illustrating a functional configuration example of the control device according to the first embodiment. The control device 1 includes an open / close determination unit 11, a position specifying unit 12, an imaging control unit 13, an image acquisition unit 14, a storage unit 15, and a communication unit 16. The open / close determination unit 11 receives an open / close signal from the open / close sensor 3. The open / close determining unit 11 detects the timing at which the door 20 is opened and closed from the received open / close signal, and notifies the position specifying unit 12 and the imaging control unit 13 respectively. The position specifying unit 12 receives detection signals from the detection devices 4a to 4i. When the door 20 is in the open state, the position specifying unit 12 specifies that there is a stored item in the detection range of the detection device if there is a change in at least one of the detection signals of the detection devices 4a to 4i. . The position specifying unit 12 notifies the imaging control unit 13 of the detection device 4 whose detection signal has changed.

  When the door 20 is closed, the imaging control unit 13 outputs a shooting instruction signal for instructing shooting to the internal imaging device 5 corresponding to the detection device 4 whose detection signal has changed. The image acquisition unit 14 acquires the internal image captured by the internal imaging device 5 and stores it in the storage unit 15 as internal image information. The storage unit 15 stores shooting position information indicating at least one of information indicating the detection device 4 in which the detection signal has changed and information indicating the image pickup device 5 that has output the shooting instruction signal in association with the image information in the store. To do. When new shooting position information and in-store image information are stored in the storage unit 15, the communication unit 16 transmits an update notification signal notifying that the in-store image information has been updated to the in-store image server 8. When the communication unit 16 receives an update request signal requesting new image information from the image server 8, the communication unit 16 sends the new image information and shooting position information stored in the storage unit 15 to the image server 8. Send. The control device 1 may be built in the refrigerator 2.

  FIG. 4 is a diagram illustrating a functional configuration example of the internal image server according to the first embodiment. The internal image server 8 includes a communication control unit 81, an internal image information acquisition unit 82, a shooting position information acquisition unit 83, an image composition unit 84, a storage unit 85, a device number acquisition unit 86, and a browsing request signal acquisition unit 87. Is provided. The communication control unit 81 is connected to the control device 1 and the user terminal 7 via the network 9 and performs communication using respective communication protocols. When the communication control unit 81 extracts the update notification signal included in the communication data received from the control device 1, the device number acquisition unit 86 identifies at least one of the control device 1 and the refrigerator 2 included in the communication data. Is extracted, and it is verified whether or not the extracted device number is set in the authentication table 851 of the storage unit 85. When the device number included in the communication data is set in the authentication table 851, the device number acquisition unit 86 transmits an update request signal to the control device 1 via the communication control unit 81.

  The in-compartment image information acquisition unit 82 extracts the in-compartment image information included in the communication data received by the communication control unit 81 from the control device 1, and sends it to the image composition unit 84. The shooting position information acquisition unit 83 extracts shooting position information associated with the in-compartment image information included in the communication data received from the control device 1 by the communication control unit 81, and sends it to the image composition unit 84. Based on the shooting position information received from the shooting position information acquisition unit 83, the image composition unit 84 displays the entire image of the storage room 21 with the internal image indicated by the internal image information received from the internal image information acquisition unit 82. One overall image in the store is generated as described above, and stored in the store image area 852 of the storage unit 85 as the entire store image information. The entire image information in the store stored in the store image area 852 is associated with the device number set in the authentication table 851.

  When the entire image information in the store is already stored in the store image area 852 of the storage unit 85, the image composition unit 84 stores the entire store image information corresponding to the device number extracted by the device number acquisition unit 86 in the store. The internal image information received from the internal image information acquisition unit 82 indicates the internal image of the portion corresponding to the imaging position information received from the imaging position information acquisition unit 83 of the entire internal image read from the image area 852. A new entire image in the store is generated by replacing with the store image, and the entire store image information is overwritten in the store image area 852. Note that the entire image information in the store may be a single image synthesized so that the entire storage room 21 is displayed, or information in which the shooting position information and the store image information are associated with each other. Or both. When the entire image information in the store includes information in which the shooting position information and the image information in the store are associated with each other, the image composition unit 84 stores the entire store image information corresponding to the device number extracted by the device number acquisition unit 86. Is read from the internal image area 852, and the internal image information corresponding to the imaging position information received from the imaging position information acquisition unit 83 is replaced with the internal image information received from the internal image information acquisition unit 82.

  FIG. 5 is a diagram showing an example of an authentication table held by the in-store image server according to the first embodiment. As shown in FIG. 5, in the authentication table 851, a “control device ID” that is a device number that identifies the control device 1 and a “refrigerator ID” that is a device number that identifies the refrigerator 2 are set in advance. A “terminal ID” that is a terminal number for identifying the user terminal 7 that can view the image inside the refrigerator 2 is associated. In the example of FIG. 5, a device number “REFA” for identifying the refrigerator 2, a device number “CNTA” for identifying the control device 1 that transfers the in-store image information of the refrigerator 2 to the in-store image server 8, and the refrigerator The terminal number “TMEA0” that identifies the user terminal 7 that can view the two internal images is associated. Further, the user terminal 7 identified by the terminal number “TMEA1” is further associated with the equipment number “CNTA” and the equipment number “REFA”, and the refrigerator 2 identified by the equipment number “REFA” from the plurality of user terminals 7. The authentication table 851 is configured so that the image inside the warehouse can be browsed.

  Further, in the example of FIG. 5, in addition to the refrigerator ID “REFA” corresponding to the control device ID “CNTA”, the control device 1 with the device number “CNTB” stores the image information in the refrigerator 2 with the device number “REFB”. The configuration for transferring to the internal image server 8 is shown, and the device number “CNTB” is associated with the device number “REFB” and the terminal number “TMEB0”. In FIG. 5, the user terminal 7 that can browse the image inside the refrigerator 2 identified by the device number “REFA” and the user terminal 7 that can view the image inside the refrigerator 2 identified by the device number “REFB” are shown. Although the example which is another independent terminal is shown, an authentication table may be held so that the image in a store of a plurality of refrigerators 2 can be browsed from one user terminal 7. For example, if the user owns both the refrigerator 2 identified by the device number “REFA” and the refrigerator 2 identified by the device number “REFB”, the device number “CNTB” and the device number “REFB” The user terminal 7 identified by the terminal number “TMEA0” is associated with the user terminal 7 having the terminal number “TMEA0”, the refrigerator 2 identified by the equipment number “REFA” and the refrigerator identified by the equipment number “REFB”. The structure which can browse both of 2 store | warehouse | chamber interior images may be sufficient.

  As shown in FIG. 1, in the in-fridge image management system 100 in which the control device 1 and the refrigerator 2 are associated with each other one by one, either the control device 1 or the refrigerator 2 is identified. Only the device number to be stored may be stored in the authentication table 851. In this case, when the control device 1 communicates with the internal image server 8, only the device number of the control device 1 or the refrigerator 2 is notified to determine whether the device is set in the authentication table 851. At this time, a MAC (Media Access Control) address used by the communication unit 16 of the control device 1 for communication with the network 9 may be used as the device number. In addition, in the case where there is one controller 1, refrigerator 2, and user terminal 7 constituting the in-fridge image management system 100, the in-chamber image server 8 does not need to hold the authentication table 851, The number acquisition unit 86 may not be provided.

  The browsing request signal acquisition unit 87 of the image server 8 in the warehouse transmits a browsing request signal for requesting the entire chamber image information included in the communication data received from the user terminal 7 by the communication control unit 81 and a browsing request signal. The terminal number for identifying the user terminal 7 is extracted, and it is verified whether the extracted terminal number is set in the authentication table 851 of the storage unit 85. When the extracted terminal number is set in the authentication table 851, the browsing request signal acquisition unit 87 reads out the entire image information in the store corresponding to the terminal number from the store image area 852 of the storage unit 85, and performs communication control. The data is transmitted to the user terminal 7 via the unit 81. In the case where the entire image information in the store includes information in which the shooting position information and the store image information are associated with each other, the browsing request signal may request the store image information for each shooting position.

  FIG. 6 is a flowchart illustrating an example of the operation of the control device according to the first embodiment. The following processing is repeatedly executed while the control device 1 is activated. The open / close determination unit 11 of the control device 1 determines whether or not the door 20 of the storage chamber 21 has been opened based on the open / close signal received from the open / close sensor 3 (step S11). When the door 20 is not opened (step S11; NO), step S11 is repeated to wait for the door to open. When the door 20 is opened (step S11; YES), the position specifying unit 12 receives detection signals from the detection devices 4a to 4i (step S12), and determines whether or not there is a change in the detection signal (step S12). Step S13). If there is no change in any of the detection signals of the detection devices 4a to 4i (step S13; NO), the process proceeds to step S15. When there is a change in any of the detection signals of the detection devices 4a to 4i (step S13; YES), the detection device 4 in which the detection signal has changed is temporarily stored (step S14).

  The open / close determination unit 11 determines whether or not the door 20 is closed based on the open / close signal received from the open / close sensor 3 (step S15). When the door 20 is not closed (step S15; NO), it returns to step S12 and repeats step S12-step S15. When the door 20 is closed (step S15; YES), the position specifying unit 12 determines whether or not there is the detection device 4 whose detection signal has changed (step S16). When there is no detection device 4 in which the detection signal has changed (step S16; NO), the position specifying unit 12 determines that the stored item has not been taken in and out, and ends the process. When there is the detection device 4 in which the detection signal has changed (step S16; YES), the position specifying unit 12 specifies that the stored item has been put in and out in the detection range of the detection device 4 in which the detection signal has changed, and temporarily in step S14. The stored detection device 4 in which all the detection signals have changed is notified to the imaging control unit 13. The imaging control unit 13 identifies the in-compartment imaging device 5 corresponding to the detection device 4 in which all the detection signals have changed (step S17), and outputs a shooting instruction signal instructing shooting (step S18).

  For example, when there is a change only in the detection signal of the detection device 4a, the position specifying unit 12 specifies that the stored item has been put in and out of the detection range of the detection device 4a, and the imaging control unit 13 corresponds to the detection device 4a. An imaging instruction signal is output to the internal imaging device 5a. When any of the detection signals of the detection devices 4d to 4f has changed, the imaging control unit 13 also outputs a signal for instructing the imaging to the internal imaging device 5b corresponding to the detection devices 4d to 4f. Similarly, when any of the detection signals of the detection devices 4g to 4i has changed, the imaging control unit 13 also outputs a shooting instruction signal to the internal imaging device 5c corresponding to the detection devices 4g to 4i. The image acquisition unit 14 acquires the internal image captured by the internal imaging device 5 in accordance with the imaging instruction signal (step S19), and stores it in the storage unit 15 as internal image information. The storage unit 15 associates shooting position information indicating at least one of information indicating the detection device 4 in which the detection signal has changed and information indicating the image pickup device 5 that has output the shooting instruction signal with image information in the store. Store (step S20), and the process ends. Note that after step S19, it is determined whether or not the in-compartment image information has been acquired from all the in-compartment imaging devices 5 that output the imaging instruction signal, and the in-compartment from all the in-compartment imaging devices 5 that output the imaging instruction signal. Step S19 may be repeated until image information is acquired.

  In addition, when the control device 1 transmits the image inside the refrigerator 2 for the first time, when the entire image information inside the refrigerator 2 is not stored in the image area 852 inside the storage unit 85, the imaging control unit 13. Outputs the shooting instruction signal to all of the in-compartment imaging devices 5a to 5c, and transmits the image to the in-compartment image server 8 including the in-compartment image corresponding to the detection device 4 in which the detection signal has not changed. 84 may be configured to generate a single whole-chamber image.

  In the first embodiment, the internal imaging device 5 may include a swing mechanism, and the internal imaging device 5 can move inside the storage chamber 21 in at least one of the vertical direction and the horizontal direction. A moving mechanism may be provided. In the case where the internal imaging device 5 includes a swing mechanism, the imaging angles of the detection device 4 and the internal imaging device 5 are associated with each other. When the internal imaging device 5 includes a moving mechanism, the imaging positions of the detection device 4 and the internal imaging device 5 are associated with each other. This modification will be described with reference to FIG.

  FIG. 7 is a flowchart illustrating an example of the operation of the control device according to the modification of the first embodiment. The following processing is repeatedly executed while the control device 1 is activated. Steps S21 to S26 are the same as steps S11 to S16 in the flowchart of FIG. When there is a detection device 4 in which the detection signal has changed (step S26; YES), the imaging control unit 13 specifies the imaging angle (imaging position) of the in-compartment imaging device 5 corresponding to the detection device 4 in which the detection signal has changed. (Step S27), a shooting instruction signal for instructing shooting at the specified shooting angle (shooting position) is output to the internal imaging device 5 (Step S28).

  When the internal imaging device 5 includes a swing mechanism, the imaging instruction signal includes an instruction to control the internal imaging device 5 so as to face the detection range of the detection device 4 whose detection signal has changed. When the internal imaging device 5 includes a moving mechanism, the imaging instruction signal includes an instruction to move the internal imaging device 5 to a position where the detection range of the detection device 4 whose detection signal has changed can be imaged. Steps S29 and S30 are the same as steps S19 and S20 in the flowchart of FIG. Since the internal imaging device 5 includes the swing mechanism, the entire storage chamber 21 can be imaged even if the lens of the internal imaging device 5 is not wide-angle. When the in-compartment imaging device 5 includes the moving mechanism, the entire storage chamber 21 can be photographed with a small number (for example, one) of the in-compartment imaging device 5.

  FIG. 8 is a flowchart illustrating an example of operations of the control device and the in-store image server according to the first embodiment. The following processing starts when the control device 1 ends the processing in FIG. 6, and is repeatedly executed while the control device 1 and the in-store image server 8 are activated. When the in-compartment image information and the shooting position information are stored in association with each other in the storage unit 15 of the control device 1, the communication unit 16 transmits an update notification signal to the in-compartment image server 8 (step S41). When the communication control unit 81 extracts the update notification signal included in the communication data received from the control device 1 (step S51), the device number acquisition unit 86 of the internal image server 8 extracts the control device 1 and the refrigerator included in the communication data. 2 is extracted, and it is verified whether or not the extracted device number is set in the authentication table 851 of the storage unit 85 (step S52). If the device number included in the communication data is not set in the authentication table 851 (step S52; NO), the process ends. When the device number included in the communication data is set in the authentication table 851 (step S52; YES), the device number acquisition unit 86 transmits an update request signal to the control device 1 via the communication control unit 81 (step S52). S53).

  When the communication unit 16 of the control device 1 receives the update request signal from the internal image server 8 (step S42), the communication unit 16 transmits new internal image information and shooting position information in the storage unit 15 to the internal image server 8 ( Step S43) and the process is terminated. The internal image information acquisition unit 82 and the imaging position information acquisition unit 83 of the internal image server 8 respectively extract the internal image information and the imaging position information included in the communication data received from the control device 1 by the communication control unit 81. (Step 54), and sent to the image composition unit 84. The image composition unit 84 reads out the entire image information in the storage corresponding to the device number extracted by the device number acquisition unit 86 from the internal image area 852 (step S55), and uses the image capturing position information received from the image capturing position information acquisition unit 83 as the image capturing position information. The corresponding internal image information is replaced with the internal image information received from the internal image information acquisition unit 82 to generate new internal entire image information. The image synthesizing unit 84 overwrites the new internal image information on the internal image area 852 to update the internal image information (step S56), and ends the process. In the configuration in which the internal image server 8 does not hold the authentication table 851, step S52 may be omitted.

  FIG. 9 is a flowchart illustrating an example of operations of the in-compartment image server and the user terminal according to the first embodiment. The following processing is repeatedly executed while the user terminal 7 and the internal image server 8 are activated. The user terminal 7 transmits a browsing request signal to the internal image server 8 at a determined timing (step S71). The determined timing may be a timing when the user inputs a browsing request or a regular timing. The browsing request signal acquisition unit 87 of the internal image server 8 extracts the browsing request signal included in the communication data received from the user terminal 7 by the communication control unit 81 (step S61). A terminal number for identifying the transmitted user terminal 7 is extracted, and it is verified whether the extracted terminal number is set in the authentication table 851 of the storage unit 85 (step S62).

  If the extracted terminal number is not set in the authentication table 851 (step S62; NO), the process ends. When the extracted terminal number is set in the authentication table 851 (step S62; YES), the browsing request signal acquisition unit 87 stores the entire image information in the store corresponding to the terminal number in the store image area 852 of the storage unit 85. Is transmitted to the user terminal 7 via the communication control unit 81 (step S63), and the process is terminated. When the user terminal 7 receives the entire image information in the store from the store image server 8 (step S72), the user terminal 7 displays the entire store image information on the display screen (step S73), and ends the process. When the entire image information in the store is one entire store image, the user terminal 7 displays the entire store image on the display screen. When the entire image information in the store is information in which the shooting position information and the image information in the store are associated with each other, the user terminal 7 may generate one entire store image and display it on the display screen. The internal image for each shooting position may be displayed on the display screen. Note that in the configuration in which the image server 8 in the warehouse does not hold the authentication table 851, step S62 may be omitted.

  As described above, according to the refrigerator 2 of the first embodiment, the image capturing device 5 captures only the area specified as having been put in and out of the stored items, thereby calculating the difference image of the entire interior after the capturing. Can be omitted, and only the image information in the limited range is transmitted, so the time until the image information is transmitted can be shortened and the amount of data to be transmitted can be reduced. The required time and load of the image information transfer process to the image 8 can be reduced.

  This is particularly effective when the communication speed of the communication unit 16 of the control device 1 is low. As a result, it is possible to reduce the time from when the stored item is put in and out until the latest image in the warehouse can be viewed. Further, if the storage items are not taken in and out, neither the driving of the in-compartment image pickup device 5 nor the transfer processing of the in-compartment image information to the in-compartment image server 8 is performed, so that driving power and network communication resources are not wasted. .

(Embodiment 2)
FIG. 10 is a diagram illustrating a configuration example of an in-fridge image management system according to Embodiment 2 of the present invention. The in-refrigerator image management system 200 of the second embodiment includes a position detection imaging device 24 instead of the detection device 4 of the in-fridge image management system 100 of the first embodiment. The position detection imaging device 24 images the front opening of the storage chamber 21 at a predetermined time interval. The position detection imaging device 24 is connected to the control device 1, and transmits the position detection images continuously captured to the control device 1.

  Based on the position detection image received from the position detection imaging device 24 while the open / close determination unit 11 detects the open state of the door 20, the control device 1 identifies the region where the stored item has been put in and out. . In the second embodiment, the position detection imaging device 24 is a detection device according to the present invention. The control device 1 controls the in-compartment imaging device 5 that can photograph the area specified as having been put in and out of the stored item, and obtains an in-compartment image. When there are a plurality of areas specified as having been put in and out of the stored items, the in-compartment imaging device 5 corresponding to each area is controlled to obtain a plurality of in-compartment images. Other configurations are the same as those in the image management system 100 in the refrigerator according to the first embodiment.

  FIG. 11 is a diagram illustrating a functional configuration example of the control device according to the second embodiment. The position specifying unit 12 of the control device 1 according to the second embodiment transmits a shooting instruction signal to the position detection imaging device 24 and receives a position detection image from the position detection imaging device 24. The position specifying unit 12 analyzes the position detection image received from the position detection imaging device 24 and specifies a region where the stored item has been taken in and out. For example, the method for identifying the region in which the storage object has been put in and out is centered on an object (for example, a user's hand) crossing a plane formed by the edge of the front opening of the storage chamber 21 in the position detection image. The area in the determined range is identified as the area where the stored items have been put in and out. Alternatively, the user's hand region shown in the position detection image may be extracted, and the region where the stored items are taken in and out may be specified from the movement pattern of the user's hand. The pattern of hand movement is a pattern of the shape of the user's hand that changes before and after the stored item is taken in and out, and the movement that temporarily stops the movement of the hand when the stored item is taken in and out. Alternatively, the region of the user's hand may be excluded from the position detection image, and the region other than the region of the user's hand may be specified as the region where the stored item has been put in and out. Further, the position detection imaging device 24 may perform the specification of the region where the stored item has been taken in and out. In this case, the position specifying unit 12 receives information indicating the area where the stored item has been taken in and out from the position detection imaging device 24.

  The position specifying unit 12 sends an area signal indicating the area specified as having been put in and out of the stored items to the imaging control unit 13. When the door 20 is closed, the imaging control unit 13 outputs an imaging instruction signal for instructing imaging to the internal imaging device 5 corresponding to the notified area signal. In the case where the internal imaging device 5 includes a swing mechanism, the imaging instruction signal includes an instruction to control the internal imaging device 5 so as to face the area where the stored items are taken in and out. In the case where the internal imaging device 5 includes a moving mechanism, the imaging instruction signal includes an instruction to move the internal imaging device 5 to a position where the area where the stored items have been taken in / out can be imaged. The storage unit 15 uses at least one of the area signal indicating the area where the stored item has been taken in and out and the information indicating the in-compartment imaging device 5 that has output the imaging instruction signal as the in-compartment image information acquired by the image acquisition unit 14. The corresponding shooting position information is stored in association with the image information in the store. Other configurations are the same as those of the control device 1 of the first embodiment.

  The position detection imaging device 24 may photograph the front opening of the storage chamber 21 with a moving image. In this case, the position specifying unit 12 analyzes the position detection image of the moving image received from the position detection imaging device 24, and specifies the region where the stored item has been put in and out. The position detection imaging device 24 acquires an opening / closing signal indicating the opening / closing state of the door 20 from the opening / closing sensor 3 or the control device 1, and the front opening of the storage chamber 21 is determined only when the door 20 is in the opening state. You may shoot at time intervals.

  FIG. 12 is a flowchart illustrating an example of the operation of the control device according to the second embodiment. The following processing is repeatedly executed while the control device 1 is activated. The open / close determination unit 11 of the control device 1 determines whether or not the door 20 of the storage chamber 21 has been opened based on the open / close signal received from the open / close sensor 3 (step S81). When the door 20 is not opened (step S81; NO), step S81 is repeated to wait for the door to open. When the door 20 is opened (step S81; YES), the position specifying unit 12 transmits a shooting instruction signal to the position detection imaging device 24, and receives a position detection image from the position detection imaging device 24 (step S81). S82) Analyzing the received position detection image, it is determined whether or not there is an area in which the stored items have been put in and out (step S83). If there is no area where the stored items have been taken in and out (step S83; NO), the process proceeds to step S85. When there is an area where the stored item has been taken in and out (step S83; YES), the area where the stored item has been taken in and out is temporarily stored (step S84).

  The open / close determination unit 11 determines whether or not the door 20 is closed based on the open / close signal received from the open / close sensor 3 (step S85). If the door 20 is not closed (step S85; NO), the process returns to step S82, and steps S82 to S85 are repeated. When the door 20 is closed (step S85; YES), if there is no area where the stored items have been put in and out (step S86; YES), the position specifying unit 12 determines that the stored items have not been put in and out, and ends the processing. To do. When there is an area where the stored item has been taken in and out (step S86; NO), the position specifying unit 12 temporarily outputs the region signal for notifying the area in which all the stored item has been taken in and out, which is temporarily stored in step S84. 13 The imaging control unit 13 identifies the in-compartment imaging device 5 corresponding to all the received area signals (step S87), and outputs a shooting instruction signal for instructing shooting (step S88).

  The image acquisition unit 14 acquires the internal image captured by the internal imaging device 5 in accordance with the imaging instruction signal (step S89) and stores it in the storage unit 15 as internal image information. The storage unit 15 associates the image position information indicating at least one of the area signal indicating the area in which the stored item has been put in and out and the information indicating the image pickup apparatus 5 that outputs the image pickup instruction signal with the image information in the box. (Step S90), and the process ends. After step S89, it is determined whether or not the in-compartment image information has been acquired from all the in-compartment imaging devices 5 that output the imaging instruction signal, and the in-compartment from all the in-compartment imaging devices 5 that output the imaging instruction signal. Step S89 may be repeated until image information is acquired.

  As described above, according to the refrigerator 2 of the second embodiment, the image capturing device 5 captures only the area specified as having been put in and out, and calculates the difference image of the entire interior after the image capturing. Since only a limited range of in-store image information can be transmitted, the time until the in-store image information is transmitted can be shortened and the amount of data to be transmitted can be reduced. It is possible to reduce the time and load required for the process of transferring the image information in the warehouse.

  In general, articles newly stored in the storage chamber 21 are often arranged in front of the storage chamber 21. According to the refrigerator 2 of the second embodiment, the change in the stored item can be more accurately identified by specifying the region where the stored item has been put in and out using the position detection image obtained by photographing the front opening of the storage chamber 21. Can be detected.

(Embodiment 3)
FIG. 13 is a diagram illustrating a configuration example of an in-fridge image management system according to Embodiment 3 of the present invention. The in-fridge image management system 300 according to the third embodiment uses the in-compartment imaging device 5 instead of the position detection imaging device 24 of the in-fridge image management system 200 according to the second embodiment. That is, in Embodiment 3, the in-compartment imaging device 5 is also used as the in-compartment imaging device and the detection device in the present invention. The in-compartment imaging devices 5a to 5c take images of the storage chamber 21 in the opened state at a predetermined time interval in accordance with an instruction from the control device 1. The in-compartment imaging devices 5 a to 5 c transmit the in-compartment images captured continuously to the control device 1. The storage chamber 21 of the refrigerator 2 according to the third embodiment includes a storage area 210a, a storage area 210b, and a storage area 210c (hereinafter collectively referred to as a storage area 210) separated by shelves.

  The internal imaging device 5a images the storage area 210a, the internal imaging device 5b images the storage area 210b, and the internal imaging device 5c images the storage area 210c. The control device 1 specifies the storage area 210 where the stored items have been taken in and out based on the internal images received from the internal imaging devices 5a to 5c. The method of specifying the storage area 210 where the storage items have been taken in and out is obtained by obtaining a difference in the time direction of the images in the storage that have been continuously captured, and the imaging range of the internal imaging device 5 in which the mobile body includes the moving object. Is a storage area 210 in which the stored items have been taken in and out. Alternatively, an area of the user's hand that appears in the image inside the cabinet may be extracted, and the storage area 210 where the stored items are put in and out of the movement pattern of the user's hand may be specified. Alternatively, the area of the user's hand may be excluded from the image in the cabinet, and the storage area 210 including the area changed outside the user's hand area may be specified as the storage area 210 where the storage items are put in and out. Other configurations are the same as those in the refrigerator image management system 200 of the second embodiment.

  FIG. 14 is a diagram illustrating a functional configuration example of the control device according to the third embodiment. The opening / closing determination unit 11 of the control device 1 according to the third embodiment detects the timing at which the door 20 is opened and closed from the opening / closing signal received from the opening / closing sensor 3 and notifies the imaging control unit 13 of the detected timing. When the door 20 is opened, the imaging control unit 13 outputs an imaging instruction signal that causes the internal imaging device 5 to image the interior at a predetermined time interval. The position specifying unit 12 receives the in-store images continuously captured by the in-store image pickup device 5. The position specifying unit 12 analyzes the in-chamber images continuously captured by the in-chamber imaging device 5, and specifies the storage area 210 in which the storage items are put in and out. The position specifying unit 12 sends to the imaging control unit 13 an area signal indicating the storage area 210 that is specified as having been put in and out. Other configurations are the same as those of the control device 1 of the first embodiment.

  FIG. 15 is a flowchart illustrating an example of the operation of the control device according to the third embodiment. The following processing is repeatedly executed while the control device 1 is activated. The open / close determination unit 11 of the control device 1 determines whether or not the door 20 of the storage chamber 21 has been opened based on the open / close signal received from the open / close sensor 3 (step S91). When the door 20 is not opened (step S91; NO), step S91 is repeated to wait for the door to open. When the door 20 is opened (step S91; YES), the imaging control unit 13 outputs an imaging instruction signal that causes the internal imaging device 5 to image the interior at a predetermined time interval (step S92). The position specifying unit 12 acquires an internal image from the internal imaging device 5 (step S93), analyzes the received internal image, and determines whether or not there is a storage area 210 where storage items have been put in and out. (Step S94). If there is no storage area 210 in which the stored items have been taken in and out (step S94; NO), the process proceeds to step S96. When there is an area where the stored item has been taken in and out (step S94; YES), the storage area 210 where the stored item has been taken in and out is temporarily stored (step S95).

  The open / close determination unit 11 determines whether or not the door 20 is closed based on the open / close signal received from the open / close sensor 3 (step S96). If the door 20 is not closed (step S96; NO), the process returns to step S93, and steps S93 to S96 are repeated. When the door 20 is closed (step S96; YES), if there is no storage area 210 where the stored items have been put in and out (step S97; YES), the position specifying unit 12 determines that the stored items have not been put in and out, and performs processing. Exit. When there is a storage area 210 in which stored items have been put in and out (step S97; NO), the position specifying unit 12 outputs a region signal indicating the storage area 210 in which all stored items have been put in and out temporarily stored in step S95. The image is sent to the imaging control unit 13. The imaging control unit 13 specifies the in-compartment imaging device 5 corresponding to all the received area signals (step S98), and outputs a shooting instruction signal for instructing shooting (step S99).

  The image acquisition unit 14 acquires the internal image captured by the internal imaging device 5 in accordance with the imaging instruction signal (step S100), and stores it in the storage unit 15 as internal image information. The storage unit 15 associates the image position information indicating at least one of the area signal indicating the area in which the stored item has been put in and out and the information indicating the image pickup apparatus 5 that outputs the image pickup instruction signal with the image information in the box. (Step S101), and the process ends. After step S100, it is determined whether or not the in-compartment image information has been acquired from all the in-compartment imaging devices 5 that have output the photographing instruction signal, and the in-compartment from all the in-compartment imaging devices 5 that have output the photographing instruction signal. Step S100 may be repeated until image information is acquired.

  As described above, according to the refrigerator 2 of the third embodiment, only the area in which the in-chamber imaging device 5 of the refrigerator 2 is specified as having been put in and out of the stored items, and the difference between the entire inside after the photographing is taken. Since image calculation can be omitted and only a limited range of in-store image information is transmitted, the time to transmit the in-store image information can be shortened and the amount of data to be transmitted can be reduced. It is possible to reduce the time and load required for the process of transferring the in-store image information to the image server 8.

  In addition, the storage area 210 where the storage items are taken in and out is specified using the internal images captured by the internal imaging devices 5a to 5c that capture the interior of the storage room 21, so that the storage items are detected. The present invention can be applied without providing a position detection imaging device 24 different from the detection device 4 represented by an infrared sensor, an optical sensor, or the like, or the internal imaging device 5, and the cost can be reduced.

  In Embodiments 1 to 3 described above, the example in which the internal image is captured using the multiple internal imaging devices 5a to 5c has been described. However, the present invention is not limited to this. For example, the refrigerator 2 is provided with the internal imaging device 5 that can image the interior of the storage room 21 with a single unit instead of the internal imaging devices 5a to 5c, and the stored items are taken in and out of the acquired internal images. The storage unit 15 may store in-store image information indicating a portion corresponding to the region and shooting position information indicating a region where the stored item has been taken in and out. Thereby, since the area | region of the warehouse image transmitted to the warehouse image server 8 can be limited, even if the warehouse imaging device 5 is one, time to transmit warehouse image information is shortened, and transmission The amount of data to be reduced can be reduced.

  FIG. 16 is a diagram illustrating an example of a hardware configuration of each device according to the embodiment of the present invention. The control device 1, the user terminal 7, and the in-compartment image server 8 constituting the in-refrigerator image management systems 100, 200, and 300 each include a processor 101, a memory 102, and an interface 103 as hardware configurations. Each function of these devices is realized by the processor 101 executing a program stored in the memory 102. The interface 103 is for connecting apparatuses and establishing communication, and may be configured by a plurality of types of interfaces 103 as necessary. FIG. 16 illustrates an example in which the processor 101 and the memory 102 are each configured as one, but a plurality of processors 101 and a plurality of memories 102 may execute each function in cooperation.

  In the above embodiment, the in-compartment image server 8 is a single component. For example, the in-compartment image server 8 includes a server that authenticates at least one of the control device 1 and the refrigerator 2, and a user terminal 7. The authentication server and the server that stores and manages the internal image data may be realized by independent servers, or may be realized by a cloud server.

  In addition, when the above-described function is realized by sharing an OS (Operating System) and an application, or by cooperation between the OS and the application, only a part other than the OS may be stored in the medium.

  Furthermore, each program can be superimposed on a carrier wave and distributed via a communication network. For example, the program may be posted on a bulletin board (BBS, Bulletin Board System) on a communication network, and the program may be distributed via the network. Then, the above-described processing may be executed by starting these programs and executing them in the same manner as other application programs under the control of the operating system.

  DESCRIPTION OF SYMBOLS 1 Control apparatus, 2 Refrigerator, 3 Open / close sensor, 4, 4a-4o detection apparatus, 5, 5a-5c Image pick-up device in a warehouse, 6 Router, 7 User terminal, 8 Image server in a warehouse, 9 Network, 11 Open / close determination part, DESCRIPTION OF SYMBOLS 12 Position identification part, 13 Imaging control part, 14 Image acquisition part, 15 Storage part, 16 Communication part, 20 Door, 21, 22, 23 Storage room, 24 Position detection imaging device, 81 Communication control part, 82 Interior image Information acquisition unit, 83 Shooting position information acquisition unit, 84 Image composition unit, 85 Storage unit, 86 Device number acquisition unit, 87 Browsing request signal acquisition unit, 100, 200, 300 Image management system in refrigerator, 101 processor, 102 memory, 103 communication interface, 210, 210a to 210c storage area, 851 authentication table, 852 internal image area.

Claims (9)

A storage room for storing things;
A door that closes the front opening of the storage chamber;
An open / close determining unit for detecting an open / closed state of the door;
An internal image pickup device for photographing the inside of the storage room;
When the open / close determination unit detects the closed state of the door, an imaging control unit that controls the internal imaging device to image the inside of the storage chamber;
A detection device for detecting a stored item in the storage chamber;
Based on the change in the detection result of the detection device while the open / close determination unit is detecting the open state of the door, a position specifying unit that specifies an area in which storage items are put in and out of the storage chamber;
With
The said imaging control part is a refrigerator which controls the said imaging device in a warehouse so that the area | region in which the stored item of the said storage room which the said position specific | specification part specified was taken in / out was imaged. The detection device is a sensor that detects objects disposed at a plurality of locations inside the storage chamber,
The said position specific | specification part specifies the detection range of the said sensor from which the detection result of the object changed among the said some sensors is an area | region where the removal / insertion of the storage thing of the said storage chamber was carried out. refrigerator. The detection device is a position detection imaging device that images the front opening of the storage chamber at a predetermined time interval,
The position specifying unit includes a front opening of the storage chamber that is captured in the position detection image continuously captured by the position detection imaging device while the open / close determination unit detects the open state of the door. The refrigerator according to claim 1, wherein a region in a predetermined range centering on an object that crosses a plane formed by an edge is specified as a region in which storage items are put in and out. The internal imaging device includes a moving mechanism capable of moving in the vertical direction and / or the horizontal direction inside the storage chamber,
4. The image capturing control unit according to claim 1, wherein the image capturing control unit is configured to move the image capturing apparatus to a position where an image can be captured of an area where the storage items specified by the position specifying unit have been taken in and out. The refrigerator according to item 1. The detection device and the in-compartment imaging device are composed of a plurality of cameras arranged in parallel at a predetermined interval inside the storage chamber,
The imaging control unit controls the plurality of cameras to capture images at a predetermined time interval while the open / close determination unit detects the open state of the door,
The position specifying unit obtains a difference in time direction of the images in the room captured by each of the plurality of cameras continuously, and the shooting range of the camera in which a moving object is included in the image in the box The refrigerator according to claim 1, wherein the refrigerator is specified as an area where a stored item has been taken in and out. Internal image information indicating an internal image captured by the internal imaging device, and imaging position information indicating an area in which the storage items of the storage room specified by the position specifying unit have been taken in and out are stored in an external device. The communication part to send,
The refrigerator according to any one of claims 1 to 5. An in-refrigerator image management system comprising: the refrigerator according to claim 6; and an external server connected to the refrigerator via a communication network.
The external server is
An in-store image information acquisition unit that acquires in-store image information indicating an in-store image captured by the in-store imaging device from the refrigerator;
A shooting position information acquisition unit for acquiring shooting position information indicating an area in which the stored items in and out of the storage room have been put in and out of the refrigerator;
A storage unit that stores the image information in the store in association with the shooting position information;
Of the image information stored in the storage unit, the image information acquisition unit acquires the image information corresponding to the image position information acquired by the image acquisition position information acquisition unit. An image composition unit to be replaced with image information;
An in-fridge image management system. The external server is
Browsing the user terminal connected via the communication network to obtain a browsing request signal requested by the in-store image information and transmitting the in-store image information of the refrigerator corresponding to the user terminal to the user terminal Request signal acquisition unit,
The image management system in a refrigerator of Claim 7 provided with. Computer
An open / close determining unit for detecting an open / closed state of a door that closes a front opening of a storage room for storing items stored in the refrigerator;
A position specifying unit for specifying a region where the stored items in and out of the storage chamber are based on a change in a detection result of a detection device that detects the stored items inside the storage chamber;
When the open / close determining unit detects the closed state of the door, an imaging control unit that controls the in-compartment imaging device so as to capture an area in which the stored items in the storage room specified by the position specifying unit are taken in and out,
A storage unit that stores in-store image information indicating an in-store image captured by the in-store image capturing apparatus in association with the capture position information indicating an area in which the stored items in and out of the storage room are put in and out, and
When the imaging control unit controls the internal imaging device, the storage unit stores the internal image information stored in the storage unit, which corresponds to an area in which the storage items specified by the position specifying unit have been put in and out. An image compositing unit that replaces the in-compartment image information with the in-compartment image information indicating the in-compartment image captured by the in-compartment imaging device;
Program to function as.

Images