JP7211700B2 - refrigerator, display system - Google Patents

Description

Embodiments of the present invention relate to refrigerators and display systems .

2. Description of the Related Art In recent years, it has been proposed to provide an imaging unit such as a camera in a storage compartment in order to grasp food stored in the refrigerator from the outside (see, for example, Patent Document 1).

JP 2007-46833 A

However, if an attempt was made to image the entire range in the storage room in order to grasp the stored articles, a large number of cameras would be required, which was not realistic. In addition, when the position of the camera is fixed, it is difficult to capture an image of the entire inside of the storage room because a blind spot is created due to obstruction by stored articles and the like.
Therefore, a refrigerator and a display system are provided that allow the position of articles in a storage room to be grasped with a simple configuration.

A refrigerator according to an embodiment includes a housing provided with one or more storage compartments, an imaging unit provided on the upper side of the housing for imaging a range in front of the storage compartment, and opening and closing of the door of the storage compartment. The open/close detection unit detects the opening and closing of the door, and the imaging unit starts capturing images from the point at which the open/close detection unit detects that the door is open. and a control unit that performs a process of estimating the location where the article is stored based on the obtained position information.

The figure which shows typically the state which looked at the refrigerator of embodiment from the front side. The figure which shows typically the state which looked at the refrigerator from the side side Diagram showing the electrical configuration of a refrigerator A diagram schematically showing an imaging area set in front of the refrigerator. A diagram schematically showing a mode of capturing an image of a stored object, estimating its location, and generating an image. A diagram schematically showing an image of a stored item and an example of a food material DB A diagram schematically showing an example of a container DB and a symbol DB Schematic diagram showing another aspect of image capturing of stored objects, location estimation, and image generation. A diagram schematically showing another display mode of an image A diagram schematically showing a display mode of an image when a stored item is taken out.

Hereinafter, embodiments will be described with reference to the drawings.
As shown in FIG. 1, the refrigerator 1 of the present embodiment has a housing 2 which includes a refrigerator compartment 3, a vegetable compartment 4, an ice making compartment 5 and a small freezer compartment 6 arranged side by side, and a large freezer compartment 7 in this order from the top. It has multiple storage rooms. Among them, the small freezer compartment 6 functions as a storage compartment for quick freezing, for example. The configuration of this refrigerator 1 is an example, and the number, types, and arrangement of storage compartments are not limited to those shown in FIG.

The refrigerator compartment 3 is opened and closed by a double door 3a and a right door 3b arranged side by side. In addition, the structure which opens and closes the refrigerator compartment 3 by the door of a single opening type may be sufficient. The vegetable compartment 4, ice making compartment 5, small freezing compartment 6 and large freezing compartment 7 are opened and closed by drawer type vegetable compartment door 4a, ice making compartment door 5a, small freezing compartment door 6a and large freezing compartment door 7a, respectively. An operation panel 8 is provided on the left door 3a of the refrigerator compartment 3. As shown in FIG.

The operation panel 8 includes an operation unit composed of a plurality of illuminated electrostatic switches, and an illuminated display area for informing the operating state of the refrigerator 1 and the like. The operation panel 8 allows the user to set the operation state of the refrigerator 1 such as temperature adjustment in the storage compartment, quick freezing operation, ice making operation, etc. by touching the operation unit, or to set the operation state of the refrigerator 1 by lighting the display area. Notify. Although not shown, the operation panel 8 is also provided with a notification unit such as a buzzer, a speaker, or the like for notifying by an operation sound, warning sound, or voice.

Two cameras 9 are provided on the top of the housing 2 in this embodiment. This camera 9 corresponds to an imaging unit. In this embodiment, the camera 9 is a so-called CCD camera or CMOS camera, and is configured to be capable of capturing color moving images and still images.

These cameras 9 are provided at symmetrical positions across the center of the refrigerator 1 so that their fields of view (α) in the horizontal direction partially overlap. At this time, the field of view (α) of each camera 9 is set so that it is possible to image the entire area of the refrigerator 1 in the horizontal direction, except for the upper end of the refrigerator 1 .

Further, the camera 9 is provided so that the center of each field of view (β) in the front-rear direction protrudes from the front end of the housing 2, as shown in FIG. At this time, the field of view (β) of each camera 9 includes the front of the refrigerator 1 and is set so as to be able to image a part of the inside of the storage compartment.
In this way, the camera 9 is provided at a position where it is possible to capture an image of the range from the front end of the refrigerator 1 to the front side from the top of the housing 2 of the refrigerator 1 .

Further, each storage room is provided with a container or the like for storing articles. For example, the refrigerator compartment 3 is provided with a plurality of shelf boards 10, a door pocket inside the door (see FIG. 4), and the like. The vegetable compartment 4 is provided with a large storage container 11a and a small storage container 11b. A storage container 12 is provided in the small freezer compartment 6 . The large freezer compartment 7 is provided with a large storage container 13a and a small storage container 13b.

At this time, when the vegetable compartment door 4a of the vegetable compartment 4 is pulled out, the inside of the large storage container 11a and the small storage container 11b can be imaged by the camera 9 provided above. The same applies to the small freezer compartment 6 and the large freezer compartment 7 .

Further, the refrigerator 1 has a display device 14 fixedly or detachably provided on the right door 3b. The display device 14 includes a display 14a, an external storage section 14b, and the like. Although the display device 14 can be provided with a display device specially designed for the refrigerator 1, it can also be configured with a touch panel type so-called tablet PC, a smart phone owned by the user, or the like. The display device 14 corresponds to a display section.

Such a refrigerator 1 is controlled by a control section 15 as shown in FIG. The control unit 15 is composed of a microcomputer having a CPU, a ROM, a RAM, etc. (not shown), and controls the entire refrigerator 1 by executing a program stored in the ROM and the like. The control unit 15 also includes a storage unit 15a configured by the above-described ROM, RAM, or other non-volatile memory.

The control unit 15 is connected to the operation panel 8, the camera 9, the communication device 16, the door opening sensors 17a to 17e, the interior lights 18a to 18c, the fan devices 19a to 19b, the compressor 20, the temperature sensors 21a to 21b, and the like. , the power is supplied through the filter substrate 22 .

The communication device 16 adopts a wireless communication system such as Wifi in this embodiment, and various data communication is performed with a portable terminal 24 owned by a user, an external server 25, etc. via an antenna 16a and a network 23. connected as possible. Also, the mobile terminal 24 and the server 25 each have a storage unit, an operation unit, a display, etc., although not shown. The portable terminal 24 and server 25 correspond to the display section.

Door open sensors 17a to 17d detect the opening and closing of the doors of the storage compartments. Specifically, the open door sensor 17a detects opening and closing of the left door 3a of the refrigerator compartment 3, the door open sensor 17b detects opening and closing of the right door 3b of the refrigerator compartment 3, and the door open sensor 17c detects opening and closing of the vegetable compartment 4. The door opening sensor 17d detects opening and closing of the small freezer compartment door 6a of the small freezer compartment 6, and the door open sensor 17e detects opening and closing of the large freezer door 7a of the large freezer compartment 7. . A door opening sensor 17 for detecting opening/closing of the ice making chamber door 5a of the ice making chamber 5 may be provided.

Interior lighting 18 illuminates the interior of the storage compartment. Specifically, the internal lighting 18 a illuminates the inside of the refrigerator compartment 3 , the internal lighting 18 a illuminates the inside of the vegetable compartment 4 , and the internal lighting 18 a illuminates the inside of the large freezer compartment 7 . Note that the number and arrangement of the interior lights 18 are merely examples, and are not limited to these.

The fan device 19 circulates cold air generated in the refrigerating cycle to each storage compartment. Specifically, the fan device 19a circulates cold air in the refrigerator compartment 3 and the vegetable compartment 4, which are storage compartments in the refrigerating temperature range. Cold air is circulated to the small freezer compartment 6 and the large freezer compartment 7. - 特許庁
The compressor 20 constitutes a refrigeration cycle together with a cooler (not shown) and the like, and generates cool air when it is in an operating state.

A temperature sensor 21 detects the temperature in the storage chamber. Specifically, the temperature sensor 21a detects the temperature of the refrigerator compartment 3 and the vegetable compartment 4 by detecting the temperature of cold air circulating in the storage compartment in the refrigerator temperature range. The temperature sensor 21b detects the temperatures of the ice making compartment 5, the small freezer compartment 6 and the large freezer compartment 7 by detecting the temperature of cold air circulating in the storage compartments in the freezing temperature range. In addition, it is also possible to adopt a configuration in which the temperature sensor 21 is individually provided in each storage chamber.

Next, the operation of the configuration described above will be described.
As described above, when an imaging unit is provided in the storage chamber in order to grasp the food stored in the refrigerator 1 from the outside, a large number of units are required to image the entire range in the storage chamber. The fixed position can create blind spots that make it difficult to image everything in the storage room.

In addition, in many conventional methods, the main purpose is to identify the type of food. In some cases, it is difficult to say that it is practical because the load of image processing to perform the processing increases, the database becomes complicated, and so on.

Therefore, in this embodiment, the position and type of articles in the storage room can be grasped with a simple configuration as follows. Although the following processing is mainly executed by the control unit 15, the refrigerator 1 will be mainly described for simplification of explanation.

As shown in FIG. 4, in front of the refrigerator 1, an imaging area (R) is set, which is an area where articles and the like are imaged by the camera 9. As shown in FIG. In the present embodiment, the imaging area (R) is set in a substantially rectangular parallelepiped shape obtained by virtually enlarging the outer shape of the refrigerator 1 toward the front side. This imaging region (R) is set as a virtual coordinate system with a predetermined origin as a reference. The origin can be arbitrarily set, for example, at the center of the front end of the refrigerator 1 in the vertical and horizontal directions.

The camera 9 captures an image of an object within the imaging region (R) that includes an article that comes into its field of view. Specifically, the camera 9 captures a continuous image such as a moving image or intermittent images such as a plurality of still images captured at predetermined time intervals in the imaging region (R). is imaged. Hereinafter, a period during which the camera 9 continues imaging is referred to as an imaging period.

Since articles are normally taken in and out of the refrigerator 1 from the front side of the refrigerator 1, a camera 9 is arranged on the upper part of the housing 2 to continuously or intermittently cover the imaging area (R) over the imaging period. Positional information of the article when it is put into and taken out of the refrigerator 1, that is, the storage room, is obtained by taking an image of the article. Here, the position information is information such as the position, moving speed, moving direction, and moving trajectory of the article. Note that the position information includes at least one or more of these pieces of information.

In the present embodiment, this position information can be obtained from the coordinates of the article in the coordinate system of the imaging region (R), the amount of temporal change in the coordinates, the direction of change, and the like. In addition, since two cameras 9 are provided in this embodiment, it is possible to three-dimensionally image an article in a stereogram, and it is possible to obtain more accurate position information when the article is taken in and out of the storage room. .

Here, as shown in plan view in FIG. 5, it is assumed that apples (hereinafter referred to as "stored goods M1") are stored on the shelf board 10 of the refrigerator compartment 3 as goods. At this time, after the door of the refrigerating chamber 3 is opened, the stored items M1 are placed on the shelf board 10 from the outside of the refrigerating chamber 3 while being held by a person's hand, for example. In other words, when an article is to be put into or taken out of the storage room, such as when the article M1 is to be stored, one of the doors is first opened.

Therefore, when the open door sensor 17 detects the opening of one of the doors, for example, the right door 3b of the refrigerator compartment 3, the refrigerator 1 starts imaging with the camera 9 from the time when the opening of the door is detected. . The imaging by the camera 9 is continued until the closing of the door is detected, or until a predetermined standby period elapses after the closing of the door is detected.

Based on the images captured by the two cameras 9, the refrigerator 1 obtains positional information such as the movement trajectory K1 and the movement speed of the storage item M1. At this time, as the positional information of the stored items M1, not only changes in the horizontal direction of the refrigerator 1 but also changes in the height direction of the refrigerator 1 are required. Although detailed description is omitted, the position information can be obtained by executing well-known image processing such as extracting a moving object in an image and obtaining changes in the coordinates of the moving object with time.

When the refrigerator 1 obtains the position information such as the movement trajectory K1 of the article M1, the refrigerator 1 performs a process of estimating the location in the storage room where the article M1 is stored. Specifically, refrigerator 1 obtains to which position in the horizontal direction and height direction of refrigerator 1 storage item M1 has moved based on the position information. Here, it is assumed that the goods M1 are stored on the second shelf plate 10 from the top in the refrigerator compartment 3 .

When the storage item M1 is out of the field of view of the camera 9, the refrigerator 1 determines to which position in the depth direction of the refrigerator 1 the storage item M1 has moved based on the elapsed time from the time when the storage item M1 is out of the field of view. Perform the process of estimating This is because, for example, the rear end side of the shelf board 10 is not within the field of view (β) of the camera 9 as shown in FIG.

Specifically, when the storage item M1 moves out of the field of view (β) in the depth direction, the refrigerator 1 determines the moving speed when the storage item M1 moves in the depth direction and By measuring the time it takes for the hand to return to the field of view (β) again, it is possible to estimate how far the stored object M1 has moved in the depth direction from the field of view (β).

Refrigerator 1 then estimates the location where article M1 is placed, as shown in FIG. In the case of FIG. 5, the location where the stored material M1 is presumed to have been stored is indicated as a contoured area X1 indicating the degree of possibility of storage. This region X1 is centered on the most probable stored position, and the further from the center, the lower the probabilities. In FIG. 5, the degree of possibility is schematically indicated by hatching shades, and the darkest hatching indicates the highest possibility.

Subsequently, the refrigerator 1 performs a process of generating a display image G1 in which the image of the storage item M1 and the image of the interior of the storage room prepared in advance are superimposed. Here, the image of the interior of the storage chamber is previously converted into data by using an image of the actual interior of the storage chamber, a modeled CG image, or the like. In this case, an image of a state in which the material M1 is stored on the far right side of the shelf board 10 is generated.

On the other hand, the refrigerator 1 acquires the image of the storage item M1 as follows. Refrigerator 1, as shown in FIG. 6, extracts an object image showing storage item M1 from the image acquired when position information is obtained. Note that FIG. 6 schematically shows a state in which only the apple is extracted by deleting the hand from the image in which the apple is grasped by the hand. Refrigerator 1 then compares the extracted object image with a food database (hereinafter referred to as a food DB) pre-stored in storage unit 15a, thereby estimating the type of storage item M1. This foodstuff DB is stored in the storage unit 15a or the like together with a container DB and a symbol DB, which will be described later, as data of a food database (hereinafter referred to as a food DB) in which foods to be stored in the refrigerator 1 are registered.

The foodstuff DB stores food image data that enables identification of foodstuffs such as shapes and colors of various foodstuffs. For example, the foodstuff DB registers apples as food A, pears as food B, tomatoes as food C, pears as food D, and the like. These food items A to D correspond to food items that have been converted into data in advance. In addition, the foodstuff DB shown here is an example, and is not limited to these.

This food image data can also be registered as 3D data. In addition, the food material DB, container DB, and symbol DB, which will be described later, can be configured to be stored in the storage unit 15 a, or can be configured to be stored in the mobile terminal 24 or the external server 25 . In addition, the food material DB may be configured such that main fruits, vegetables, etc. are registered in advance, or may be configured to allow new registration, correction, or addition by the user.

Refrigerator 1 selects food that can be probabilistically assumed to be the same as storage M1 from these foods, and displays display image G1 (see FIG. 5) in which the image of the selected food is superimposed on the image in the storage compartment. ). Here, probabilistic inference means that if characteristics such as “red” and “round” can be extracted as a custom-made stock item M1, it can be inferred that the apple or tomato is the same item, but the “size” It means an aspect in which it is highly likely that apples are assumed to be apples.

When attempting to identify foods reliably as in the conventional art, the image processing and food database become complicated or bloated, and the user is burdened with extra operations when storing food so that it can be photographed in detail by the camera 9. This is to open the door and make the configuration simple. That is, by adopting a configuration in which foods that can be probabilistically presumed to be the same food are selected, the image processing and the food database are not complicated or enlarged, and the user is not overburdened. can be identified.

In addition, it is considered that the user who stored the storage item M1 knows what kind of food is stored to some extent. Therefore, for example, if it is possible to indicate that a red round food item is stored on the second shelf board 10 from the top, the user will be able to remember what the food item is. It is considered that the user is not so disadvantaged even if the configuration is such that the food that can be reasonably guessed is selected.

The generated display image G<b>1 can be displayed on the display device 14 provided in the refrigerator 1 , but can also be displayed on the mobile terminal 24 or the external server 25 . At this time, the refrigerator 1 causes the display device 14 to display the display image G1 when the display of the image of the inside of the storage room is input from the operation panel 8, for example.

Alternatively, the refrigerator 1 can display the display image G1 or information capable of displaying the display image G1 when the user requests the display of the image inside the storage room from the mobile terminal 24 or the external server 25 in order to check the image inside the storage room. can be displayed on the mobile terminal 24 or the external server 25 by transmitting the .

The information that enables the display image G1 to be displayed is, for example, information that can specify that apples are stored on the right rear side of the shelf board 10 on the second stage from the top. The portable terminal 24 or the external server 25 that receives the information generates an image similar to the display image G1 by itself and displays it on its own display unit. Therefore, the refrigerator 1 does not store the display image G1 itself, but stores information on the type of the article M1 and the storage location, and may superimpose the image as necessary.

By the way, although the example in which the food itself is stored has been shown so far, as a general usage form of the refrigerator 1, it is assumed that the food is stored in a container and the container is stored. In that case, it is assumed that there are a wide variety of types of containers, and even the same container is used differently depending on the household.

Therefore, as shown in FIG. 7, the refrigerator 1 has a container database (hereinafter referred to as a container DB) that associates the shape and color of the container with the food contained in the container, and a symbol attached to the container. It can be configured to store a symbol database (hereinafter referred to as a symbol DB) in which characters and the like are associated with foods.

For example, in the container DB, a square container for storing rice as container A, a square container with a yellow lid for storing curry as container B, and a container with a round red lid for storing fukujinzuke as container C are registered. In addition, in FIG. 7, the difference in the color of the lid is schematically shown by the difference in hatching.
Alternatively, symbol DB contains a month mark indicating that lightly pickled vegetables are stored as symbol A,
A star mark indicating that pickled plums are stored as symbol B, a triangular mark indicating that scallops are stored as symbol C, etc. are registered.

The identification information registered in the container DB and the symbol DB can be configured such that food items are registered in advance in a blank state, or can be newly registered, modified, or added by the user. You can also In addition, in the case of a storage object that matches both the container DB and the symbol DB, it is possible to set a priority order when judging the same object, such as prioritizing matching with the symbol DB.

Now, the refrigerator 1 creates a database and stores the storage state of the food such as the type of the food and the storage location identified each time the food is accommodated, generates an image to be displayed, updates an already generated image, or , update information for displaying images, etc. For example, as shown in FIG. 8, it is assumed that a new storage item M2 is stored in a state where an article, for example, a storage item M1 is already stored on the right side of the shelf board 10 . Here, canned juice is assumed as the storage item M2.

Now, suppose that it is specified based on the image captured by the camera 9 that the movement trajectory K2 of the storage item M2 is directed to the right side of the shelf board 10 . In this case, the refrigerator 1 sets a new area X2 on the right side of the shelf board 10 based on the positional information of the item M2 so as not to overlap the area X1 of the item M1. Note that FIG. 8 shows an example in which the area X2 is on the front side and the area X1 is on the back side. set based on

After setting the area X1 and the area X2, the refrigerator 1 generates an image to be displayed. In this case, as the display image G2, an image is generated in which the storage item M1 is stored on the far right side of the shelf board 10 and the storage item M2 is stored on the front side thereof.

Since a wide variety of canned juices are on the market, and new products are also on the market, it is not realistic to construct a database covering all types. Therefore, the refrigerator 1 can use a cylindrical CG image as shown in FIG. 8 as the image of the storage item M2. This can indicate to the user that a beverage is being stored. Of course, if an image that can be estimated to be the same as the stored item M2 is registered in the food DB or the like, that image may be used.

Alternatively, when the refrigerator 1 cannot estimate the type of the stored item M2 and cannot select it from the food DB, or when the possibility of being the same item is lower than a predetermined reference value, As such, it is possible to generate a transmission image G3 showing that the cylindrical article has been stored. Such a display mode can also indicate to the user that the beverage is being stored. Note that the reference value can be appropriately set such that the matching rate between the image of the stored food and the image registered in the food DB is, for example, 80% or more.

In addition, the refrigerator 1 can extract the stock M2 from the image captured by the camera 9 and generate the actual image G4 by superimposing the extracted image. Such a display mode can also indicate to the user that the beverage is being stored. In this case, since the image of the stored item M2 itself is displayed, even when a plurality of types of canned juice are stored, it is possible to accurately indicate to the user what type of canned juice is stored. can. In addition, when the type of the stored item M2 cannot be estimated and cannot be selected from the food DB, or when the possibility of being the same item is lower than the reference value, the image of the extracted stored item M2 is superimposed. can also be

Further, when the refrigerator 1 detects the food item M3 contained in the container, the refrigerator 1 can generate a detailed image G5 superimposed with characters indicating the corresponding food item. According to such a display mode, not only the container but also the type of contents can be indicated. Also, not only when it is detected that the container has been stored, but also when it is detected that the food itself has been stored like the storage item M1, it can be displayed together with the characters "apple", for example. In addition, when a symbol is attached to the container, the symbol or letters indicating the corresponding food can be displayed.

So far, an example of storing articles has been described. A process of inferring where the item was stored may also be performed based on the change in the location of the item as it is stored.

For example, as shown in FIG. 9, in a state in which a storage item M1 and a storage item M3 are stored, it is assumed that the storage item M3 is removed from the refrigerator 1, for example. In this case, when the opening of the door is detected, the camera 9 first picks up an image of the empty hand and picks up the article taken out of the storage room, here the article M3.

Therefore, when an article is imaged at the end of the imaging period such as when the door is closed, the refrigerator 1 determines that the article has been taken out of the storage compartment and identifies the article. Then, when the refrigerator 1 identifies that the article is, for example, the storage item M3, it executes a process of estimating the place where the storage item M3 was stored from the stored database. At this time, since it is assumed that a plurality of items M3 are stored, the refrigerator 1 determines from where in the refrigerator 1 the item M3 was taken out based on the trajectory of movement of the hand or the like.

Then, the refrigerator 1 removes the image of the storage item M3 from the display image of the storage item M3, here the detailed image G5 before update shown in FIG. 10, and generates or updates the detailed image G6 after update. Alternatively, the refrigerator 1 deletes the information of the storage item M3 taken out this time from the database updated when the storage item M3 was stored.

In this case, taking out any of the stored items as a trigger notifies the portable terminal 24 or the external server 25 that the item has been taken out, or transmits an image of the inside of the storage room after the item has been taken out. Or it can be displayed.

According to the refrigerator 1 described above, the following effects can be obtained.
The refrigerator 1 includes a housing 2 in which one or more storage compartments (3 to 7) are provided, and an imaging unit provided on the upper side of the housing 2 for imaging a range (R) in front of the storage compartment. camera 9, door opening sensor 17 as an opening/closing detection unit for detecting the opening and closing of the doors (3a to 7a) of the storage room, and camera 9, starting from the time when the opening of the door is detected by door opening sensor 17. a control unit 15 for executing a process of determining the position information of an article when it is taken in and out of the storage room based on the captured image, and estimating the location where the article is stored based on the obtained position information; , provided.

As a result, by using a small number of cameras 9 and by using cameras 9 whose positions are fixed, the front range of the refrigerator 1, that is, the range in which the articles to be taken in and out of the refrigerator 1 move can be easily and comprehensively controlled. It is possible to capture images with the configuration, and it is possible to achieve both a reduction in the number of cameras 9 and a reduction in the risk of blind spots. Therefore, it is possible to grasp the position of the article in the storage compartment with a simple configuration.

At this time, the camera 9 is provided in the upper part of the housing 2, that is, outside the storage room. For this reason, dew condensation on the lens is less likely to occur, and an article being taken in and out of the refrigerator 1 can be imaged clearly. In addition, since an external light source such as room lighting or sunlight can be used to capture an image of articles being put in and taken out of the refrigerator 1, it is not necessary to use high-intensity lighting for imaging.

Refrigerator 1 selects foods that can be probabilistically assumed to be the same as the stored articles from the food DB in which pre-stored foods are registered based on the images of the articles captured by camera 9. Then, the image of the selected food is superimposed on the image of the inside of the storage chamber, which has been converted into data in advance, and displayed on the display unit.

This makes it possible to easily grasp the situation in the storage compartment. In addition, by adopting a configuration that selects foods that can be probabilistically guessed, foods can be identified without complicating or enlarging the image processing and the food DB, and without imposing an extra burden on the user. be able to

Further, when the food cannot be selected, or when the probability of being the same as the selected food is lower than a predetermined reference value, the refrigerator 1 allows the image of the food to be transparent when superimposing the image. Alternatively, by making the image of the food relatively dark, a process of recognizably notifying the possibility that the food is not the same is executed. As a result, it is possible to present an approximate stored item, and it is possible to suppress the erroneous presentation of uncertain information.

The refrigerator 1 performs a process of superimposing the image of the article captured by the camera 9 on the image of the inside of the storage compartment and displaying it on the display unit. This allows the user to be shown exactly what food is being stored. This is significant when the type of the stored item M2 cannot be estimated and cannot be selected from the food DB, or when the possibility of being the same item is lower than the reference value.

Refrigerator 1 includes a storage unit 15a that stores food in association with identification information including at least one of the shape, color, character, and symbol of a container containing the food, and stores the food based on the identification information. identify. As a result, the type of food can be notified even when the food cannot be identified because it is contained in a container, or when containers of the same shape are used.

Refrigerator 1 performs a process of estimating the location where the article was stored based on the change in the position of the article when it is put into and taken out of the storage compartment. That is, the refrigerator 1 detects not only the articles stored in the storage compartment, but also the articles taken out of the storage compartment. As a result, it is possible to keep the situation in the storage room stored as data realistic.

The refrigerator 1 identifies the item that has been removed from the storage compartment and removes the image of the item from the image generated when the item was stored. This makes it possible to present the actual situation inside the storage room to the user.
Moreover, the refrigerator 1 is not limited to the configuration shown in the above-described embodiment, and the configuration can be changed as appropriate without changing the gist of the invention.

In the present embodiment, an example in which position information is obtained using two cameras 9 has been described, but for example, one camera 9 may be used to obtain position information based on the focal length. Alternatively, the camera 9 and an infrared rangefinder or the like may be provided so that the image pickup and the distance, that is, the position measurement are performed separately.

In the embodiment, a configuration is shown in which imaging is started at the timing when the opening of the door is detected. can also be configured to start In this case, the human detection sensor is provided on the operation panel 8, for example, and can be configured to have a function of assisting the operation by turning on the illuminated electrostatic switch or the display area when a person is detected. .

In the embodiment, an example of estimating the location where items are stored and displaying an image of the inside of the storage room has been shown, but the estimated location can also be used for more comfortable use of the refrigerator 1 . In the refrigerator 1, for example, in the case of the refrigerating compartment 3, there is a case where a recommended arrangement is set such as what kind of food should be stored in which place. In addition, the refrigerator 1 is generally configured to circulate cold air through a cold air inlet/outlet whose position is fixed.

Therefore, the food to be stored is identified, and if it is estimated that the food is stored in a place different from the recommended place, the user is notified of the recommended place from the notification unit or the like, or the food is stored near the entrance of cold air. When it is estimated that this is the case, by notifying the user to avoid that place, it is possible to maintain the freshness of the food and achieve uniform cooling in the storage room.

In addition, although it depends on the type, the temperature of the items stored in the refrigerating compartment and the vegetable compartment may be higher than in the storage compartment. be able to.
The above embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and its equivalents.

In the drawings, 1 is a refrigerator, 2 is a housing, 3 is a refrigerating chamber (storeroom), 3a is a left door (door), 3b is a right door (door), 4 is a vegetable compartment (storeroom), and 4a is a vegetable compartment. Door (door), 5 ice making compartment (storage compartment), 5a ice making compartment door (door), 6 small freezer compartment (storage compartment), 6a small freezer compartment door (door), 7 large freezer compartment (storage compartment) 7a is a large freezer compartment door (door); 9 is a camera (imaging unit); 14 is a display device (display unit); 15a is a storage unit; 16 is a communication device (communication unit); 17, 17a to 17d are door opening sensors (open/close detection units); 24 is a mobile terminal (display unit, storage unit); section, storage section), container DB, symbol DB, ingredient DB, and food DB indicate the food database.

Claims (8)

a housing provided with an imaging unit and one or more storage chambers;
A food is selected from a food database in which foods are registered in advance based on the information acquired by the imaging unit of the article stored in the storage room, and the image of the selected food is modeled. The computer graphic image is superimposed on the image of the inside of the storage chamber that has been converted into data in advance and displayed on the display unit, and the probability that the stored item and the selected food item are the same is a predetermined probability. If it is lower than the reference value, when superimposing the image of the selected food, by displaying the image of the food in a manner different from usual, control to execute a process of notifying the possibility that it is not the same food in an identifiable way Department and
Refrigerator with The imaging unit is provided on the upper side of the housing and captures an image of a range in front of the storage chamber,
and an open/close detection unit that detects opening/closing of the door of the storage room,
The control unit starts image capturing by the image capturing unit at the time when the open/close detection unit detects that the door is opened, and based on the captured image that is the information of the article, the article is taken in and out of the storage room. 2. The refrigerator according to claim 1, wherein positional information of the article at the moment is obtained, and a process of estimating the place where the article is stored is executed based on the obtained positional information. The control unit makes the image of the food transparent when superimposing the image of the selected food, or makes the image of the food relatively dark, so that the possibility that the food is not the same is identifiably notified . The refrigerator according to claim 1 or 2. 4. The controller according to any one of claims 1 to 3, wherein the controller superimposes an image of the article captured by the imaging unit on an image of the inside of the storage chamber that has been converted into data in advance, and displays the image on the display unit. Refrigerator as described. A storage unit for pre-associating and storing food and identification information including at least one of the shape, color, character and symbol of a container containing the food,
5. The refrigerator according to any one of claims 1 to 4, wherein said control unit identifies food based on said identification information. 6. The refrigerator according to any one of claims 1 to 5, wherein the control unit performs a process of estimating a place where the article was stored based on a change in the position of the article when it is taken in and out of the storage compartment. . The control unit identifies the item taken out from the storage room based on the image of the item captured by the imaging unit, and removes the image of the item from the image generated when the item is stored. The refrigerator according to claim 6. A food is selected from a food database in which foods are registered in advance based on information obtained by an imaging unit, a display unit for displaying an image, and the imaging unit for articles stored in a storage room. Then, the image of the selected food is superimposed on the image of the inside of the storage chamber that has been converted into data in advance as a modeled computer graphic image and displayed on the display unit, and the stored item is selected. If the probability that the selected food is the same food is lower than a predetermined reference value, when the image of the selected food is superimposed, the image of the food is displayed in a manner different from usual, so that the food may not be the same food A display system comprising: a control unit that executes processing for identifiably informing the user of the gender .

Images