WO2018021229A1

WO2018021229A1 - Simulation system, simulation method, and refrigerator

Info

Publication number: WO2018021229A1
Application number: PCT/JP2017/026652
Authority: WO
Inventors: 哲郎志田
Original assignee: 三菱電機株式会社
Priority date: 2016-07-27
Filing date: 2017-07-24
Publication date: 2018-02-01
Also published as: JPWO2018021229A1; JP6771563B2

Abstract

A simulation system is provided with: an imaging unit (111) that captures an image of the inside of a refrigerator (110) as a captured image; a refrigerator structural information memory unit (141a) that stores refrigerator structural information including dimensional information indicating the dimensions of a predetermined section inside the refrigerator (110); and a storage feasibility determining unit (143) that calculates the dimensions of an item to be stored in a captured image and determines whether the item to be stored can be stored by comparing the dimensions of the predetermined section in the captured image to the dimensions indicated by the dimensional information.

Description

Simulation system, simulation method, and refrigerator

The present invention relates to a simulation system, a simulation method, and a refrigerator, and in particular, a simulation system, a simulation method, and a simulation storage system for determining whether or not a storage object can be stored in the refrigerator. The present invention relates to a refrigerator that is a target for determining whether or not it can be stored.

2. Description of the Related Art Conventionally, a technique has been proposed in which a camera that captures an image of the interior of a refrigerator is installed in the refrigerator in order to check the storage state in the refrigerator.
For example, Patent Document 1 and Patent Document 2 propose a system in which a camera is installed in a refrigerator, and a storage state in a warehouse imaged by the camera can be confirmed by a display terminal such as a mobile phone.

Moreover, in patent document 3, the system which calculates the empty space by comparing the image in the warehouse in the state where the storage is not performed and the image in the storage in which the storage is stored is grasped. Proposed.

JP 2002-236798 A JP 2002-81818 A JP 2014-209048 A

The conventional system as described in Patent Document 1 or Patent Document 2 enables the user to take an image of the storage room and check the storage status with a display terminal or the like. It has been difficult to easily determine whether or not can be stored.
Furthermore, even if the empty space is explicitly displayed as in the system described in Patent Document 3, the size of the empty space cannot be compared with the size of the storage object, so whether or not it can be stored. It was difficult to judge easily.

Therefore, if you purchase a storage item (foodstuff, etc.) that is larger than the empty space, you may not be able to store it overflowing from the refrigerator, or you may not be able to store the storage item in the refrigerator. The situation that it was not possible to purchase was caused.

Therefore, an object of the present invention is to make it possible to easily determine whether or not an item to be stored can be stored.

A simulation system according to an aspect of the present invention is a simulation system for determining whether or not an object to be stored can be stored in a refrigerator, and an imaging unit that captures an image in the refrigerator as a captured image A refrigerator structure information storage unit for storing refrigerator structure information including dimension information indicating dimensions of a predetermined part in the refrigerator, dimensions of the predetermined part in the captured image, and the dimension information The size of the storage object is calculated in the captured image by comparing with the size of the predetermined portion indicated by the storage, and it is determined whether the storage object can be stored. An availability determination unit.

A simulation method according to an aspect of the present invention is a simulation method for determining whether or not a storage-scheduled item can be stored in a refrigerator, and images the image in the refrigerator as a captured image, By comparing the size of the predetermined portion in the refrigerator with the size of the predetermined portion in the captured image, the size in the captured image of the storage object is calculated, and the calculated Generating a first outline image of the planned storage object with a predetermined size, combining the first external image with the captured image to generate a first simulation image, and storing the planned storage object. It is characterized by determining whether or not it is possible.

The refrigerator which concerns on the 1st aspect of this invention is connected with the imaging part which images the image in a store | warehouse | chamber as a captured image, and an external network, and receives the shape information which shows the shape of the storage plan thing stored in the said store | warehouse | chamber A communication unit, a refrigerator structure information storage unit for storing refrigerator structure information including size information indicating the size of a predetermined part in the storage, the size of the predetermined part in the captured image, Comparing the dimension of the predetermined portion indicated by the dimension information to calculate the dimension in the captured image of the planned storage object based on the shape information, and storing the planned storage object And a storage availability determination unit that determines whether or not it can be performed.

The refrigerator which concerns on the 2nd aspect of this invention is connected with the imaging part which images the image in a store | warehouse | chamber as a captured image, the external network, and the shape information which shows the shape of the storage thing to store in the said store | warehouse | chamber, and the said store | warehouse | chamber A communication unit for receiving refrigerator structure information including dimension information indicating a dimension of a predetermined part of the image, a dimension of the predetermined part in the captured image, and the predetermined value indicated by the dimension information By comparing the size of the portion to be stored, the size of the planned storage object in the captured image is calculated based on the shape information, and it is determined whether or not the planned storage object can be stored. A storage availability determination unit.

According to one aspect of the present invention, it is possible to easily determine whether or not a storage object can be stored.

It is a block diagram which shows the structure of a storage condition simulation system schematically. (A) And (B) is a block diagram which shows an example of a hardware configuration. It is a flowchart which shows the process in a storage condition simulation system. It is the schematic which shows an example of a goods selection image. It is the schematic which shows an example of the captured image of a storage chamber. (A)-(D) are schematic diagrams showing an example of a perspective image of a purchase desired product. It is the schematic which shows the 1st example of a simulation image. It is the schematic which shows the 2nd example of a simulation image. It is the schematic which shows the 3rd example of a simulation image. It is the schematic which shows the 4th example of a simulation image. It is the schematic which shows the 5th example of a simulation image. It is the schematic which shows the 6th example of a simulation image. It is the schematic which shows the 7th example of a simulation image. It is the schematic which shows the 8th example of a simulation image. It is a block diagram which shows roughly the structure of the storage condition simulation system in a 1st modification. It is another block diagram which shows roughly the structure of the storage condition simulation system in a 2nd modification. It is another block diagram which shows roughly the structure of the storage condition simulation system in a 3rd modification.

FIG. 1 is a block diagram schematically showing a configuration of a storage situation simulation system 100 according to the embodiment.
The storage situation simulation system 100 is a simulation system for determining whether or not a storage object can be stored in a refrigerator. A method performed by the storage situation simulation system 100 is a simulation method.

The storage situation simulation system 100 includes a refrigerator 110, an operation terminal 120, a store server 130, and a cloud server 140.
The refrigerator 110 stores objects and cools the stored objects. Here, it is assumed that the user intends to store the purchase desired product, which is a planned storage item, in the refrigerator 110.
The operation terminal 120 is a terminal operated by a user.
The store server 130 is an article information server of a store where a user purchases a product.
The cloud server 140 is a simulation server that provides a storage situation simulation service.
The refrigerator 110, the operation terminal 120, the store server 130, and the cloud server 140 are connected to a communication line 150 such as the Internet and can perform communication. The communication line 150 is an external network.

The storage situation simulation system 100 includes the refrigerator 110, the operation terminal 120, the store server 130, and the cloud server 140, but the configuration of the storage situation simulation system 100 is not limited to such an example. For example, the refrigerator 110 may include the function of the cloud server 140. Further, the store server 130 may include the function of the cloud server 140. Furthermore, the refrigerator 110 may include the function of the operation terminal 120.

The refrigerator 110 includes an imaging unit 111, a refrigerator model number information providing unit 112, and a communication unit 113.
The imaging unit 111 captures an image in the storage room of the refrigerator 110 as a captured image. For example, the imaging unit 111 can be configured by a camera. Then, the imaging unit 111 provides a captured image that is a captured image to the cloud server 140 via the communication unit 113.

The refrigerator model number information providing unit 112 provides model number information, which is refrigerator identification information for identifying the refrigerator 110, to the cloud server 140 via the communication unit 113. The model number information is information indicating the model number of the refrigerator 110 and is used to specify the shape of the refrigerator 110. Therefore, the refrigerator identification information may be information other than the model number information as long as the shape of the refrigerator 110 can be uniquely specified. The refrigerator model number information providing unit 112 includes a model number information storage unit 112a that stores model number information. The model number information storage unit 112a can be configured by, for example, a nonvolatile memory.
The communication unit 113 performs communication via the communication line 150. For example, the communication unit 113 can be configured by a NIC (Network Interface Card).

Part or all of the refrigerator model number information providing unit 112 described above includes, for example, a memory 10 and a CPU that executes a program stored in the memory 10 (see FIG. 2A). It can be configured with a processor 11 such as a Central Processing Unit. Such a program may be provided through a network, or may be provided by being recorded on a recording medium.

In addition, as shown in FIG. 2B, for example, a part or all of the refrigerator model number information providing unit 112 may include a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC ( It can also be configured by a processing circuit 12 such as Application Specific Integrated Circuits (FPGA) or Field Programmable Gate Array (FPGA).

Returning to FIG. 1, the operation terminal 120 includes a display unit 121, an input unit 122, a purchase desired product designation unit 123, a storage posture selection unit 124, a storage side change unit 125, a storage position specification unit 126, and a communication Part 127.
The display unit 121 displays various images related to the storage situation simulation system 100.
The input unit 122 receives input of instructions from the user.
The display unit 121 and the input unit 122 can be configured by a touch panel, for example.

Based on the product selection image displayed on the display unit 121, the purchase desired product designating unit 123 receives designation of a product that the user desires to purchase via the input unit 122. Since the product designated here is a storage plan item to be stored in the refrigerator 110, the purchase desired product specification unit 123 uses the product selection image to store the selected product as a storage plan specification unit. Function as. The purchase desired product specifying unit 123 notifies the store server of the specified purchase desired product via the communication unit 127.
Based on the simulation image displayed on the display unit 121, the storage posture selection unit 124 selects a posture in which the user intends to store the desired product for purchase in the refrigerator 110 via the input unit 122. The storage posture selection unit 124 notifies the cloud server 140 of the selected storage posture via the communication unit 127.

Based on the simulation image displayed on the display unit 121, the storage side changing unit 125 changes the side on which the user intends to store the purchase desired product in the refrigerator 110 via the input unit 122. For example, the user can change whether the purchase desired product is stored on the right side toward the refrigerator 110 or on the left side. The storage side changing unit 125 notifies the cloud server 140 of the changed side via the communication unit 127.
Based on the simulation image displayed on the display unit 121, the storage position designation unit 126 receives a designation of a position where the user intends to store the purchase desired product in the refrigerator 110 via the input unit 122. The storage location designation unit 126 notifies the cloud server 140 of the designated storage location via the communication unit 127.
The communication unit 127 performs communication via the communication line 150. For example, the communication unit 127 can be configured by a NIC.

Part or all of the purchase desired product designation unit 123, the storage orientation selection unit 124, the storage side change unit 125, and the storage position specification unit 126 described above are, for example, as illustrated in FIG. The memory 10 and a processor 11 such as a CPU for executing a program stored in the memory 10 can be used. Such a program may be provided through a network, or may be provided by being recorded on a recording medium.

In addition, as shown in FIG. 2B, for example, as shown in FIG. 2B, a part or all of the purchase desired product specifying unit 123, the storage orientation selecting unit 124, the storage side changing unit 125, and the storage position specifying unit 126 may be a single unit. A circuit, a composite circuit, a programmed processor, a parallel programmed processor, or a processing circuit 12 such as an ASIC or FPGA can also be used.

Returning to FIG. 1, the store server 130 includes a product selection image providing unit 131, a purchase desired product storage unit 132, a product shape information providing unit 133, a product suitable storage room information providing unit 134, and a communication unit 135. .
The product selection image providing unit 131 causes the display unit 121 to display a product selection image for selecting a desired product from a plurality of products. For example, the product selection image providing unit 131 provides the operation terminal 120 with a product selection image for selecting a product from the products handled by the store server 130 via the communication unit 135. For example, the product selection image providing unit 131 includes a product image information storage unit 131a that stores product information related to products handled by the store server 130 and image information for generating a product selection image.
The purchase desired product storage unit 132 stores an article ID that is purchase desired product identification information for identifying the purchase desired product specified by the purchase desired product specifying unit 123 of the operation terminal 120.

The product shape information providing unit 133 provides the cloud server 140 with shape information indicating the shape of the purchase desired product specified by the purchase desired product specifying unit 123 of the operation terminal 120 via the communication unit 135. For example, the product shape information providing unit 133 includes a product shape information storage unit 133 a that stores product shape information indicating the shape of a product handled by the store server 130. Then, the product shape information providing unit 133 extracts information related to the purchase desired product from the product shape information, and uses the extracted information as shape information. Here, the product shape information indicates the external shape and dimensions of the product, and when a plurality of storage postures are set for the product, the external shape and dimensions of the product in each storage posture. Specifically, the storage posture is a posture such as vertical placement and horizontal placement. Depending on the product, depending on the storage position, liquid may spill or be damaged, so there are one type of storage position and multiple types of storage position.

The product suitable storage room information providing unit 134 provides the cloud server 140 with storage room information suitable for the purchase desired product designated by the purchase desired product designating unit 123 of the operation terminal 120 via the communication unit 135. For example, the product preferred storage room information providing unit 134 includes a product preferred storage room information storage unit 134a that stores product preferred storage room information indicating the storage room of the refrigerator 110 in which products handled by the store server 130 are to be stored. ing. And the suitable product storage room information provision part 134 extracts the information regarding purchase desired goods from the suitable product storage room information, and makes the extracted information storage room information.

The communication unit 135 performs communication via the communication line 150. For example, the communication unit 135 can be configured by a NIC.

A part or all of the product selection image providing unit 131, the product shape information providing unit 133, and the product suitable storage room information providing unit 134 described above may be, for example, a memory as illustrated in FIG. 10 and a processor 11 such as a CPU that executes a program stored in the memory 10. Such a program may be provided through a network, or may be provided by being recorded on a recording medium.

Further, a part or all of the product selection image providing unit 131, the product shape information providing unit 133, and the product suitable storage room information providing unit 134 are, for example, a single circuit, as shown in FIG. It can also be configured by a composite circuit, a programmed processor, a parallel programmed processor, or a processing circuit 12 such as an ASIC or FPGA.

The product image information storage unit 131a, the purchase desired product storage unit 132, the product shape information storage unit 133a, and the product preferred storage room information storage unit 134a are configured by a storage device such as an HDD (Hard Disk Drive) or a nonvolatile memory. be able to.

The cloud server 140 includes a refrigerator structure information provision unit 141, a storage availability determination unit 143, a simulation image generation unit 142, and a communication unit 144.
The refrigerator structure information providing unit 141 provides information indicating the internal structure of the refrigerator 110 to the storage availability determining unit 143 and the simulation image generating unit 142 based on the model number information provided from the refrigerator model number information providing unit 112 of the refrigerator 110. To do. For example, the refrigerator structure information providing unit 141 includes a refrigerator structure information storage unit 141 a that stores refrigerator structure information indicating various internal structures of the refrigerator 110. And the refrigerator structure information provision part 141 extracts the information corresponding to the notified model number information from the refrigerator structure information, and notifies the extracted information.

The refrigerator structure information includes structure information indicating the shape, size, and position of the storage room, and the shape, size, and position of the structure constituting the storage room. More specifically, the structure information indicates the internal dimensions of the refrigerator storage room and the shape, size, and position of a structure such as a shelf provided in the storage room.
The refrigerator structure information includes dimension information indicating the dimension of a predetermined portion of the refrigerator. The predetermined part is any part of the storage room or the structure constituting the storage room. In order to compare with a dimension in a captured image of a storage object to be described later, that portion is specified in advance.
Furthermore, when a moving structure that can move the position is included in the plurality of structures that constitute the storage room, the refrigerator structure information includes the shape, size, and movement of the moving structure. It further includes movable range information indicating a possible range. Here, the movable range is a range in which the position of the moving structure can be moved.
In addition, when a detachable detachable structure is included in the plurality of structures constituting the storage room, the refrigerator structure information indicates the detachable structure indicating the shape, size, and position of the detachable structure. Further information.

The simulation image generation unit 142 forms the shape information sent from the store server 130 into the picked-up image of the storage room indicated by the storage room information sent from the store server 130 among the picked-up images sent from the refrigerator 110. A simulation image is generated by synthesizing a product image (outer shape image) of a product having the outer shape and dimensions shown in FIG.
For example, the simulation image generation unit 142 sets the purchase desired product selected by the user as a storage object. Then, the simulation image generation unit 142 compares the dimension of the predetermined portion in the captured image with the dimension indicated by the dimension information included in the refrigerator structure information, so that the captured image of the storage object is stored. The dimensions at are calculated, and a perspective image (first outline image) that is an outline image of the storage object is generated with the calculated dimensions. The simulation image generation unit 142 generates a simulation image (first simulation image) by synthesizing the perspective image at a predetermined position of the captured image.
Moreover, the simulation image generation part 142, when the moving structure which can move a position is contained in the structure which comprises the refrigerator 110, the simulation image of the moving structure is included. Indicates the movable range.

Then, the simulation image generation unit 142 gives the generated simulation image to the operation terminal 120 via the communication unit 144. The operation terminal that has received such a simulation image displays the image on the display unit 121.

When the storage posture selection unit 124 receives a storage posture selection instruction for storing a storage object, via the simulation image displayed on the display unit 121, the simulation image generation unit 142 calculates as described above. A perspective image (second outline image) of the storage object in the selected storage posture with the selected dimensions is generated, and the generated perspective image is combined with the captured image to generate a simulation image (second simulation image) Is generated (updated). The simulation image generated in this way is sent to the display unit 121 and displayed.

In addition, when the storage side changing unit 125 receives a side change instruction for storing the storage object in the refrigerator 110 via the simulation image displayed on the display unit 121, the simulation image generating unit 142 A perspective image of an object to be stored when stored on the changed side with the calculated dimensions is generated, and a simulation image is generated (updated) by synthesizing the generated perspective image with the captured image. The simulation image generated in this way is sent to the display unit 121 and displayed.

Furthermore, when the storage position specifying unit 126 receives the storage position specification for storing the storage object, via the simulation image displayed on the display unit 121, the simulation image generating unit 142 performs the above-described operation. A perspective image (third outer shape image) of a storage object to be stored at the specified storage position with the calculated dimensions is generated, and the generated perspective image is combined with the captured image at the specified storage position. Thus, a simulation image (third simulation image) is generated (updated). The simulation image generated in this way is sent to the display unit 121 and displayed.

Also, the simulation image generation unit 142 updates the simulation image according to the determination result from the storage availability determination unit 143.

When the storage position is specified by the storage position specifying unit 126 using the simulation image, the storage availability determination unit 143 receives the captured image sent from the refrigerator 110 and the information sent from the refrigerator structure information providing unit 141. Based on the above, it is determined whether or not a product having the outer shape and dimensions indicated by the shape information sent from the store server 130 can be stored in the specified storage position.
For example, the storage availability determination unit 143 specifies the storage items already stored in the refrigerator 110 based on the captured image transmitted from the refrigerator 110 and the information transmitted from the refrigerator structure information providing unit 141. Whether or not the storage of the scheduled storage is hindered by the storage already stored in the refrigerator 110 or the structure constituting the refrigerator 110 when storing the planned storage in the specified storage position. Judging. The storage possibility determination unit 143 determines that the storage object cannot be stored when the storage of the storage object is hindered. When the storage of the storage object is not prevented, the storage possibility determination unit 143 determines that the storage object is not stored. It can be stored.

The storage availability determination unit 143 includes a determination information storage unit 143a that stores information necessary for performing the above determination.
Then, the storage availability determination unit 143 gives the determination result to the simulation image generation unit 142. In response to such a determination result, the simulation image generation unit 142 updates the simulation image to indicate the portion of the planned storage object that is hindering storage when the storage of the planned storage object is hindered.

In addition, when the structure which comprises the refrigerator 110 contains the moving structure which can move a position, the storage availability determination part 143 moves such a moving structure. Thus, it can be determined whether or not it can be stored.
For example, when the storage possibility determination unit 143 refers to the movable range information and moves the moving structure so that the storage of the storage object is not hindered, the storage of the storage object is not hindered. The position of the moving structure is specified as the moving position. Then, the storage availability determination unit 143 notifies the simulation image generation unit 142 of the movement position.
The simulation image generation unit 142 receives such notification, updates the simulation image, and moves the moving structure to the notified moving position, thereby indicating that the storage object can be stored.

In addition, when the structure constituting the refrigerator 110 includes a detachable detachable structure, the storage availability determination unit 143 considers the case where such a detachable structure is removed. Thus, it can be determined whether or not the storage is possible.
For example, when the storage possibility determination unit 143 refers to the attachment / detachment information and removes the attachment / detachment structure so that the storage of the storage object is not hindered, the storage structure is prevented from being blocked. And the simulation image generation unit 142 is notified of the detachable structure.
The simulation image generation unit 142 receives such notification, updates the simulation image, and indicates that the planned storage object can be stored by removing the notified detachable structure.

The communication unit 144 performs communication via the communication line 150. For example, the communication unit 144 can be configured by a NIC.

A part or all of the refrigerator structure information providing unit 141, the simulation image generating unit 142, and the storage availability determining unit 143 described above includes, for example, a memory 10 and a memory as illustrated in FIG. 10 and a processor 11 such as a CPU that executes a program stored in the computer 10. Such a program may be provided through a network, or may be provided by being recorded on a recording medium.

Further, a part or all of the refrigerator structure information providing unit 141, the simulation image generating unit 142, and the storage availability determining unit 143 may be configured as a single circuit, a composite circuit, a program, as illustrated in FIG. 2B, for example. It can also be constituted by a processing circuit 12 such as an integrated processor, a processor programmed in parallel, an ASIC or an FPGA.

In addition, the refrigerator structure information storage unit 141a and the determination information storage unit 143a can be configured by a storage device such as an HDD or a nonvolatile memory.

FIG. 3 is a flowchart showing processing in the storage situation simulation system 100.
First, the imaging unit 111 captures an image in the storage room of the refrigerator 110 (S10). When the refrigerator 110 includes a plurality of storage rooms such as a refrigerator room, a vegetable room, and a freezer room, the imaging unit 111 may be installed in each storage room. The imaging unit 111 transmits the captured image from the communication unit 113 to the simulation image generation unit 142 via the communication line 150.

Also, the refrigerator model number information providing unit 112 transmits the model number information stored in the model number information storage unit 112a from the communication unit 113 to the refrigerator structure information providing unit 141 via the communication line 150 (S11).

The refrigerator structure information providing unit 141 searches information indicating the internal structure of the refrigerator 110 using the model number indicated by the model number information from the refrigerator structure information stored in the refrigerator structure information storage unit 141a (S12). Then, the refrigerator structure information providing unit 141 gives information corresponding to the refrigerator 110 to the simulation image generating unit 142.

The product selection image providing unit 131 generates a product selection image for selecting a product from the products handled by the store server 130 and transmits the product selection image to the operation terminal 120 from the communication unit 113 via the communication line 150. To do. Then, the display unit 121 of the operation terminal 120 displays the product selection image (S13).

FIG. 4 is a schematic diagram illustrating an example of a product selection image.
In the product selection image IM1, the first product PD1, the second product PD2, and the third product PD3 are displayed. These products are all products handled by the store server 130.
The user can change the product displayed on the product selection image IM1 by inputting an execution instruction for the operation buttons B1 and B2 using the input unit 122.
In the product selection image IM1, the user moves the selection frame SL using the input unit 122, specifies a desired product, and inputs a selection instruction (S14). Thereby, the purchase desired product specifying unit 123 transmits information indicating the selected position from the communication unit 113 via the communication line 150 to the product selection image providing unit 131 as purchase desired product information. Upon receiving such information, the product selection image providing unit 131 stores the product ID of the purchase desired product corresponding to the selected position in the purchase desired product storage unit 132. The product ID may be any information that can uniquely specify the product handled by the store server 130, and the data format of the information does not specify a specific product.

Returning to FIG. 3, the product shape information providing unit 133 uses the product ID of the purchase desired product stored in the purchase desired product storage unit 132 from the product shape information stored in the product shape information storage unit 133 a, The shape information of the desired product for purchase is searched (S15). Then, the product shape information providing unit 133 transmits the shape information of the purchase desired product from the communication unit 135 to the simulation image generating unit 142 via the communication line 150.

Further, the product preferred storage room information providing unit 134 obtains the product ID of the purchase desired product stored in the purchase desired product storage unit 132 from the product preferred storage room information stored in the product preferred storage room information storage unit 134a. The storage room information of the desired product for purchase is searched (S16). Then, the preferred product storage room information providing unit 134 transmits the storage room information of the desired product to be purchased to the simulation image generating unit 142 via the communication unit 135 via the communication line 150.

Next, the simulation image generation unit 142 generates a simulation image (S17).
For example, the simulation image generation unit 142 determines a storage room in which the desired product for purchase is to be stored based on the storage room information transmitted from the preferred product storage room information providing unit 134. And the simulation image generation part 142 is a ratio between the dimension of the storage room determined based on the dimension information received from the refrigerator structure information provision part 141, and the dimension of the storage room determined in the captured image. Calculate the dimension ratio.

FIG. 5 is a schematic diagram illustrating an example of a captured image in the storage room.
The captured image IM2 includes the storage room 160 of the refrigerator 110, and the storage room 160 already stores the stored

items

161 and 162.
For example, when the information received from the refrigerator structure information providing unit 141 includes the three-dimensional position information of the point P1 and the point P2, the simulation image generating unit 142 is a distance between the point P1 and the point P2. The dimensional ratio is calculated from the distance between the points P1 and P2 included in the captured image. That is, the simulation image generation unit 142 calculates a dimensional ratio between the actual size of the storage room and the size of the storage room in the captured image.

Next, the simulation image generation unit 142 is a perspective view that is a three-dimensional outline image of the desired purchase product of the outer shape and dimensions indicated by the shape information transmitted from the product shape information providing unit 133 in accordance with the calculated size ratio. Create an image.
For example, the simulation image generation unit 142 corrects the actual size of the desired purchase product indicated by the shape information to the size in the captured image at the above-described size ratio, and the corrected size of the shape indicated by the shape information. Create a perspective image.

FIGS. 6A to 6D are schematic views showing examples of perspective images of products desired to be purchased.
FIG. 6A is a perspective image when the product desired to be purchased is stored vertically with the left side of the storage room being the storage position.
FIG. 6B is a perspective image when the product desired to be purchased is stored horizontally with the left side facing the storage room being the storage position.
FIG. 6C is a perspective image when the purchase desired product is stored horizontally with the right side of the storage room being the storage position.
FIG. 6D is a perspective image when the purchase desired product is stored vertically with the right side of the storage room being the storage position.
Here, the simulation image generation unit 142 may create one predetermined perspective image among the perspective images shown in FIGS. 6 (A) to (D).

Returning to FIG. 3, the simulation image generation unit 142 synthesizes the perspective image of the purchase desired product created as described above with the captured image of the storage room transmitted from the imaging unit 111, thereby storing the storage state in the storage room. (S17) is generated. For example, the simulation image generation unit 142 has an initial setting in which the right side or the left side of the image where the purchase desired product is to be installed and the installation posture of the purchase function product are determined. Is synthesized.

FIG. 7 is a schematic diagram illustrating an example of a simulation image.
The simulation image IM3 shown in FIG. 7 is created by synthesizing the perspective image shown in FIG. 6B with the captured image IM2 shown in FIG.
Here, the operation button B3 shown in the simulation image IM3 is a vertical setting that selects the vertical storage position of the purchase desired product PRD when the storage position is vertical or horizontal. Select button.
The operation button B4 is a horizontal selection button for selecting that the storage posture is horizontal.
In the simulation image IM3 shown in FIG. 7, since landscape orientation is selected, the perspective image of the purchase desired product PRD drawn at the bottom is depicted horizontally.

In addition, a shelf that is one of the structures in the storage room of the refrigerator 110 is a moving shelf, and when the movable range of the moving shelf is notified from the refrigerator structure information providing unit 141, the simulation image generating unit 142 Displays the movable range of the movable shelf in the simulation image.
For example, if the moving shelf SS can move from the position SP1 to the position SP2 as in the simulation image IM4 as shown in FIG. 8, in order to show such a movable range, the position SP1 to the position SP2 Or an arrow from position SP1 to position SP2.

3, the generated simulation image is transmitted from the communication unit 135 to the operation terminal 120 via the communication line 150 and displayed on the display unit 121 (S18).

When the user selects the storage posture of the desired product for purchase via the input unit 122 in the simulation image displayed on the display unit 121 (Yes in S19), the process proceeds to step S20.
In step S 20, the storage posture selection unit 124 notifies the simulation image generation unit 142 of the selected storage posture via the communication line 150 from the communication unit 135. The simulation image generation unit 142 updates the simulation image according to the notified storage posture, and transmits the updated simulation image to the operation terminal 120 from the communication unit 135 via the communication line 150. Then, the process returns to step S18.

FIG. 7 is a simulation image IM3 when the purchase desired product PRD is stored horizontally on the left side of the storage room as described above. For example, when an execution instruction specifying the operation button B3 is input in the simulation image IM3, the simulation image generation unit 142 generates a simulation image IM5 as shown in FIG. The simulation image IM5 is an image when the desired product PRD is to be stored vertically on the left side of the storage room.

Returning to FIG. 3, when the user changes the storage side of the purchase desired product via the input unit 122 in the simulation image displayed on the display unit 121 (Yes in S21), the process proceeds to step S22. Proceed to
In step S 22, the storage side changing unit 125 notifies the simulation image generating unit 142 of the changed side from the communication unit 135 via the communication line 150. The simulation image generation unit 142 updates the simulation image according to the notified side, and transmits the updated simulation image to the operation terminal 120 from the communication unit 135 via the communication line 150. Then, the process returns to step S18.

FIG. 7 is a simulation image IM3 in the case where the purchase desired product PRD is stored horizontally on the left side of the storage room as described above. For example, in the simulation image IM3, when the storage side is changed to the right side toward the storage room as in the case where an operation for shifting the purchase desired product PRD to the right is input to the input unit 122, the simulation image generation unit 142 generates a perspective image of the purchase desired product PRD when the product is to be stored horizontally on the right side of the storage room as in the simulation image IM6 shown in FIG. 10, and captures the image. Composite to the image.

Further, in the simulation image IM5 shown in FIG. 9, when the storage side is changed to the right side of the storage room, the simulation image generation unit 142, like the simulation image IM7 shown in FIG. Then, on the right side of the storage room, a perspective image of the purchase desired product PRD when the product is to be stored vertically is generated, and the image is combined with the captured image.

Returning to FIG. 3, when the user designates the storage position of the purchase desired product via the input unit 122 in the simulation image displayed on the display unit 121 (Yes in S 23), the process proceeds to step S 24. Proceed to
In step S 24, the storage position designation unit 126 notifies the simulation image generation unit 142 of the designated storage position from the communication unit 135 via the communication line 150. The simulation image generation unit 142 updates the simulation image according to the notified storage position, and transmits the updated simulation image to the operation terminal 120 from the communication unit 135 via the communication line 150. Then, the process returns to step S18.

For example, in the simulation image IM3 shown in FIG. 7, when the storage position is changed into the storage room, such as when an operation for shifting the purchase desired product PRD onto the shelf SEL is input to the input unit 122. In other words, the storage availability determination unit 143 determines whether or not the purchase desired product can be stored in the storage position in the designated storage room.

Specifically, first, the storage possibility determination unit 143 estimates the range of stored items already stored in the refrigerator 110 from the captured image sent from the refrigerator 110. For example, the storage possibility determination unit 143 can perform such estimation by storing the captured image of the storage room before the object is stored in the refrigerator 110 in the determination information storage unit 143a. Further, for example, the storage availability determination unit 143 can perform such estimation by installing a pressure sensor on the shelf of the refrigerator 110 and receiving the detected value.

Next, the storage availability determination unit 143 calculates the range of the purchase desired product when stored in the storage position specified by the storage position specifying unit 126 from the shape information provided from the product shape information providing unit 133.
Next, the storage possibility determination unit 143 determines the range of the structure indicated by the structure information of the storage room of the refrigerator 110 provided from the refrigerator structure information providing unit 141, the estimated range of the stored item, and the calculated purchase desired product. It is determined whether or not these ranges overlap each other. Then, if they do not overlap, the storage availability determination unit 143 determines that the storage of the purchase desired product is possible without being hindered. In the case of overlapping, the storage of the desired product for purchase is hindered, and it is determined that it cannot be stored.

The storage availability determination unit 143 gives the determination result of storage availability determination to the simulation image generation unit 142. Here, when the determination result indicates that storage is not possible, the storage availability determination unit 143 also provides the simulation image generation unit 142 with disturbance range information indicating the range of the portion corresponding to the purchase desired product in the overlapping range.
When the determination result is storable, the simulation image generation unit 142 updates the simulation image by synthesizing a perspective image of the desired product for purchase with the selected storage position at the specified storage position.
On the other hand, when the determination result is unacceptable, the simulation image generation unit 142 synthesizes a perspective image of the desired product for purchase with the selected storage position at the specified storage position, and an image indicating the interference range. As a result, the simulation image is updated. For example, the simulation image generation unit 142, when storing the purchase desired product in the storage position from the range indicated by the interference range information, specifies a portion that prevents the storage by contacting the storage object and the structure, An image having that portion as the interference range is generated and synthesized with the captured image.

FIG. 12 is a schematic diagram illustrating a simulation image when the determination result cannot be stored.
In the simulation image IM8 shown in FIG. 12, the desired purchase product PRD is stored in the specified storage position in the selected vertical position.
However, in the simulation image IM8, the range that overlaps the moving shelf SS at the position SP2 in the shape range of the purchase desired product PRD is shown as the interference range ITA. For example, the simulation image generation unit 142 can indicate such an interference range ITA by synthesizing an image indicating such an interference range ITA.

The simulation image generation unit 142 causes the display unit 121 to display the generated simulation image IM8.
The user who has confirmed the simulation image IM8 as shown in FIG. 12 on the display unit 121 can easily recognize that the user can not store the product in the refrigerator 110 even if he purchases the purchase desired product PRD.

Further, the storage availability determination unit 143 determines the status of the storage room of the refrigerator 110 when the shelf that prevents the storage of the purchase desired product is a movable movable shelf or a removable removable shelf in the determination of storage availability. Can be determined by changing the state of the storage chamber.

For example, when the shelf preventing the purchase desired product PRD from being stored is the moving shelf SS as shown in FIG. 8, the moving shelf SS installed at the position SP2 can be moved to the position SP1. it can.
Therefore, the storage availability determination unit 143 assumes that the moving shelf SS has moved to the position SP1 based on the movable range information of the moving shelf included in the structure information provided from the refrigerator structure information providing unit 141. Then, the storage possibility determination is performed. Here, when the storage availability determination result does not overlap and the storage is possible, the storage availability determination unit 143 indicates the storage availability determination result and the conditions for enabling storage, and the simulation image generation unit 142. To give. Here, the condition for enabling storage is to move the movable shelf SS to the position SP1.

FIG. 13 is a schematic diagram illustrating an example of a simulation image when conditions for enabling storage are indicated.
In this case, since there is no disturbing range, the disturbing range is not shown in the simulation image IM9. However, in the simulation image IM9, a message field MEA1 indicating conditions for enabling storage is displayed.
The message field MEA1 shown in FIG. 13 notifies the user that it is a simulation image when it is assumed that the moving shelf SS has been moved to the position SP1.
The simulation image generation unit 142 displays the simulation image IM9 on the display unit 121.
Note that the simulation image generation unit 142 processes the captured image so as to move the movable shelf SS to the position SP1, and generates the simulation image IM9 shown in FIG. 13, but performs such processing. It does not have to be.

If the user who has confirmed the simulation image IM9 as shown in FIG. 13 on the display unit 121 moves the movable shelf SS to the position SP1, the user can store the desired product PRD in the refrigerator 110 even if he / she purchases it. Can be easily recognized.

In addition, when the shelf that prevents the purchase desired product PRD from being stored is a removable shelf, such a shelf can be removed.
Therefore, the storage availability determination unit 143 performs storage availability determination when it is assumed that the detachable shelf has been removed based on the detachment information of the detachable shelf included in the structure information provided from the refrigerator structure information providing unit 141. . Here, when the storage availability determination result does not overlap and the storage is possible, the storage availability determination unit 143 indicates the storage availability determination result and the conditions for enabling storage, and the simulation image generation unit 142. To give. Here, the condition for enabling storage is to remove the detachable shelf.

FIG. 14 is a schematic diagram illustrating an example of a simulation image when conditions for enabling storage are indicated.
In this case, since there is no disturbing range, the disturbing range is not shown in the simulation image IM10. However, in the simulation image IM10, a message field MEA2 indicating conditions for enabling storage is displayed.
The message field MEA2 shown in FIG. 14 notifies the user that it is a simulation image when it is assumed that the detachable shelf has been removed.
The simulation image generation unit 142 displays the simulation image IM10 on the display unit 121.

The user confirming the simulation image IM10 as shown in FIG. 14 on the display unit 121 can easily recognize that the user can store the product in the refrigerator 110 even if he / she purchases the desired product PRD by removing the detachable shelf. .
Note that the simulation image generation unit 142 processes the captured image so as to delete the detachable shelf and generates the simulation image IM10 shown in FIG. 14, but such processing may not be performed. .

According to the embodiment described above, since the captured image in the storage room of the refrigerator 110 can be compared with the perspective image of the storage object having the same dimensional ratio as the captured image, the user can store It is possible to accurately know whether or not the planned item can be stored in the storage chamber of the refrigerator 110. For this reason, a situation occurs in which a storage item is purchased but cannot be stored in the refrigerator 110, or a purchase opportunity is missed because the storage item cannot be stored in the refrigerator 110. Can be prevented.

According to the embodiment described above, the structure information of the storage room of the refrigerator 110 can be retrieved and obtained from the model number information of the refrigerator 110, so that the structure information of the storage room of the refrigerator 110 can be obtained from the system. It is possible to save the user's input and to use accurate structural information in the storage room.

According to the embodiment described above, the shape information of the storage object can be retrieved and obtained from the item ID of the storage object, so that the user of the system can acquire the shape information of the storage object. It is possible to save input and to use accurate shape information of a storage object.

According to the embodiment described above, the storage room information suitable for the storage object can be retrieved and obtained from the item ID of the storage object, so that the storage room information suitable for the storage object can be obtained. Therefore, it is possible to save the trouble of inputting by the user of the system. In addition, it is possible to prevent the risk of making a mistake in determining whether or not a planned storage item can be stored by comparing a captured image in a storage room other than the storage room suitable for the planned storage item and a perspective image of the planned storage item. .

According to the embodiment described above, the captured image in the storage room can be compared with the perspective images of a plurality of storage postures set in advance in the storage object, so that it can be stored in a plurality of storage postures. Can be determined. This makes it possible to try to store in some storage postures, but to store in other storage postures, and to prevent the risk of determining that storage is possible, although it can actually be stored.

According to the embodiment described above, the position of the stored item already stored in the refrigerator 110 is estimated from the captured image in the storage chamber, and the already stored item and the planned stored item do not collide with each other. In order to determine whether or not it is possible, the user of the system determines whether or not storage is possible by simply comparing the captured image in the storage room other than the storage room suitable for the storage object and the perspective image of the storage object. Therefore, it is possible to determine whether or not the storage is possible more easily and accurately.

According to the embodiment described above, the use of the system is used to determine whether the structure and the planned storage object do not collide and can be stored based on the position and shape information of the structure in the storage room. It is easier and more accurate for the user to determine whether or not storage is possible by simply comparing the captured image in a storage room other than the storage room suitable for the storage object and the perspective image of the storage object. Judgment can be made.

According to the embodiment described above, when the position of the structure in the storage room can be changed, the structure when the structure is installed at a position that can be taken and the planned storage object do not collide, In order to determine whether or not storage is possible, the user of the system can determine whether or not storage is possible by simply comparing a captured image in a storage room other than the storage room suitable for the storage object and a perspective image of the storage object. Therefore, it is possible to easily and accurately determine whether or not storage is possible.

<First Modification>
By the way, as described at the beginning, the refrigerator 110 may include the function of the cloud server 140. The refrigerator 110 may include the function of the operation terminal 120.
FIG. 15 is a block diagram schematically showing the configuration of the refrigerator 210 and the storage situation simulation system 200 including the refrigerator 210 in the first modification.
The refrigerator 210 incorporates the function of the cloud server 140.
The refrigerator 210 includes an imaging unit 111, a refrigerator model number information providing unit 112, a communication unit 113, a refrigerator structure information providing unit 141 having a refrigerator structure information storage unit 141a, and a storage availability determination unit 143.

The imaging unit 111 captures an image inside the refrigerator 210 as a captured image.
The refrigerator model number information providing unit 112 gives the model number information stored in the model number information storage unit 112 a to the refrigerator structure information providing unit 141.
The communication unit 113 is connected to the communication line 150 and receives shape information of an object to be stored in the warehouse.

The refrigerator structure information storage unit 141a stores refrigerator structure information including dimension information indicating the dimensions of a predetermined portion in the refrigerator.
The storage possibility determination unit 143 compares the size of a predetermined portion in the captured image with the size indicated by the size information, thereby calculating the size in the captured image of the storage object to be stored in the storage, It is determined whether or not a storage object can be stored. The determination as to whether or not the storage object can be stored is as described above.
Note that the refrigerator structure information storage unit 141 a may not be provided in the refrigerator 210, and the communication unit 113 receives dimensional information indicating the dimensions of a predetermined portion in the refrigerator 210 via the communication line 150. May be.

The refrigerator 210 may include a simulation image generation unit 142. The simulation image generation unit 142 generates a first product image (outer shape image) of an object to be stored based on the dimensions in the captured image calculated by the storage availability determination unit 143, and combines the product image with the captured image. To generate a simulation image (first simulation image). The communication unit 113 transmits the simulation image to the operation terminal 120 via the communication line 150. The operation terminal 120 includes a display unit 121. The received simulation image is displayed on the display unit 121.
Note that the simulation image generation unit 142 may not be provided in the refrigerator 210 but may be provided in the operation terminal 120. In this case, the communication unit 113 transmits the dimensions in the captured image calculated by the storage availability determination unit 143 to the operation terminal 120 via the communication line 150.

<Second Modification>
Further, as described at the beginning, the refrigerator 110 may include the function of the operation terminal 120.
FIG. 16 is a block diagram schematically showing the configuration of the refrigerator 310 and the storage situation simulation system 300 including the refrigerator 310 in the second modified example.
The refrigerator 310 incorporates the function of the operation terminal 120.
In this case, the storage availability determination unit 143 provided in the cloud server 140 determines whether or not the storage-scheduled item can be stored. Then, the simulation image generation unit 142 generates a first product image (outer shape image) of an object to be stored based on the dimensions in the captured image calculated by the storage availability determination unit 143, and combines the product image with the captured image. By doing so, a simulation image (first simulation image) is generated. The communication unit 113 receives the simulation image via the communication line 150. The display unit 121 displays the received simulation image.

<Third Modification>
The refrigerator 110 may include the functions of the cloud server 140 and the operation terminal 120. FIG. 17 is a block diagram schematically showing a configuration of a refrigerator 410 and a storage situation simulation system 400 including the refrigerator 410 in the third modification.
In this case, inside the refrigerator 410, it is determined whether or not a storage object can be stored, whether a simulation image is generated, and a simulation image is displayed. The communication unit 113 communicates with the store server 130 via the communication line 150 and receives article information and the like of the store where the user purchases the product.

100, 200, 300, 400 storage situation simulation system, 110, 210, 310, 410 refrigerator, 111 imaging unit, 112 refrigerator model number information providing unit, 113 communication unit, 120 operation terminal, 121 display unit, 122 input unit, 123 purchase Desired product designation unit, 124 storage orientation selection unit, 125 storage side change unit, 126 storage position specification unit, 127 communication unit, 130 store server, 131 product selection image providing unit, 132 purchase desired product storage unit, 133 product shape information provision Part, 134 Goods suitable storage room information provision part, 135 communication part, 140 cloud server, 141 refrigerator structure information provision part, 143 storage availability determination part, 142 simulation image generation part, 144 communication part 10 memory, 11 processor, 12 processing circuit.

Claims

A simulation system for determining whether or not a storage object can be stored in a refrigerator,
An imaging unit that captures an image in the refrigerator as a captured image;
A refrigerator structure information storage unit for storing refrigerator structure information including dimensional information indicating dimensions of a predetermined portion in the refrigerator;
By comparing the size of the predetermined portion in the captured image with the size of the predetermined portion indicated by the size information, the size of the storage object in the captured image is calculated. And a storage availability determination unit that determines whether or not the storage object can be stored.
A first outline image of the planned storage object is generated based on a size of the planned storage object in the captured image, and a first simulation image is generated by synthesizing the first external image with the captured image. A simulation image generation unit;
The simulation system according to claim 1, further comprising: a display unit that displays the first simulation image.
A storage posture selection unit that receives an instruction to select a storage posture for storing the planned storage object via the first simulation image displayed on the display unit;
The simulation image generation unit generates a second outer shape image of the planned storage object in the selected storage posture with the calculated size, and combines the second outer shape image with the captured image. 2 simulation images are generated,
The simulation system according to claim 2, wherein the display unit displays the second simulation image.
A storage position specifying unit for receiving specification of a storage position for storing the storage object through the first simulation image displayed on the display unit;
The simulation image generation unit generates a third outline image of the storage object to be stored in the specified storage position with the calculated size, and stores the third outline image in the specified storage. A third simulation image is generated by combining the captured image at the position,
The simulation system according to claim 2, wherein the display unit displays the third simulation image.
A storage position designation unit for receiving designation of a storage position for storing the storage object through the second simulation image displayed on the display unit;
The simulation image generation unit generates a third outline image of the storage object to be stored in the specified storage position with the calculated size, and stores the third outline image in the specified storage. A third simulation image is generated by combining the captured image at the position,
The simulation system according to claim 3, wherein the display unit displays the third simulation image.
The refrigerator structure information further includes structure information indicating the shape, size and position of a plurality of structures provided in the refrigerator,
The storage availability determination unit specifies a stored item already stored in the refrigerator from the captured image, and refers to the structure information when storing the planned storage item in the storage position. Determining whether the storage object or the plurality of structures prevents storage of the storage object,
5. The storage device according to claim 4, wherein when the storage of the planned storage object is hindered, the simulation image generation unit indicates a part of the storage target object that prevents storage in the third simulation image. 5. The simulation system according to 5.
The plurality of structures include moving structures that can move positions,
The refrigerator structure information further includes movable range information indicating the shape, size and movable range of the movable structure,
When the storage possibility determination unit refers to the movable range information and moves the moving structure so that the storage of the storage object is not hindered, the storage of the storage object is prevented. Specify the position of the moving structure as a moving position,
The simulation image generation unit indicates whether or not the storage object can be stored by moving the moving structure to the moving position in the third simulation image. Item 7. The simulation system according to Item 6.
The plurality of structures include a detachable detachable structure,
The refrigerator structure information further includes attachment / detachment information indicating the shape, size and position of the attachment / detachment structure,
The storage availability determination unit refers to the attachment / detachment information and removes the attachment / detachment structure, so that the storage of the storage object is not hindered, the storage of the storage object is not interrupted, Identify the detachable structure,
The simulation image generation unit indicates whether or not the planned storage object can be stored by removing the specified detachable structure in the third simulation image. The simulation system described in 1.
The plurality of structures include moving structures that can move positions,
The refrigerator structure information further includes movable range information indicating the shape, size and movable range of the movable structure,
The simulation system according to claim 6, wherein the simulation image generation unit indicates the movable range in the first simulation image.
A product selection image providing unit for displaying a product selection image for selecting a desired product from a plurality of products on the display unit;
Via the product selection image displayed on the display unit, a storage plan specifying unit that sets the selected product as the storage plan,
A product shape information providing unit that provides the simulation image generating unit with shape information indicating the outer shape and dimensions of the selected product,
The simulation system according to claim 2, wherein the simulation image generation unit generates the first outer shape image based on the shape information.
The refrigerator is provided with a plurality of storage rooms,
The imaging unit captures images in the plurality of storage rooms,
A product suitable storage room information providing unit for providing storage room information indicating a storage room for storing the selected product to the simulation image generating unit;
The simulation system according to claim 10, wherein the simulation image generation unit synthesizes the first outer shape image with a captured image in a storage room indicated by the storage room information.
A simulation method for judging whether or not a storage object can be stored in a refrigerator,
Capture the image in the refrigerator as a captured image,
By comparing the dimension of the predetermined part in the refrigerator with the dimension of the predetermined part in the captured image, the dimension in the captured image of the storage object is calculated,
Generating a first outline image of the storage object with the calculated dimensions;
A simulation method comprising: generating a first simulation image by combining the first outer shape image with the captured image and determining whether or not the storage object can be stored.
An imaging unit that captures an image in the cabinet as a captured image;
A communication unit that is connected to an external network and receives shape information indicating the shape of a storage object to be stored in the storage;
A refrigerator structure information storage unit for storing refrigerator structure information including dimensional information indicating dimensions of a predetermined portion in the storage;
Based on the shape information, the imaging of the storage object is performed by comparing the dimension of the predetermined part in the captured image with the dimension of the predetermined part indicated by the dimension information. A storage availability determination unit that calculates a size in an image and determines whether or not the planned storage object can be stored.
An imaging unit that captures an image in the cabinet as a captured image;
A communication unit that is connected to an external network and receives shape information indicating the shape of a storage object to be stored in the storage and refrigerator structure information including size information indicating a dimension of a predetermined portion in the storage;
Based on the shape information, the imaging of the storage object is performed by comparing the dimension of the predetermined part in the captured image with the dimension of the predetermined part indicated by the dimension information. A storage availability determination unit that calculates a size in an image and determines whether or not the planned storage object can be stored.
The refrigerator according to claim 13 or 14, wherein the communication unit transmits a size of the storage object in the captured image to the external network.
A first outline image of the planned storage object is generated based on a size of the planned storage object in the captured image, and a first simulation image is generated by synthesizing the first external image with the captured image. A simulation image generation unit;
The refrigerator according to claim 13, further comprising a display unit that displays the first simulation image.