JP6771563B2 - Simulation system, simulation method and refrigerator - Google Patents

Description

The present invention relates to a simulation system, a simulation method and a refrigerator, and in particular, a simulation system for determining whether or not an object to be stored can be stored in the refrigerator, a simulation method, and an object to be stored by this simulation system. Regarding the refrigerator that is the target of determining whether or not it can be stored.

Conventionally, in order to check the storage status in the refrigerator, a technique of installing a camera that images the inside of the refrigerator in the refrigerator has been proposed.
For example, Patent Document 1 and Patent Document 2 propose a system in which a camera is installed in a refrigerator and the storage status in the refrigerator imaged by the camera can be confirmed by a display terminal such as a mobile phone.

Further, in Patent Document 3, there is a system for grasping the storage status and calculating the empty space by comparing the image in the refrigerator in a state where it is not stored and the image in the refrigerator in use where the stored items are stored. Proposed.

JP-A-2002-236798 JP-A-2002-81818 JP-A-2014-209048

The conventional system as described in Patent Document 1 or Patent Document 2 makes it possible to take an image of the storage room and check the storage status on a display terminal or the like, but a storage schedule to be newly stored. It was difficult to easily determine whether or not the device could be stored.
Further, as in the system described in Patent Document 3, even if the empty space is explicitly displayed, it is not possible to compare the size of the empty space with the size of the planned storage object, so whether or not the space can be stored. It was difficult to judge easily.

Therefore, you may purchase items (foodstuffs, etc.) that exceed the empty space, causing a situation where the items you plan to store overflow from the refrigerator and cannot be stored, or you may not be able to store the items you plan to store in the refrigerator. There was a situation where I couldn't buy it.

Therefore, an object of the present invention is to make it possible to easily determine whether or not the planned storage object can be stored.

The simulation system according to one aspect of the present invention is a simulation system for determining whether or not an object to be stored can be stored in the refrigerator, and is an imaging unit that captures an image in the refrigerator as an image. A desired product is selected from a plurality of products, a refrigerator structure information storage unit that stores refrigerator structure information including dimensional information indicating dimensions of a predetermined portion in the refrigerator, a display unit that displays an image, and a plurality of products. A product selection image providing unit for displaying a product selection image for the purpose on the display unit, and a storage schedule designation unit for which the selected product is the storage schedule via the product selection image displayed on the display unit. Then, a first external image of the planned storage object is generated based on the dimensions in the captured image of the planned storage object, and the first external shape image is combined with the captured image to obtain a first simulation image. A simulation image generation unit to be generated, a product shape information providing unit that provides shape information indicating the outer shape and dimensions of the selected product to the simulation image generation unit, and dimensions of the predetermined portion in the captured image. By comparing the dimensions of the predetermined portion indicated by the dimensional information with the outer diameter and the dimensions indicated by the shape information, the dimensions of the planned storage object in the captured image are calculated. A storage availability determination unit for determining whether or not the storage schedule can be stored is provided , and the simulation image generation unit generates the first outer shape image based on the shape information, and the above-mentioned The display unit is characterized by displaying the first simulation image .

The simulation method according to one aspect of the present invention is a simulation method for determining whether or not an object to be stored can be stored in the refrigerator, and the image in the refrigerator is captured as an image and described above. By comparing the dimensions of the predetermined portion in the refrigerator with the dimensions of the predetermined portion in the captured image and the outer shape and dimensions of the planned storage object, the captured image of the planned storage object is compared. calculating the dimensions of the inner, and generates a first contour image of the storage schedule was in the calculated sizes, the first outline image to generate a first simulated image by synthesizing the captured image It is characterized in that it is determined whether or not the planned storage object can be stored.

The refrigerator according to the first aspect of the present invention is connected to an imaging unit that captures an image in the refrigerator as an captured image and an external network, and receives shape information indicating the shape of the storage object to be stored in the refrigerator. The communication unit, the refrigerator structure information storage unit that stores refrigerator structure information including dimensional information indicating the dimensions of the predetermined portion in the refrigerator, the dimensions of the predetermined portion in the captured image, and the above. By comparing with the dimensions of the predetermined portion indicated by the dimensional information, the dimensions of the planned storage object in the captured image are calculated based on the shape information, and the storage planned object is stored. It is characterized by including a storage availability determination unit for determining whether or not the image can be stored.

The refrigerator according to the second aspect of the present invention is connected to an imaging unit that captures an image in the refrigerator as an image, an external network, and shape information indicating the shape of a storage object to be stored in the refrigerator and the refrigerator. A communication unit that receives refrigerator structure information including dimensional information indicating the dimensions of a predetermined portion of the image, the dimensions of the predetermined portion in the captured image, and the predetermined size indicated by the dimensional information. By comparing with the dimensions of the portion, the dimensions of the planned storage object in the captured image are calculated based on the shape information, and it is determined whether or not the storage schedule can be stored. It is characterized by including a storage availability determination unit.

According to one aspect of the present invention, it is possible to easily determine whether or not the planned storage object can be stored.

It is a block diagram which shows schematic structure of the storage situation simulation system. (A) and (B) are block diagrams showing an example of a hardware configuration. It is a flowchart which shows the processing in a storage situation simulation system. It is the schematic which shows an example of the product selection image. It is the schematic which shows an example of the captured image of a storage room. (A) to (D) are schematic views showing an example of a perspective image of a product desired to be purchased. It is the schematic which shows the 1st example of the simulation image. It is the schematic which shows the 2nd example of the simulation image. It is the schematic which shows the 3rd example of the simulation image. It is the schematic which shows the 4th example of the simulation image. It is the schematic which shows the 5th example of the simulation image. It is the schematic which shows the 6th example of the simulation image. It is the schematic which shows the 7th example of the simulation image. It is the schematic which shows the 8th example of the simulation image. It is a block diagram which shows schematic structure of the storage situation simulation system in 1st modification. It is another block diagram which shows schematic the structure of the storage situation simulation system in the 2nd modification. It is another block diagram which shows roughly the structure of the storage situation simulation system in the 3rd modification.

FIG. 1 is a block diagram schematically showing the configuration of the storage status simulation system 100 according to the embodiment.
The storage status simulation system 100 is a simulation system for determining whether or not the storage schedule can be stored in the refrigerator. The method performed by the storage status simulation system 100 is a simulation method.

The storage status simulation system 100 includes a refrigerator 110, an operation terminal 120, a store server 130, and a cloud server 140.
Refrigerator 110 stores things and cools the stored things. Here, it is assumed that the user intends to store the desired product to be stored in the refrigerator 110.
The operation terminal 120 is a terminal operated by the user.
The store server 130 is an article information server of a store where a user purchases a product.
The cloud server 140 is a simulation server that provides a storage status simulation service.
The refrigerator 110, the operation terminal 120, the store server 130, and the cloud server 140 are connected to a communication line 150 such as the Internet, and can communicate with each other. The communication line 150 is an external network.

The storage status simulation system 100 includes a refrigerator 110, an operation terminal 120, a store server 130, and a cloud server 140, but the configuration of the storage status simulation system 100 is not limited to such an example. For example, the refrigerator 110 may include the function of the cloud server 140. Further, the store server 130 may include the function of the cloud server 140. Further, the refrigerator 110 may include the function of the operation terminal 120.

The refrigerator 110 includes an imaging unit 111, a refrigerator model number information providing unit 112, and a communication unit 113.
The imaging unit 111 captures an image of the storage room of the refrigerator 110 as an captured image. For example, the imaging unit 111 can be configured by a camera. Then, the image pickup unit 111 provides the captured image, which is the captured image, to the cloud server 140 via the communication unit 113.

The refrigerator model number information providing unit 112 provides the model number information, which is the refrigerator identification information for identifying the refrigerator 110, to the cloud server 140 via the communication unit 113. The model number information is information indicating the model number of the refrigerator 110, and is used to specify the shape of the refrigerator 110. Therefore, the refrigerator identification information may be information other than the model number information as long as the shape of the refrigerator 110 can be uniquely identified. The refrigerator model number information providing unit 112 is provided with a model number information storage unit 112a for storing model number information. The model number information storage unit 112a can be configured by, for example, a non-volatile memory.
The communication unit 113 communicates via the communication line 150. For example, the communication unit 113 can be configured by a NIC (Network Interface Card).

A part or all of the refrigerator model number information providing unit 112 described above includes, for example, a memory 10 and a CPU that executes a program stored in the memory 10, as shown in FIG. 2 (A). It can be configured by a processor 11 such as Central Processing Unit). Such a program may be provided through a network, or may be recorded and provided on a recording medium.

Further, a part or all of the refrigerator model number information providing unit 112 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, or an ASIC, as shown in FIG. 2 (B). It can also be configured by a processing circuit 12 such as an Application Specific Integrated Circuits (Application Special Integrated Circuits) or an FPGA (Field Programmable Gate Array).

Returning to FIG. 1, the operation terminal 120 communicates with the display unit 121, the input unit 122, the purchase desired product designation unit 123, the storage posture selection unit 124, the storage side change unit 125, and the storage position designation unit 126. A unit 127 is provided.
The display unit 121 displays various images related to the storage status simulation system 100.
The input unit 122 receives an input of an instruction from the user.
The display unit 121 and the input unit 122 can be configured by, for example, a touch panel.

The purchase desired product designation unit 123 receives the designation of the product that the user wishes to purchase via the input unit 122 based on the product selection image displayed on the display unit 121. Since the product specified here is a storage schedule to be stored in the refrigerator 110, the purchase desired product designation unit 123 is a storage schedule designation unit that sets the selected product as a storage schedule through the product selection image. Functions as. The purchase desired product designation unit 123 notifies the store server of the designated purchase desired product via the communication unit 127.
The storage posture selection unit 124 selects a posture in which the user plans to store the product desired to be purchased in the refrigerator 110 via the input unit 122 based on the simulation image displayed on the display unit 121. The storage posture selection unit 124 notifies the cloud server 140 of the selected storage posture via the communication unit 127.

The storage side changing unit 125 changes the side on which the user plans to store the product desired to be purchased in the refrigerator 110 via the input unit 122 based on the simulation image displayed on the display unit 121. For example, the user can change whether the product to be purchased is stored on the right side of the refrigerator 110 or on the left side of the refrigerator 110. The storage side changing unit 125 notifies the cloud server 140 of the changed side via the communication unit 127.
Based on the simulation image displayed on the display unit 121, the storage position designating unit 126 receives the designation of the position where the user plans to store the product desired to be purchased in the refrigerator 110 via the input unit 122. The storage position designation unit 126 notifies the cloud server 140 of the designated storage position via the communication unit 127.
The communication unit 127 communicates via the communication line 150. For example, the communication unit 127 can be configured by a NIC.

A part or all of the purchase desired product designation unit 123, the storage posture selection unit 124, the storage side changing unit 125, and the storage position designation unit 126 described above are, for example, as shown in FIG. 2 (A). , The memory 10 and a processor 11 such as a CPU that executes a program stored in the memory 10 can be configured. Such a program may be provided through a network, or may be recorded and provided on a recording medium.

Further, a part or all of the purchase desired product designation unit 123, the storage posture selection unit 124, the storage side change unit 125, and the storage position designation unit 126 are, for example, single as shown in FIG. 2 (B). It can also be composed of a circuit, a composite circuit, a programmed processor, a parallel programmed processor, and a processing circuit 12 such as an ASIC or FPGA.

Returning to FIG. 1, the store server 130 includes a product selection image providing unit 131, a purchase desired product storage unit 132, a product shape information providing unit 133, a product suitable storage room information providing unit 134, and a communication unit 135. ..
The product selection image providing unit 131 causes the display unit 121 to display a product selection image for selecting a desired product from a plurality of products. For example, the product selection image providing unit 131 provides the operation terminal 120 with a product selection image for selecting a product from the products handled by the store server 130 via the communication unit 135. For example, the product selection image providing unit 131 includes a product image information storage unit 131a that stores product information related to products handled by the store server 130 and image information for generating a product selection image.
The purchase desired product storage unit 132 stores an article ID which is purchase desired product identification information for identifying the purchase desired product designated by the purchase desired product designation unit 123 of the operation terminal 120.

The product shape information providing unit 133 provides the cloud server 140 with shape information indicating the shape of the desired product to be purchased, which is designated by the desired product specifying unit 123 of the operation terminal 120, via the communication unit 135. For example, the product shape information providing unit 133 includes a product shape information storage unit 133a that stores product shape information indicating the shape of the product handled by the store server 130. Then, the product shape information providing unit 133 extracts information about the product desired to be purchased from the product shape information, and uses the extracted information as the shape information. Here, the product shape information indicates the outer shape and dimensions of the product, and, when a plurality of storage postures are set for the product, the outer shape and dimensions of the product in each storage posture. Specifically, the storage posture is a posture such as vertical placement and horizontal placement. Depending on the product, the liquid may spill or be damaged depending on the storage posture. Therefore, some products have one storage posture and some have multiple storage postures.

The product suitable storage room information providing unit 134 provides the cloud server 140 with storage room information suitable for the desired purchase product designated by the purchase desired product designation unit 123 of the operation terminal 120 via the communication unit 135. For example, the product suitable storage room information providing unit 134 includes a product suitable storage room information storage unit 134a that stores product suitable storage room information indicating the storage room of the refrigerator 110, which should store the products handled by the store server 130. ing. Then, the product suitable storage room information providing unit 134 extracts information about the product desired to be purchased from the product suitable storage room information, and uses the extracted information as the storage room information.

The communication unit 135 communicates via the communication line 150. For example, the communication unit 135 can be configured by a NIC.

A part or all of the product selection image providing unit 131, the product shape information providing unit 133, and the product suitable storage room information providing unit 134 described above are stored in memory, for example, as shown in FIG. 2 (A). It can be composed of 10 and a processor 11 such as a CPU that executes a program stored in the memory 10. Such a program may be provided through a network, or may be recorded and provided on a recording medium.

Further, a part or all of the product selection image providing unit 131, the product shape information providing unit 133, and the product suitable storage room information providing unit 134 are, for example, as shown in FIG. 2B, a single circuit. It can also be composed of a composite circuit, a programmed processor, a parallel programmed processor, and a processing circuit 12 such as an ASIC or FPGA.

The product image information storage unit 131a, the desired product storage unit 132, the product shape information storage unit 133a, and the product suitable storage room information storage unit 134a are composed of a storage device such as an HDD (Hard Disk Drive) or a non-volatile memory. be able to.

The cloud server 140 includes a refrigerator structure information providing unit 141, a storage availability determination unit 143, a simulation image generation unit 142, and a communication unit 144.
The refrigerator structure information providing unit 141 provides the storage possibility determination unit 143 and the simulation image generation unit 142 with information indicating the internal structure of the refrigerator 110 based on the model number information provided by the refrigerator model number information providing unit 112 of the refrigerator 110. To do. For example, the refrigerator structure information providing unit 141 includes a refrigerator structure information storage unit 141a that stores refrigerator structure information indicating the internal structures of various refrigerators 110. Then, the refrigerator structure information providing unit 141 extracts the information corresponding to the notified model number information from the refrigerator structure information, and notifies the extracted information.

Refrigerator structure information includes the shape, dimensions and position of the storage chamber, and structural information indicating the shape, dimensions and position of the structures constituting the storage chamber. More specifically, the structural information indicates the inner dimensions of the storage chamber of the refrigerator and the shape, dimensions and position of structures such as shelves provided in the storage chamber.
Further, the refrigerator structure information includes dimensional information indicating the dimensions of a predetermined portion of the refrigerator. The predetermined portion is either a storage chamber or a portion of a structure constituting the storage chamber. The portion is specified in advance for comparison with the dimensions in the captured image of the storage object to be described later.
Further, when the plurality of structures constituting the storage chamber include a moving structure whose position can be moved, the refrigerator structure information is based on the shape, size and movement of the moving structure. It further includes movable range information indicating the possible range. Here, the movable range is a range in which the position of the moving structure can be moved.
Further, when a removable structure is included in the plurality of structures constituting the storage chamber, the refrigerator structure information indicates the shape, size and position of the removable structure. Includes more information.

The simulation image generation unit 142 includes the shape information sent from the store server 130 in the captured image of the storage room indicated by the storage room information sent from the store server 130 among the captured images sent from the refrigerator 110. A simulation image is generated by synthesizing a product image (outer shape image) of a product having an outer shape and dimensions shown by.
For example, the simulation image generation unit 142 sets the purchase desired product selected by the user as a storage target. Then, the simulation image generation unit 142 compares the dimensions of the predetermined portion in the captured image with the dimensions indicated by the dimensional information included in the refrigerator structure information in the captured image of the storage object. The dimensions in the above are calculated, and a perspective image (first external image) which is an external image of the planned storage object is generated with the calculated dimensions. The simulation image generation unit 142 generates a simulation image (first simulation image) by synthesizing a perspective image at a predetermined position of the captured image.
Further, when the simulation image generation unit 142 includes a moving structure whose position can be moved in the structure constituting the refrigerator 110, the simulation image includes the moving structure. Indicates the movable range.

Then, the simulation image generation unit 142 gives the generated simulation image to the operation terminal 120 via the communication unit 144. The operation terminal that has received such a simulation image displays the image on the display unit 121.

When the storage posture selection unit 124 receives a storage posture selection instruction for storing the planned storage object via the simulation image displayed on the display unit 121, the simulation image generation unit 142 calculates as described above. A simulation image (second simulation image) by generating a perspective image (second outer shape image) of the object to be stored in the selected storage posture with the determined dimensions and synthesizing the generated perspective image with the captured image. Is generated (updated). The simulation image generated in this way is sent to the display unit 121 and displayed.

Further, when the storage side change unit 125 receives an instruction to change the side for storing the storage object in the refrigerator 110 via the simulation image displayed on the display unit 121, the simulation image generation unit 142 describes the above. With the dimensions calculated in this way, a perspective image of the object to be stored when it is stored in the changed side is generated, and a simulation image is generated (updated) by synthesizing the generated perspective image with the captured image. The simulation image generated in this way is sent to the display unit 121 and displayed.

Further, when the storage position designating unit 126 receives the designation of the storage position for storing the planned storage object via the simulation image displayed on the display unit 121, the simulation image generation unit 142 performs as described above. With the calculated dimensions, a perspective image (third external image) of the object to be stored when it is stored in the specified storage position is generated, and the generated perspective image is combined with the captured image at the specified storage position. By doing so, a simulation image (third simulation image) is generated (updated). The simulation image generated in this way is sent to the display unit 121 and displayed.

Further, the simulation image generation unit 142 updates the simulation image according to the determination result from the storage availability determination unit 143.

When the storage position is specified by the storage position designating unit 126 using the simulation image, the storage availability determination unit 143 receives the captured image sent from the refrigerator 110 and the information sent from the refrigerator structure information providing unit 141. Based on the above, it is determined whether or not the product having the outer shape and the dimensions indicated by the shape information sent from the store server 130 can be stored in the designated storage position.
For example, the storage availability determination unit 143 identifies the stored items already stored in the refrigerator 110 based on the captured image sent from the refrigerator 110, and the information sent from the refrigerator structure information providing unit 141. Whether or not the storage items already stored in the refrigerator 110 or the structures constituting the refrigerator 110 hinder the storage of the planned storage items when the planned storage items are stored in the designated storage position based on the above. To judge. Then, the storage availability determination unit 143 determines that the storage schedule cannot be stored when the storage of the storage schedule is hindered, and when the storage of the storage schedule is not hindered, the storage schedule is not hindered. Judge that it can be stored.

The storage availability determination unit 143 includes a determination information storage unit 143a that stores information necessary for performing the above determination.
Then, the storage availability determination unit 143 gives the determination result to the simulation image generation unit 142. In response to such a determination result, the simulation image generation unit 142 updates the simulation image when the storage of the planned storage object is hindered, and indicates the portion of the planned storage object that is hindering the storage.

If the structure constituting the refrigerator 110 includes a moving structure whose position can be moved, the storage availability determination unit 143 moves such a moving structure. Therefore, it is possible to judge whether or not the storage is possible.
For example, if the storage availability determination unit 143 refers to the movable range information and moves the moving structure so that the storage of the planned storage object is not hindered, the storage of the planned storage object is not hindered. The position of the moving structure is specified as the moving position. Then, the storage availability determination unit 143 notifies the simulation image generation unit 142 of the movement position.
Upon receiving such a notification, the simulation image generation unit 142 updates the simulation image to move the moving structure to the notified moving position, thereby indicating that the storage schedule can be stored.

Further, when the structure constituting the refrigerator 110 includes a removable structure, the storage availability determination unit 143 considers the case where such a removable structure is removed. Then, it is possible to judge whether or not the storage is possible.
For example, if the removable structure is removed from the detachable structure by referring to the detachable structure, the storage availability determination unit 143 does not interfere with the storage of the planned storage object. Is specified, and the detachable structure is notified to the simulation image generation unit 142.
Upon receiving such a notification, the simulation image generation unit 142 updates the simulation image to indicate that the scheduled storage object can be stored by removing the notified detachable structure.

The communication unit 144 communicates via the communication line 150. For example, the communication unit 144 can be configured by a NIC.

A part or all of the refrigerator structure information providing unit 141, the simulation image generation unit 142, and the storage availability determination unit 143 described above are, for example, a memory 10 and a memory, as shown in FIG. 2A. It can be configured by a processor 11 such as a CPU that executes a program stored in the 10. Such a program may be provided through a network, or may be recorded and provided on a recording medium.

Further, a part or all of the refrigerator structure information providing unit 141, the simulation image generation unit 142, and the storage availability determination unit 143 are, for example, as shown in FIG. 2B, a single circuit, a composite circuit, or a program. It can also be composed of a processed processor, a parallel programmed processor, and a processing circuit 12 such as an ASIC or FPGA.

The refrigerator structure information storage unit 141a and the determination information storage unit 143a can be configured by a storage device such as an HDD or a non-volatile memory.

FIG. 3 is a flowchart showing processing in the storage status simulation system 100.
First, the imaging unit 111 captures an image of the storage chamber of the refrigerator 110 (S10). When the refrigerator 110 is composed of a plurality of storage chambers such as a refrigerator compartment, a vegetable compartment, and a freezer compartment, an imaging unit 111 may be installed in each storage chamber. The imaging unit 111 transmits the captured image from the communication unit 113 to the simulation image generation unit 142 via the communication line 150.

Further, the refrigerator model number information providing unit 112 transmits the model number information stored in the model number information storage unit 112a from the communication unit 113 to the refrigerator structure information providing unit 141 via the communication line 150 (S11).

The refrigerator structure information providing unit 141 searches the refrigerator structure information stored in the refrigerator structure information storage unit 141a for information indicating the internal structure of the refrigerator 110 by using the model number indicated by the model number information (S12). Then, the refrigerator structure information providing unit 141 gives the information corresponding to the refrigerator 110 to the simulation image generation unit 142.

The product selection image providing unit 131 generates a product selection image for selecting a product from the products handled by the store server 130, and transmits the product selection image from the communication unit 113 to the operation terminal 120 via the communication line 150. To do. Then, the display unit 121 of the operation terminal 120 displays the product selection image (S13).

FIG. 4 is a schematic view showing an example of a product selection image.
The first product PD1, the second product PD2, and the third product PD3 are displayed on the product selection image IM1. All of these products are products handled by the store server 130.
The user can change the product displayed on the product selection image IM1 by inputting the execution instructions of the operation buttons B1 and B2 using the input unit 122.
Then, in the product selection image IM1, the user uses the input unit 122 to move the selection frame SL, specifies a desired product, and inputs a selection instruction (S14). As a result, the purchase desired product designation unit 123 transmits information indicating the selected position from the communication unit 113 to the product selection image providing unit 131 as purchase desired product information via the communication line 150. In response to such information, the product selection image providing unit 131 stores the article ID of the product desired to be purchased corresponding to the selected position in the product storage unit 132 desired to be purchased. The article ID may be any information that can uniquely identify the product handled by the store server 130, and the data format of the information does not specify a specific one.

Returning to FIG. 3, the product shape information providing unit 133 uses the product ID of the product desired to be purchased stored in the product storage unit 132 to be purchased from the product shape information stored in the product shape information storage unit 133a. Search for shape information of the product you want to purchase (S15). Then, the product shape information providing unit 133 transmits the shape information of the product desired to be purchased from the communication unit 135 to the simulation image generation unit 142 via the communication line 150.

Further, the product suitable storage room information providing unit 134 obtains the article ID of the desired product to be purchased stored in the desired product storage unit 132 from the product suitable storage room information stored in the product suitable storage room information storage unit 134a. It is used to search the storage room information of the product to be purchased (S16). Then, the product suitable storage room information providing unit 134 transmits the storage room information of the product desired to be purchased from the communication unit 135 to the simulation image generation unit 142 via the communication line 150.

Next, the simulation image generation unit 142 generates a simulation image (S17).
For example, the simulation image generation unit 142 determines a storage room in which the desired product to be purchased should be stored based on the storage room information transmitted from the product suitable storage room information providing unit 134. Then, the simulation image generation unit 142 is a ratio between the size of the storage room determined based on the dimensional information received from the refrigerator structure information providing unit 141 and the size of the storage room determined in the captured image. Calculate the dimensional ratio.

FIG. 5 is a schematic view showing an example of a captured image of the storage chamber.
The captured image IM2 includes a storage chamber 160 of the refrigerator 110, and the storage chamber 160 already stores the stored items 161 and 162.
For example, when the information received from the refrigerator structure information providing unit 141 includes the three-dimensional position information of the points P1 and P2, the simulation image generation unit 142 is the distance between the points P1 and P2. And the distance between the points P1 and P2 included in the captured image, the dimension ratio is calculated. That is, the simulation image generation unit 142 calculates the dimensional ratio between the actual size of the storage room and the size of the storage room in the captured image.

Next, the simulation image generation unit 142 is a perspective image that is a three-dimensional outer shape image of the desired product with the outer shape and dimensions indicated by the shape information transmitted from the product shape information providing unit 133 according to the calculated dimensional ratio. Create an image.
For example, the simulation image generation unit 142 corrects the actual dimensions of the product desired to be purchased indicated by the shape information to the dimensions in the captured image at the above-mentioned dimensional ratio, and the corrected dimensions have the shape indicated by the shape information. Create a perspective image.

6 (A) to 6 (D) are schematic views showing an example of a perspective image of a product desired to be purchased.
FIG. 6A is a perspective image when the product to be purchased is stored vertically with the left side facing the storage chamber as the storage position.
FIG. 6B is a perspective image when the product to be purchased is stored horizontally with the left side facing the storage chamber as the storage position.
FIG. 6C is a perspective image when the product to be purchased is stored horizontally with the right side facing the storage chamber as the storage position.
FIG. 6D is a perspective image when the product to be purchased is stored vertically with the right side facing the storage chamber as the storage position.
Here, the simulation image generation unit 142 may create one predetermined perspective image from the perspective images shown in FIGS. 6A to 6D.

Returning to FIG. 3, the simulation image generation unit 142 synthesizes the perspective image of the product desired to be purchased created as described above with the image captured in the storage room transmitted from the image pickup unit 111, so that the storage state in the storage room A simulation image showing the above is generated (S17). For example, in the simulation image generation unit 142, the purchase desired product is to be installed on the right side or the left side of the image by default, and the installation posture of the purchase function product is determined, and the perspective image corresponding to these is determined. Is synthesized.

FIG. 7 is a schematic view showing an example of a simulation image.
The simulation image IM3 shown in FIG. 7 is created by synthesizing the perspective image shown in FIG. 6B with the captured image IM2 shown in FIG.
Here, the operation button B3 shown in the simulation image IM3 is placed vertically in which the storage posture of the desired product PRD can be set vertically or horizontally when the storage posture can be set vertically or horizontally. It is a selection button.
Further, the operation button B4 is a horizontal selection button that selects the horizontal storage posture.
In the simulation image IM3 shown in FIG. 7, since the horizontal placement is selected, the perspective image of the desired product PRD drawn at the bottom is drawn in the horizontal placement.

Further, when the shelf which is one of the structures in the storage chamber of the refrigerator 110 is a moving shelf and the refrigerator structure information providing unit 141 notifies the movable range of the moving shelf, the simulation image generation unit 142 Displays the movable range of this moving shelf on the simulation image.
For example, assuming that the moving shelf SS can move from the position SP1 to the position SP2 as in the simulation image IM4 as shown in FIG. 8, the position SP1 to the position SP2 are shown in order to show such a movable range. It may be colored up to, and an arrow may be attached from the position SP1 to the position SP2.

Returning to FIG. 3, the generated simulation image is transmitted from the communication unit 135 to the operation terminal 120 via the communication line 150, and is displayed on the display unit 121 (S18).

When the user selects the storage posture of the product to be purchased via the input unit 122 in the simulation image displayed on the display unit 121 (Yes in S19), the process proceeds to step S20.
In step S20, the storage posture selection unit 124 notifies the simulation image generation unit 142 of the selected storage posture from the communication unit 135 via the communication line 150. The simulation image generation unit 142 updates the simulation image according to the notified storage posture, and transmits the updated simulation image from the communication unit 135 to the operation terminal 120 via the communication line 150. Then, the process returns to step S18.

As described above, FIG. 7 is a simulation image IM3 when the product PRD desired to be purchased is to be stored horizontally on the left side of the storage room. For example, in the simulation image IM3, when the execution instruction with the operation button B3 specified is input, the simulation image generation unit 142 generates the simulation image IM5 as shown in FIG. The simulation image IM5 is an image when the product PRD to be purchased is to be stored vertically on the left side of the storage room.

Returning to FIG. 3, when the user changes the storage side of the product desired to be purchased via the input unit 122 in the simulation image displayed on the display unit 121 (Yes in S21), the process is step S22. Proceed to.
In step S22, the storage side changing unit 125 notifies the simulation image generation unit 142 from the communication unit 135 via the communication line 150 of the changed side. The simulation image generation unit 142 updates the simulation image according to the notified side, and transmits the updated simulation image from the communication unit 135 to the operation terminal 120 via the communication line 150. Then, the process returns to step S18.

FIG. 7 is a simulation image IM3 when the desired product PRD for purchase is to be stored horizontally on the left side of the storage room as described above. For example, in the simulation image IM3, when the storage side is changed to the right side of the storage chamber as in the case where the operation of shifting the purchase desired product PRD to the right side is input to the input unit 122, the simulation image generation unit Like the simulation image IM6 shown in FIG. 10, 142 generates a perspective image of the product PRD to be purchased when it is intended to be stored horizontally on the right side of the storage chamber, and captures the image. Combine with image.

Further, in the simulation image IM5 shown in FIG. 9, when the storage side is changed to the right side of the storage chamber, the simulation image generation unit 142 is like the simulation image IM7 shown in FIG. , A perspective image of the product PRD desired to be purchased when the product is to be stored vertically is generated on the right side of the storage room, and the image is combined with the captured image.

Returning to FIG. 3, when the user specifies the storage position of the product to be purchased via the input unit 122 in the simulation image displayed on the display unit 121 (Yes in S23), the process is step S24. Proceed to.
In step S24, the storage position designation unit 126 notifies the simulation image generation unit 142 of the designated storage position from the communication unit 135 via the communication line 150. The simulation image generation unit 142 updates the simulation image according to the notified storage position, and transmits the updated simulation image from the communication unit 135 to the operation terminal 120 via the communication line 150. Then, the process returns to step S18.

For example, in the simulation image IM3 shown in FIG. 7, when the storage position is changed to the storage room, such as when the operation of shifting the desired product PRD onto the shelf SEL is input to the input unit 122. The storage availability determination unit 143 determines whether or not the desired product can be stored in the storage position in the designated storage room.

Specifically, first, the storage availability determination unit 143 estimates the range of the stored items already stored in the refrigerator 110 from the captured image sent from the refrigerator 110. For example, the storage availability determination unit 143 can perform such an estimation by storing the captured image of the storage room before the object is stored in the refrigerator 110 in the determination information storage unit 143a. Further, for example, by installing a pressure sensor on the shelf of the refrigerator 110 and receiving the detected value, the storage availability determination unit 143 can perform such an estimation.

Next, the storage availability determination unit 143 calculates the range of the desired product to be purchased when the product is stored in the storage position designated by the storage position designation unit 126 from the shape information provided by the product shape information providing unit 133.
Next, the storage availability determination unit 143 includes the range of the structure indicated by the structural information of the storage room of the refrigerator 110 provided by the refrigerator structure information providing unit 141, the estimated range of the stored items, and the calculated purchase desired product. It is determined whether or not the ranges of are overlapped with each other. If they do not overlap, the storage availability determination unit 143 determines that the storage of the desired product is not hindered and that the storage is possible. If they overlap, the storage of the desired product is hindered, and it is judged that the product cannot be stored.

The storage availability determination unit 143 gives the determination result of the storage availability determination to the simulation image generation unit 142. Here, when the determination result cannot be stored, the storage availability determination unit 143 also gives the simulation image generation unit 142 interference range information indicating the range of the portion corresponding to the desired product to be purchased within the overlapping range.
When the determination result can be stored, the simulation image generation unit 142 updates the simulation image by synthesizing the perspective image of the product to be purchased at the designated storage position in the selected storage posture.
On the other hand, when the determination result cannot be stored, the simulation image generation unit 142 synthesizes a perspective image of the product to be purchased at the designated storage position in the selected storage posture, and an image showing the interference range. The simulation image is updated by synthesizing. For example, the simulation image generation unit 142 identifies a portion that interferes with the storage by contacting the stored object and the structure when the product to be purchased is stored in the storage position from the range indicated by the interference range information. An image with that portion as the interference range is generated and combined with the captured image.

FIG. 12 is a schematic view showing a simulation image when the determination result cannot be stored.
In the simulation image IM8 shown in FIG. 12, the product PRD to be purchased is stored in the designated storage position in the selected vertical position.
However, in the simulation image IM8, the range of the shape range of the desired product PRD that overlaps with the moving shelf SS at the position SP2 is shown as the interference range ITA. For example, the simulation image generation unit 142 can show such a jamming range ITA by synthesizing an image showing such a jamming range ITA.

The simulation image generation unit 142 causes the display unit 121 to display the generated simulation image IM8.
A user who confirms the simulation image IM8 as shown in FIG. 12 on the display unit 121 can easily recognize that even if he / she purchases the desired product PRD, he / she cannot store it in the refrigerator 110.

In addition, the storage availability determination unit 143 determines the state of the storage room of the refrigerator 110, such as when the shelf obstructing the storage of the product to be purchased is a movable movable shelf or a removable removable shelf in the determination of storage availability. If it is possible to change, the state of the storage room can be changed to make a judgment.

For example, when the shelf obstructing the storage of the desired product PRD is a moving shelf SS as shown in FIG. 8, the moving shelf SS installed at the position SP2 may be moved to the position SP1. it can.
Therefore, the storage availability determination unit 143 assumes that the moving shelf SS has moved to the position SP1 based on the movable range information of the moving shelf included in the structural information provided by the refrigerator structure information providing unit 141. , Judge whether storage is possible. Here, when the storage availability determination results do not overlap and can be stored, the storage availability determination unit 143 simulates the storage availability determination result and the conditions for enabling storage 142. Give to. The condition for allowing storage is, here, to move the moving shelf SS to the position SP1.

FIG. 13 is a schematic view showing an example of a simulation image when conditions for enabling storage are shown.
In this case, since there is no interference range, the simulation image IM9 does not show the interference range. However, in the simulation image IM9, the message column MEA1 indicating the conditions for allowing storage is displayed.
The message field MEA1 shown in FIG. 13 notifies the user that it is a simulation image assuming that the moving shelf SS has been moved to the position SP1.
The simulation image generation unit 142 causes the simulation image IM9 to be displayed on the display unit 121.
The simulation image generation unit 142 processes the captured image so as to move the moving shelf SS to the position SP1 to generate the simulation image IM9 shown in FIG. 13. However, such processing is performed. It does not have to be.

The user who confirmed the simulation image IM9 as shown in FIG. 13 on the display unit 121 can store the desired product PRD in the refrigerator 110 by moving the moving shelf SS to the position SP1. Can be easily recognized.

Further, when the shelf obstructing the storage of the desired product PRD is a removable removable shelf, such a shelf can be removed.
Therefore, the storage availability determination unit 143 determines the storage availability when it is assumed that the detachable shelf is removed based on the attachment / detachment information of the detachable shelf included in the structural information provided by the refrigerator structure information providing unit 141. .. Here, when the storage availability determination results do not overlap and can be stored, the storage availability determination unit 143 simulates the storage availability determination result and the conditions for enabling storage 142. Give to. The condition for the storage is to remove the removable shelf here.

FIG. 14 is a schematic view showing an example of a simulation image when conditions for enabling storage are shown.
In this case, since there is no interference range, the simulation image IM10 does not show the interference range. However, in the simulation image IM10, the message column MEA2 indicating the conditions for allowing storage is displayed.
The message field MEA2 shown in FIG. 14 notifies the user that it is a simulation image assuming that the removable shelf is removed.
The simulation image generation unit 142 causes the display unit 121 to display the simulation image IM10.

The user who confirmed the simulation image IM10 as shown in FIG. 14 on the display unit 121 can easily recognize that the product PRD desired to be purchased can be stored in the refrigerator 110 by removing the removable shelf. ..
The simulation image generation unit 142 processes the captured image so as to erase the detachable shelf to generate the simulation image IM10 shown in FIG. 14, but such processing does not have to be performed. ..

According to the embodiment described above, the captured image in the storage chamber of the refrigerator 110 can be compared with the perspective image of the storage object having the same dimensional ratio as the dimensional ratio of the captured image, so that the user can store the image. It is possible to know exactly whether or not the planned object can be stored in the storage room of the refrigerator 110. For this reason, a situation may occur in which the planned storage item is purchased but cannot be stored in the refrigerator 110, or a situation may occur in which the planned storage item may not be stored in the refrigerator 110 and the purchase opportunity may be missed. Can be prevented.

According to the embodiment described above, the structural information of the storage chamber of the refrigerator 110 can be obtained by searching from the model number information of the refrigerator 110, so that the structural information of the storage chamber of the refrigerator 110 can be obtained from the system. It is possible to save the trouble of inputting by the user and to use accurate structural information in the storage room.

According to the embodiment described above, the shape information of the planned storage object can be searched and obtained from the article ID of the planned storage object, so that the user of the system can obtain the shape information of the planned storage object. It is possible to save the trouble of inputting and to use accurate shape information of the planned storage object.

According to the embodiment described above, the storage room information suitable for the planned storage object can be searched and obtained from the article ID of the planned storage object, so that the storage room information suitable for the planned storage object can be obtained. , It is possible to save the trouble of inputting by the system user. In addition, it is possible to prevent the risk of making a mistake in determining whether or not the planned storage object can be stored by comparing the captured image of the storage room other than the storage room suitable for the planned storage object with the perspective image of the planned storage object. ..

According to the embodiment described above, it is possible to compare the captured image in the storage room with the perspective images of a plurality of storage postures preset in the storage object, so that the images can be stored in a plurality of storage postures. You can judge whether or not. As a result, it is possible to try to store the product in a certain storage posture, but to store the product in another storage posture, and it is possible to prevent the risk of determining that the product cannot be stored even though it can actually be stored.

According to the embodiment described above, the position of the stored items already stored in the refrigerator 110 is estimated from the captured image in the storage room, and the already stored items and the planned storage items do not collide with each other and are stored. In order to determine whether or not it is possible, the user of the system determines whether or not the storage is possible by simply comparing the captured image of the storage room other than the storage room suitable for the planned storage object with the perspective image of the planned storage object. It is easier and more accurate to determine whether or not storage is possible.

According to the embodiment described above, the system is used to determine whether or not the structure and the planned storage object do not collide with each other and can be stored from the position and shape information of the structure in the storage chamber. It is easier and more accurate than a person to judge whether or not storage is possible only by comparing the captured image of the storage room other than the storage room suitable for the planned storage object with the perspective image of the planned storage object. I can judge.

According to the embodiment described above, when the position of the structure in the storage chamber can be changed, the structure when the structure is installed at a position where the structure can be taken does not collide with the planned storage object. In order to determine whether or not the storage is possible, the user of the system does not judge whether or not the storage is possible by simply comparing the captured image of the storage room other than the storage room suitable for the planned storage object with the perspective image of the planned storage object. , It is possible to judge whether storage is possible more easily and accurately.

<First modification>
By the way, as described at the beginning, the refrigerator 110 may include the function of the cloud server 140. Further, the refrigerator 110 may include the function of the operation terminal 120.
FIG. 15 is a block diagram schematically showing the configuration of the refrigerator 210 in the first modification and the storage situation simulation system 200 including the refrigerator 210.
The refrigerator 210 incorporates the functions of the cloud server 140.
The refrigerator 210 includes an imaging unit 111, a refrigerator model number information providing unit 112, a communication unit 113, a refrigerator structure information providing unit 141 having a refrigerator structure information storage unit 141a, and a storage availability determination unit 143.

The imaging unit 111 captures an image of the inside of the refrigerator 210 as an captured image.
The refrigerator model number information providing unit 112 gives the model number information stored in the model number information storage unit 112a to the refrigerator structure information providing unit 141.
The communication unit 113 is connected to the communication line 150 and receives the shape information of the object to be stored in the refrigerator.

The refrigerator structure information storage unit 141a stores refrigerator structure information including dimensional information indicating the dimensions of a predetermined portion of the refrigerator.
The storage availability determination unit 143 calculates the dimensions of the object to be stored in the refrigerator in the captured image by comparing the dimensions of the predetermined portion in the captured image with the dimensions indicated by the dimensional information. Determine whether or not the planned storage object can be stored. The determination of whether or not the planned storage object can be stored has already been described.
The refrigerator structure information storage unit 141a does not have to be provided in the refrigerator 210, and the communication unit 113 receives dimensional information indicating the dimensions of a predetermined portion of the refrigerator 210 via the communication line 150. You may.

Further, the refrigerator 210 may include a simulation image generation unit 142. The simulation image generation unit 142 generates a first product image (outer shape image) of the object to be stored based on the dimensions in the captured image calculated by the storage availability determination unit 143, and synthesizes this product image with the captured image. Generates a simulation image (first simulation image) with. The communication unit 113 transmits the simulation image to the operation terminal 120 via the communication line 150. The operation terminal 120 includes a display unit 121. The received simulation image is displayed on the display unit 121.
The simulation image generation unit 142 may not be provided in the refrigerator 210, but may be provided in the operation terminal 120. In this case, the communication unit 113 transmits the dimensions in the captured image calculated by the storage availability determination unit 143 to the operation terminal 120 via the communication line 150.

<Second modification>
Further, as described at the beginning, the refrigerator 110 may include the function of the operation terminal 120.
FIG. 16 is a block diagram schematically showing the configuration of the refrigerator 310 in the second modification and the storage situation simulation system 300 including the refrigerator 310.
The refrigerator 310 incorporates the functions of the operation terminal 120.
In this case, the storage availability determination unit 143 provided in the cloud server 140 determines whether or not the storage schedule can be stored. Then, the simulation image generation unit 142 generates the first product image (outer shape image) of the storage object based on the dimensions in the captured image calculated by the storage availability determination unit 143, and synthesizes this product image with the captured image. By doing so, a simulation image (first simulation image) is generated. The communication unit 113 receives the simulation image via the communication line 150. The display unit 121 displays the received simulation image.

<Third variant>
Further, the refrigerator 110 may include the functions of the cloud server 140 and the operation terminal 120. FIG. 17 is a block diagram schematically showing the configuration of the refrigerator 410 in the third modification and the storage situation simulation system 400 including the refrigerator 410.
In this case, inside the refrigerator 410, it is determined whether or not the items to be stored can be stored, a simulation image is generated, and the simulation image is displayed. The communication unit 113 communicates with the store server 130 via the communication line 150, and receives the goods information of the store where the user purchases the product.

100,200,300,400 Storage status simulation system, 110,210,310,410 Refrigerator, 111 Imaging unit, 112 Refrigerator model number information provider, 113 Communication unit, 120 Operation terminal, 121 Display unit, 122 Input unit, 123 Purchase Desired product designation unit, 124 storage posture selection unit, 125 storage side change unit, 126 storage position designation unit, 127 communication unit, 130 store server, 131 product selection image provision unit, 132 product selection image provision unit, 132 product shape storage unit, 133 product shape information provision Unit, 134 Product suitable storage room information providing unit, 135 Communication unit, 140 Cloud server, 141 Refrigerator structure information providing unit, 143 Storage availability judgment unit, 142 Simulation image generation unit, 144 Communication unit, 10 memory, 11 Processor, 12 processing circuit.

Claims (15)

It is a simulation system for determining whether or not the items to be stored can be stored in the refrigerator.
An imaging unit that captures an image in the refrigerator as an imaging image,
A refrigerator structure information storage unit that stores refrigerator structure information including dimensional information indicating the dimensions of a predetermined portion in the refrigerator, and a refrigerator structure information storage unit.
A display unit that displays images and
A product selection image providing unit that displays a product selection image for selecting a desired product from a plurality of products on the display unit, and a product selection image providing unit.
Through the product selection image displayed on the display unit, a storage schedule designation unit that sets the selected product as the storage schedule, and a storage schedule designation unit.
A first external image of the planned storage object is generated based on the dimensions in the captured image of the planned storage object, and the first simulation image is generated by synthesizing the first external shape image with the captured image. Simulation image generator and
A product shape information providing unit that provides shape information indicating the outer shape and dimensions of the selected product to the simulation image generation unit, and a product shape information providing unit.
By comparing the dimensions of the predetermined portion in the captured image, the dimensions of the predetermined portion indicated by the dimension information, and the outer diameter and the dimensions indicated by the shape information, the storage is performed. It is provided with a storage availability determination unit that calculates the dimensions of the planned object in the captured image and determines whether or not the planned storage object can be stored.
The simulation image generation unit generates the first outer shape image based on the shape information.
The display unit is a simulation system characterized by displaying the first simulation image. Further, a storage posture selection unit for receiving a storage posture selection instruction for storing the storage object is further provided through the first simulation image displayed on the display unit.
The simulation image generation unit generates a second outer shape image of the planned storage object in the selected storage posture with the calculated dimensions, and synthesizes the second outer shape image with the captured image. Generate a simulation image of 2 and
The simulation system according to claim 1, wherein the display unit displays the second simulation image. Further, a storage position designation unit for receiving a designation of a storage position for storing the planned storage object is further provided through the first simulation image displayed on the display unit.
The simulation image generation unit generates a third outer image of the planned storage object when it is stored in the designated storage position with the calculated dimensions, and stores the third outer image in the designated storage. A third simulation image is generated by combining with the captured image at the position.
The simulation system according to claim 1, wherein the display unit displays the third simulation image. Further, a storage position designation unit for receiving a designation of a storage position for storing the planned storage object is further provided through the second simulation image displayed on the display unit.
The simulation image generation unit generates a third outer image of the planned storage object when it is stored in the designated storage position with the calculated dimensions, and stores the third outer image in the designated storage. A third simulation image is generated by combining with the captured image at the position.
The simulation system according to claim 2, wherein the display unit displays the third simulation image. The refrigerator structure information further includes structural information indicating the shape, size, and position of a plurality of structures provided in the refrigerator.
The storage possibility determination unit specifies the stored items already stored in the refrigerator from the captured image, and refers to the structural information to store the stored items in the storage position. It is determined whether or not the stored object or the plurality of structures hinders the storage of the planned storage object.
3. When the storage of the planned storage object is hindered, the simulation image generation unit indicates, in the third simulation image, a portion of the planned storage object that hinders the storage. The simulation system according to 4. The plurality of structures include a moving structure whose position can be moved.
The refrigerator structure information further includes movable range information indicating the shape, dimensions, and movable range of the moving structure.
When the movable structure is moved by referring to the movable range information and the storage of the planned storage object is not hindered, the storage availability determination unit is hindered from storing the planned storage object. The position of the moving structure that disappears is specified as the moving position,
The simulation image generation unit is characterized in that, in the third simulation image, by moving the moving structure to the moving position, it is shown whether or not the storage object can be stored. Item 5. The simulation system according to item 5. The plurality of structures include removable removable structures.
The refrigerator structure information further includes attachment / detachment information indicating the shape, size and position of the attachment / detachment structure.
The storage possibility determination unit refers to the attachment / detachment information, and when the attachment / detachment structure is removed so that the storage of the planned storage object is not hindered, the storage of the planned storage object is not hindered. Identify the detachable structure,
5. The simulation image generation unit is characterized in that, in the third simulation image, it indicates whether or not the storage object can be stored by removing the specified detachable structure. Simulation system described in. The plurality of structures include a moving structure whose position can be moved.
The refrigerator structure information further includes movable range information indicating the shape, dimensions, and movable range of the moving structure.
The simulation system according to claim 5, wherein the simulation image generation unit indicates the movable range on the first simulation image. The refrigerator is provided with a plurality of storage rooms.
The image pickup unit captures images of the plurality of storage chambers and captures images.
Further provided with a product suitable storage room information providing unit that provides the simulation image generation unit with storage room information indicating a storage room for storing the selected product.
The simulation system according to any one of claims 1 to 8, wherein the simulation image generation unit synthesizes the first external image with an image captured in the storage room indicated by the storage room information. It is a simulation method for determining whether or not the items to be stored can be stored in the refrigerator.
The image in the refrigerator is captured as an image to be captured.
By comparing the dimensions of the predetermined portion in the refrigerator with the dimensions of the predetermined portion in the captured image and the outer shape and dimensions of the planned storage object, the imaging of the planned storage object is performed. Calculate the dimensions in the image
A first external image of the planned storage object is generated with the calculated dimensions,
Simulation method characterized by determining whether said first contour image to generate a first simulated image by synthesizing the captured image, it is possible to accommodate the storage schedule thereof. An imaging unit that captures the image inside the refrigerator as an captured image,
A communication unit that is connected to an external network and receives shape information indicating the shape of the planned storage object to be stored in the refrigerator.
A refrigerator structure information storage unit that stores refrigerator structure information including dimensional information indicating the dimensions of a predetermined portion of the refrigerator, and a refrigerator structure information storage unit.
By comparing the dimensions of the predetermined portion in the captured image with the dimensions of the predetermined portion indicated by the dimensional information, the imaging of the storage object is performed based on the shape information. A refrigerator characterized by including a storage availability determination unit that calculates dimensions in an image and determines whether or not the planned storage object can be stored. An imaging unit that captures the image inside the refrigerator as an captured image,
A communication unit that is connected to an external network and receives shape information indicating the shape of an object to be stored in the refrigerator and refrigerator structure information including dimensional information indicating the dimensions of a predetermined portion of the refrigerator.
By comparing the dimensions of the predetermined portion in the captured image with the dimensions of the predetermined portion indicated by the dimensional information, the imaging of the storage object is performed based on the shape information. A refrigerator provided with a storage availability determination unit that calculates dimensions in an image and determines whether or not the storage object can be stored. The refrigerator according to claim 11 or 12, wherein the communication unit transmits the dimensions of the storage object in the captured image to the external network. A first external image of the planned storage object is generated based on the dimensions in the captured image of the planned storage object, and the first simulation image is generated by synthesizing the first external shape image with the captured image. Simulation image generator and
The refrigerator according to claim 11 or 12, further comprising a display unit for displaying the first simulation image. It is a simulation system for determining whether or not the items to be stored can be stored in the refrigerator.
A communication unit that acquires shape information indicating the outer shape and dimensions of the planned storage object,
An imaging unit that captures an image in the refrigerator as an imaging image,
A refrigerator structure information storage unit that stores refrigerator structure information including dimensional information indicating the dimensions of a predetermined portion in the refrigerator, and a refrigerator structure information storage unit.
By comparing the dimensions of the predetermined portion in the captured image, the dimensions of the predetermined portion indicated by the dimension information, and the outer shape and dimensions indicated by the shape information, the storage schedule The first outer shape image of the object to be stored is generated by calculating the dimensions of the object in the captured image, and the first outer shape image is combined with the captured image by combining the first outer shape image with the captured image. It is provided with a storage availability determination unit that generates a simulation image and determines whether or not the storage schedule can be stored.
A simulation system featuring.

Images