WO2023162312A1

WO2023162312A1 - Image management system, image management server, image management method, and method for producing image management system

Info

Publication number: WO2023162312A1
Application number: PCT/JP2022/034931
Authority: WO
Inventors: 慎太郎草野; 亮長谷川; 敦子船山; 勝青山
Original assignee: 日立グローバルライフソリューションズ株式会社
Priority date: 2022-02-22
Filing date: 2022-09-20
Publication date: 2023-08-31
Also published as: JP2023122170A; CN116680424A

Abstract

The present invantion hides a customer and the living environment of the customer when viewing an image of the inside of a customer's refrigerator. An image management system comprises a refrigerator (1) including a cold room (116) for storing food products and a camera unit (10) for capturing an image of a product range including the cold room (116), a server (2a) that acquires an unprocessed image captured by the camera unit (10), a storage server (6a) for storing the unprocessed image, a server (2b) that performs masking or mosaic processing on the unprocessed image to create a processed image, a storage server (6b) for storing the processed image, and a deletion means that deletes the unprocessed image at least before the processed image is deleted.

Description

Image management system, image management server, image management method, and method of producing image management system

The present invention relates to an image management system, an image management server, an image management method, and a method of producing an image management system.

In recent years, a service has been provided that allows users to view the inside of a refrigerator on a mobile device by uploading an image of the inside of the refrigerator to a server and processing the uploaded image. Patent Literature 1 describes an invention capable of photographing many places with a small number of cameras.

JP-A-2003-42626

　The images taken inside the customer's refrigerator must be handled with care from the perspective of personal information protection. This is because there is a risk that the customer or the customer's living environment may be unintentionally seen by others.

In order to solve the above-described problems, an image management system of the present invention provides a storage that includes a storage area for storing articles, an imaging unit that images an article range including the storage area, and an image that is captured by the imaging unit. pre-processing image acquisition means for acquiring a pre-processing image; first storage means for storing the pre-processing image; post-processing image creation means for creating a post-processing image by executing a process to lower it; second storage means for storing the post-processing image; and deletion means for deleting the pre-processing image after the post-processing image is created. And prepare.

The image management server of the present invention includes pre-processing image acquiring means for acquiring a pre-processing image obtained by capturing an article range including a storage area for storing the article by an imaging unit; first storage means for storing the pre-processing image; post-processing image creation means for creating a post-processing image by deleting at least a part of the range other than the article range or reducing information amount from the pre-processing image; a second storage means for storing; a deletion means for deleting the unprocessed image at least before the processed image is deleted;
characterized by comprising

According to the present invention, there is provided a method for producing an image management system, comprising: a storage including a storage area for storing articles; an imaging unit for imaging a range of articles including the storage area; a first storage means for storing the pre-processed image; and processing to delete at least a part of the pre-processed image other than the article range or to reduce the amount of information. a post-processing image creating means for creating a post-processing image, a second storage means for storing the post-processing image, a deleting means for deleting the pre-processing image after the post-processing image is created, A method for producing an image management system comprising: providing a control program for controlling the imaging unit to the storage or the terminal used by the user.

Other means will be explained in the form for carrying out the invention.

It is a block diagram of a refrigerator concerning this embodiment. Figure 2 is a block diagram of two types of servers; 1 is a block diagram of a mobile terminal; FIG. 1 is a logical block diagram of an image management system; FIG. It is a front view of a refrigerator. It is a side view of a refrigerator. It is a front view in the state where the right-and-left door of the refrigerator compartment of a refrigerator was opened. FIG. 4 is a plan view of the refrigerator with left and right doors of the refrigerating compartment opened. It is a perspective view at the time of looking up at the camera unit of a refrigerator from diagonally below. FIG. 10 is a sequence diagram of the first image processing; FIG. 4 is a diagram showing an unprocessed image taken by a camera of a refrigerator; FIG. 10 is a diagram showing an unprocessed image in which unnecessary portions are masked; It is a figure which shows an example of a post-processing image. It is a figure which shows the selection screen displayed on a portable terminal. It is a figure which shows a warning screen. It is a figure which shows a registration continuation user selection screen. FIG. 11 shows a confirmation dialog box; It is a figure which shows a caution screen at the time of transfer. It is a figure which shows a caution screen at the time of transfer. FIG. 10 is a diagram showing an image utilization warning screen; It is a figure which shows a home screen. It is a figure which shows the contents screen of a refrigerator. FIG. 11 is a sequence diagram of image processing according to the second embodiment;

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to each drawing. In the image management system of this embodiment, the refrigerator 1 shown in FIG. 1, the

servers

2a and 2b and the

storage servers

6a and 6b shown in FIG. 2, and the portable terminal 3 shown in FIG. do. Different IP addresses are assigned to the refrigerator 1, the

servers

2a and 2b, the

storage servers

6a and 6b, and the portable terminal 3, respectively.

The configuration of the refrigerator 1 will be described with reference to FIG.
The refrigerator 1 has a hardware configuration including a camera unit 10, an arithmetic unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, a communication unit 115, and a camera LED (Light Emitting Diode) 100. have Refrigerator 1 further includes left door 12, right door 13, door sensor 117, and refrigerator compartment 116 shown in FIG.

The calculation unit 112 is, for example, a CPU (Central Processing Unit) and controls the refrigerator 1 in an integrated manner. The RAM 114 serves as a work area for the computing section 112 . The RAM 114 is a non-temporary or temporary recording medium for storing various programs and data, and stores programs (not shown).

The camera unit 10 is an imaging unit that takes an image of the refrigerator compartment 116, such as a fisheye camera or a panorama camera. Camera unit 10 is attached to the upper side of refrigerator compartment 116 . The camera LED 100 is an indicator that lights up when the camera unit 10 is imaging. The left door 12 and the right door 13 are doors provided in the refrigerating compartment 116 , and their open/closed states are detected by a door sensor 117 . When the user opens the left door 12 and/or the right door 13 , the camera unit 10 operates to capture an image of the refrigerator compartment 116 . Refrigerator compartment 116 is a container for storing articles, typically foods.

The communication unit 115 connects to a network and transmits and receives data to and from the server 2a, which is an external device.

The configurations of the

servers

2a and 2b and the

storage servers

6a and 6b will be described with reference to FIG. The

servers

2a and 2b and the

storage servers

6a and 6b each have a CPU 21, a ROM 22, a RAM 23, a storage section 24, and a communication section 25.

The CPU 21 comprehensively controls the

servers

2a and 2b and the

storage servers

6a and 6b, respectively. The ROM 22 is a non-volatile memory, and stores BIOS (Basic I/O System), for example. The RAM 23 serves as a work area for the CPU 21 . The RAM 23 is a non-temporary or temporary recording medium for storing various programs and data, and stores programs (not shown).

The storage unit 24 is a non-temporary or temporary recording medium that stores various programs and data, and is composed of, for example, a HDD (Hard Disk Drive), flash memory, and the like. It is desirable that the storage units 24 of the

storage servers

6a and 6b have a large capacity and can be accessed at high speed. The communication unit 25 connects to a network and transmits/receives data to/from an external device.

The configuration of the mobile terminal 3 will be described with reference to FIG. Each mobile terminal 3 has a CPU 31 , a ROM 32 , a RAM 33 , a storage section 34 , a communication section 35 and a touch panel display 36 .

The CPU 31 comprehensively controls the mobile terminal 3. The ROM 32 is a non-volatile memory and stores BIOS, for example. The RAM 33 serves as a work area for the CPU 31 . The RAM 33 is a non-temporary or temporary recording medium for storing various programs and data, and stores programs (not shown).

The storage unit 34 is a non-temporary or temporary recording medium for storing various programs and data, and is composed of, for example, flash memory. The communication unit 35 connects to a network and transmits/receives data to/from an external device. The touch panel display 36 is configured by stacking a touch panel on a planar display. The touch panel display 36 is display means for displaying characters, images, or graphics, and is also input means for detecting user operations.

FIG. 4 is a logical block diagram of an image management system.
The unprocessed image acquiring means 81 acquires the unprocessed image captured by the camera unit 10 . The unprocessed image acquiring means 81 is embodied by executing a program by the CPU 21 of the server 2a.
The first storage means 82 stores the unprocessed image. This first storage means 82 corresponds to the storage server 6a.

The post-processing image creation means 84 creates a post-processing image by deleting at least a part of the range other than the article range or reducing the amount of information on the pre-processing image. Article area refers to the area of the image in which, for example, the refrigerator 116, or other storage room, is taken. The post-processing image creating means 84 creates a post-processing image by deleting or reducing the amount of information including a predetermined range from the pre-processing image. Then, the post-processing image creating means 84 creates a post-processing image by deleting or reducing the amount of information including the body part of a person when a body part of a person is reflected in any of the pre-processing images. do. The post-processing image creating means 84 is embodied by executing a program by the CPU 21 of the server 2b.

The deleting means 83 deletes the pre-processed image after creating the post-processed image. The deleting means 83 deletes the unprocessed image at least before the processed image is deleted. The deletion means is embodied by the CPU 21 of the storage server 6a executing the program.

The second storage means 85 stores the processed image. This second storage means 85 corresponds to the storage server 6a.
The registration unit 86 registers a user who can view the processed image by linking it to the individual number of the storage or the imaging unit. The registered user information presenting means 87 presents the information on the already registered user to the user when newly registering the user to be linked to the individual number.

When registering a new user associated with this individual number, the registered user authority deletion means 88 causes this user to select an already registered user and deletes the viewing authority of the selected registered user. The registering means 86, the registered user information presenting means 87, and the registered user authority deleting means 88 are embodied by the CPU 31 of the mobile terminal 3 executing programs.

FIG. 5 is a front view of the refrigerator 1. FIG.
The refrigerator 1 is a device that cools food and the like, and includes a left door 12 and a right door 13 of a refrigerator compartment 116 and a camera unit 10 . The refrigerator 1 has a plurality of storage compartments inside its housing. As storage compartments of the refrigerator 1, a refrigerating compartment 116, an ice making compartment 14 and an upper freezing compartment 15 arranged side by side, a vegetable compartment 16, and a lower freezing compartment 17 are provided in this order from the top.

The housing of the refrigerator 1 has a configuration in which a heat insulating material (not shown) such as urethane foam is filled between an outer box made of steel plate and an inner box made of resin (not shown). A plurality of openings corresponding to the respective compartments are provided on the front side (front side) of the housing of the refrigerator 1 . For example, when food or the like is put through the opening of the refrigerator compartment 116, the left door 12 and the right door 13 are opened. Further, when the left door 12 and the right door 13 are closed, the opening of the refrigerator compartment 116 is closed. Thus, the left door 12 and the right door 13 have functions such as closing the opening of the refrigerator compartment 116 . The same applies to other doors.

The refrigerator 1 has a left door 12 and a right door 13 as French doors that form a refrigerator compartment 116 together with the housing. The left door 12 is rotatable around the axis of the hinge 121 (see FIG. 8) at the left end. The same applies to the right door 13 as well. In addition, the refrigerator 1 includes a vegetable compartment door and a lower freezer compartment door as well as an ice making compartment door and an upper freezer compartment door as drawer doors.

The refrigerator compartment 116 is provided with a plurality of shelf boards that partition the refrigerator compartment 116 in a predetermined manner. A plurality of door pockets 123 (see FIG. 7) for storing foods and the like are provided inside the left door 12 . The right door 13 is also the same. The ice making chamber 14 is provided with an ice making chamber container (not shown) that is pulled out integrally with the ice making chamber door. Similarly, the upper freezer compartment 15 is provided with an upper freezer compartment container (not shown), and the vegetable compartment 16 is provided with a vegetable compartment container (not shown). A lower freezer compartment container (not shown) is provided.

Although not shown, the refrigerator 1 includes a compressor, a radiator (condenser), a capillary tube (throttling mechanism), and an evaporator. The refrigerant circulates sequentially through the compressor, radiator, capillary tube, and evaporator, and the air in the storage room is cooled by heat exchange with the refrigerant flowing through the evaporator. there is The camera unit 10 shown in FIG. 5 is for taking an image of the storage compartment of the refrigerator 1, etc., and is installed on the upper surface of the housing.

FIG. 6 is a side view of the refrigerator 1. FIG.
As shown in FIG. 6, the camera unit 10 includes a body portion 101 and a support portion 102 . The main unit 101 has a function of capturing an image of a storage room or the like. The support portion 102 supports the body portion 101 and is installed on the upper surface of the housing of the refrigerator 1 .

A lens 103 (see also FIG. 9) is provided near the front end of the main body 101 . The lens 103 is an optical element that refracts and converges light. A fisheye lens, for example, is used as such a lens 103 . Then, when the left door 12 and/or the right door 13 (see FIG. 5) is opened, the lens 103 faces downward so that the camera unit 10 can image the refrigerator compartment 116 (see FIG. 7). It has become. The camera unit 10 is attached at a position where there is a possibility of imaging the living space range including the living space outside the refrigerator compartment 116 .

As shown in FIG. 6, the lens 103 is located in front of the front end of the housing (opening of the housing). More preferably, the lens 103 is located further forward than the front surfaces of the closed left door 12 and right door 13 (see FIG. 5). As a result, for example, when the left door 12 and the right door 13 of the refrigerator compartment 116 are opened, the refrigerator compartment 116 can easily enter the field of view of the lens 103 . It is also possible to take an image of the vegetable compartment 16 and the like (see FIG. 5) with the camera unit 10 in a state where a drawer type door such as the vegetable compartment door is open. In this way, the lens 103 is provided at a position where each storage chamber can be viewed from above.

FIG. 7 is a front view of the refrigerator 1 with the left door 12 and the right door 13 opened.
As described above, the inner plate 122 of the left door 12 is provided with a plurality of door pockets 123 for storing foodstuffs and the like (the same applies to the right door 13). When the left door 12 and the right door 13 are opened, foodstuffs in the refrigerator compartment 116 and the door pockets 123 and 133 enter the view of the lens 103 of the camera unit 10 from above.

The rotary partition 124 shown in FIG. 7 is a partition for suppressing cold air leakage from the gap between the left door 12 and the right door 13. In the example of FIG. 7, a rotating partition 124 is provided at the end of the left door 12 opposite to the axis of the hinge 121 (see FIG. 8), and rotates as the left door 12 is opened and closed. ing.

Also, when manually photographing the refrigerator compartment 116 using the camera unit 10 , a photographing button 125 is provided on the left door 12 and a photographing button 135 is provided on the right door 13 to be pressed by the user. In the example of FIG. 7, in the left door 12, a shooting button 125 is provided at the bottom of the surface of the left door 12 opposite to the hinge 121 (see FIG. 8) (the same applies to the right door 13). From another point of view, when the left door 12 and the right door 13 are closed (see FIG. 5), one of the left door 12 and the right door 13 faces the other (rotating partition 124 in FIG. 7). ) is provided with a shooting button 125 .

Shooting performed when at least one of these shooting

buttons

125 and 135 is pressed by the user is called "manual shooting". Also, the photographing performed by the calculation unit 112 based on the elapsed time from the timing at which the left door 12 and the right door 13 start to open is referred to as "automatic photographing".

FIG. 8 is a plan view of the refrigerator 1 with the left door 12 and the right door 13 opened.
In the example of FIG. 8, the body portion 101 of the camera unit 10 is arranged slightly to the left of the center of the housing of the refrigerator 1 in the horizontal direction. More specifically, the main body 101 of the camera unit 10 is arranged directly above the seam between the closed left door 12 and right door 13 (see also FIGS. 5 and 6). By arranging the main body 101 in this way, for example, when the left door 12 and the right door 13 are opened, food items in the door pockets 123 and 133 (see FIG. 7) can easily enter the field of view of the camera unit 10. Become.

In addition, when using the refrigerator 1, the user is often conscious of the seam (boundary) between the left door 12 and the right door 13. Therefore, by arranging the body part 101 directly above the seam of the left door 12 and the right door 13, the user feels uncomfortable compared to the case where the body part 101 is positioned out of the seam in the left-right direction. can be suppressed.

FIG. 9 is a perspective view when the camera unit 10 is viewed obliquely from below.
A body portion 101 of the camera unit 10 shown in FIG. 9 includes a case and a camera LED 100 (Light Emitting Diode: illumination portion) in addition to the lens 103 (see also FIG. 6). In the example of FIG. 6, the case of the main body part 101 generally has a rectangular parallelepiped shape elongated in the front-rear direction. A circular hole (not shown) is provided in the lower surface of the main body 101 near the front end of the case, through which the lens 103 is exposed. A camera LED 100 is provided on the rear side (back side) of the lens 103 .

The camera LED 100 illuminates the refrigerator compartment 116 and the like so that the camera unit 10 can photograph the refrigerator compartment 116 (see FIG. 7) and the door pockets 123 and 133 (see FIG. 7) under moderate brightness. A light source. Also, the camera LED 100 (illumination unit) is provided outside the housing of the refrigerator 1 (see FIG. 6). Although the details will be described later, in the present embodiment, the camera unit 10 photographs the refrigerator compartment 116 and the like (see FIG. 7) while the interior light of the refrigerator 1 is turned off and the camera LED 100 is turned on. ing.

As shown in FIG. 9, by providing a lens 103 on the front side of the camera LED 100, with the left door 12 and the right door 13 opened, reflected light from the refrigerator compartment 116 (see FIG. 5) enters the lens. easier to do. Therefore, the food stored in the refrigerator compartment 116 or the like can be photographed appropriately. In addition, it is preferable that the camera LED 100 is positioned in front of the front end (the opening of the housing) of the housing (see FIG. 6) of the refrigerator 1 . This makes it easier for the light emitted from the camera LED 100 to enter the refrigerating compartment 116 or the like, thereby making it easier to obtain a clear photographing result.

The refrigerator 1 is a storage that includes a refrigerator compartment 116 that is a storage area for storing articles, and a camera unit 10 that captures an image of the article range including the refrigerator compartment 116 .

FIG. 10 is a sequence diagram of image processing according to the first embodiment.
When the user opens the door of the refrigerator 1, the calculation unit 112 detects that the door has been opened by the door sensor 117. FIG. Thereby, the calculation unit 112 takes an image of the refrigerator compartment 116 in which the food is stored by the camera unit 10 (step S10). The calculation unit 112 uploads the captured camera image of the refrigerator compartment 116 to the server 2a (step S11). This camera image is an image before processing that can be viewed by the user.

The CPU 21 of the server 2a functions as pre-processing image acquisition means for acquiring pre-processing images captured by the camera unit 10. The server 2a transmits the unprocessed camera image to the storage server 6a (step S12).

When the storage server 6a notifies the server 2b of the event (step S13), it saves the unprocessed camera image (step S14). The storage server 6a functions as first storage means for storing pre-processed images.
When the storage server 6a transmits the unprocessed camera image to the server 2b (step S15), the storage server 6a deletes the unprocessed camera image stored therein (step S16). In other words, the CPU 21 of the storage server 6a functions as deletion means for deleting the unprocessed camera image.

The server 2b corrects the lens distortion of the unprocessed camera image (step S17), and processes the camera image into a front view image by trapezoidal correction (step S18). In FIG. 10, correction of distortion by the lens is abbreviated as "lens correction".
Then, the server 2b requests the existing processed image from the storage server 6b (step S19), and receives the processed image from the storage server 6b (step S20). This existing image after processing is an image in which, for example, the range of articles inside both doors and the range of articles in the storage covered by these doors are photographed, and the distortion caused by the lens has been corrected. Say.

When the server 2b synthesizes the existing post-processing image and the keystone-corrected image (step S21), the synthesized post-processing image is transmitted to the storage server 6b for storage (step S22). At this time, the product range of the new keystone-corrected image is combined with the existing unprocessed image. As a result, it is possible to synthesize an image of a state in which only one door is open and an image of a state in which the other door is also open in the past, and to display the entire storage as an image.
The CPU 21 of the server 2b functions as post-processing image creating means for creating a post-processing image by deleting at least a part of the range other than the article range or reducing the amount of information on the pre-processing image. .

The storage server 6b stores the processed image received from the server 2b (step S23). The storage server 6b functions as second storage means for storing the processed image.
Then, the storage server 6b transmits the processed image to the mobile terminal 3 in response to the request from the mobile terminal 3 (step S24). Accordingly, the customer can browse the image of the refrigerator compartment 116 using the mobile terminal 3 .

The storage server 6a stores the unprocessed image and the storage server 6b stores the combined processed image, making it easier to manage each image. The mobile terminal 3 can easily acquire the post-processing image by simply accessing the storage server 6b.

The mobile terminal 3 never accesses the storage server 6a, and therefore does not know the IP address of the storage server 6a. As a result, it is impossible to know how to access the storage server 6a from the portable terminal 3, so that the customer or the customer's living environment in the unprocessed image may be unintentionally seen by others. can be suppressed.

The storage server 6b deletes the processed image after a predetermined period of time has elapsed after storing the processed image. That is, before the storage server 6b deletes the processed image, the storage server 6a deletes the unprocessed image stored by itself. As a result, it is possible to prevent others from unintentionally seeing the customer or the customer's living environment in the unprocessed image.

FIG. 11 is a diagram showing an unprocessed image captured by the camera unit 10 of the refrigerator 1. FIG. A refrigerator compartment 116 is shown in the lower half of the image. The inside of the right door 13 is shown on the left side of the image. The inside of the left door 12 is shown on the right side of the image. The customer's living environment is reflected in the upper half of the image. This portion is an unnecessary portion when viewed from the processed image.

FIG. 12 is a diagram showing an image before processing in which unnecessary portions are filled.
This system masks unnecessary parts of the fisheye image. Masking processing executed by this system is processing processing for deleting at least a part of the range other than the article range. As a result, it is possible to prevent the customer and the customer's living environment from being unintentionally seen by others when viewing the image of the inside of the customer's refrigerator.
Although the server 2b executes the masking process of the unnecessary part, the server 2a or the refrigerator 1 may execute the masking process.

FIG. 13 is a diagram showing an example of a processed image.
The server 2b corrects the lens distortion of the unprocessed image captured by the camera unit 10 of the refrigerator 1, performs trapezoidal correction, and processes the image into a front view image. At this time, when the server 2b detects a face in the image, it performs mosaic processing on the area in which the face is shown.

The mosaic processing performed by this system is processing that reduces the amount of information in at least a portion of the range other than the range of goods. As a result, it is possible to prevent the customer from unintentionally viewing the image of the inside of the customer's refrigerator when viewing the image. The face detection processing and mosaic processing are performed by the server 2b, but may be performed by the server 2a or the refrigerator 1 on the image before processing.

14 to 20 show a series of images for confirming precautions regarding the handling of camera images to the customer when pairing the mobile terminal 3 and the refrigerator 1. FIG. These screens are displayed when the CPU 31 of the mobile terminal 3 executes the refrigerator management program installed in the mobile terminal 3 .

FIG. 14 is a diagram showing a new/used selection screen 41 displayed on the mobile terminal 3. As shown in FIG.
The selection screen 41 includes a new purchase button 411 , an other button 412 and a return button 419 .

The new purchase button 411 is for users who have newly purchased. When this button is tapped, the portable terminal 3 is paired with the refrigerator 1 and then registered as a user.

The other button 412 is a button for those who purchased the refrigerator 1 second-hand. When this button is tapped, the portable terminal 3 transitions to the warning screen 42 of FIG. 15 .
A return button 419 is a button for transitioning to the selection screen 41 one step before.

FIG. 15 is a diagram showing the warning screen 42. As shown in FIG.
The warning screen 42 is a screen that warns of precautions to be taken when the refrigerator 1 is second-hand.
A confirmation button 421 is a button for ending confirmation by the warning screen 42 . When this confirmation button 421 is tapped, the mobile terminal 3 performs user registration after pairing with the refrigerator 1 .
The return button 429 is a button that transitions to the warning screen 42 one step before.

By user registration, the identification number (individual number) of the refrigerator 1 and this mobile terminal 3 are linked. User registration may be performed by associating the individual number of the camera unit 10 with the portable terminal 3 instead of the identification number (individual number) of the refrigerator 1 .

Then, after user registration, if there is a user who has already been registered, the screen transitions to the registration continuation user selection screen 43 of FIG. After the user registration, if there is no user already registered, the screen transitions to the attention screen 45 for transfer shown in FIG.

FIG. 16 shows the continued registration user selection screen 43. As shown in FIG.
The registration continuation user selection screen 43 is a screen for selecting a user to continue registration with the refrigerator 1 . Check boxes 431 to 433 are check boxes for selecting users to continue registration.

The decision button 434 is a button for selecting a user to continue registration and then deciding it. By tapping the decision button 434, the portable terminal 3 transitions to the confirmation dialog box 44 of FIG. The return button 439 is a button for transitioning to the screen 43 for selecting users to continue registration.

FIG. 17 shows a confirmation dialog box 44. As shown in FIG.
A confirmation dialog box 44 is a dialog for confirming cancellation of registration of a user who does not want to continue registration.

The Yes button 441 is a button for canceling the registration of users who have not been selected.
A No button 442 is a button that does not cancel the registration of users that have not been selected. When the yes button 441 or the no button 442 is tapped, the screen transitions to the attention screen 45 for transfer shown in FIG.

FIG. 18 is a diagram showing a caution screen 45 when transferring.
The transfer warning screen 45 is a screen that displays a warning when transferring the refrigerator 1 . A confirmation button 451 is a button indicating that this caution has been confirmed.
The return button 459 is a button that makes a transition to the screen 45 immediately before the caution screen for transfer.

FIG. 19 is a diagram showing a camera use warning screen 46. As shown in FIG.
The camera use caution screen 46 is a screen for calling the customer's attention to photographing with the camera unit 10 which is the camera of the refrigerator 1 . When this screen is swiped to reach the lower end, a confirmation button for transitioning to the next image utilization warning screen 47 is displayed.
The return button 469 is a button that transitions to the one before transition to the camera use warning screen 46 .

FIG. 20 is a diagram showing an image utilization warning screen 47. As shown in FIG.
The image use warning screen 47 is a screen for obtaining the customer's consent to use the image captured by the camera unit 10 which is the camera of the refrigerator 1 .
A consent button 471 is a button for the customer to consent to the utilization of the image. When the consent button 471 is tapped, a series of settings are completed and the screen transitions to the home screen 50 .
The return button 479 is a button for transitioning to the image utilization warning screen 47 one step before.

FIG. 21 is a diagram showing the home screen 50. As shown in FIG.
The home screen 50 is a screen displayed after starting this refrigerator management program.
A refrigerator contents check button 501 is a button for browsing the contents of the customer's refrigerator 1 . When the refrigerator contents check button 501 is tapped, the portable terminal 3 transitions to the refrigerator contents screen 51 of FIG. 22 .
The usage concierge button 502 is a button for explaining how to use the refrigerator 1 .

FIG. 22 is a diagram showing the contents screen 51 of the refrigerator.
The refrigerator contents screen 51 is a screen for viewing a front view image of the refrigerating compartment 116 of the refrigerator 1 .
The indoor selection button 512 is a button for selecting which of the refrigerator compartment 116, the inside of the left door 12, and the inside of the right door 13 is to be displayed.
The return button 519 is a button for transitioning to the home screen 50 that is one screen before transitioning to the contents screen 51 of the refrigerator.

<<Second embodiment>>
In the second embodiment, lens correction processing, trapezoidal correction processing, and the like are performed by the calculator 112 of the refrigerator 1 instead of the server 2b. Since it is not necessary to send unprocessed camera images over the network, it is possible to prevent the customer and the customer's living environment from being unintentionally seen by others.

FIG. 23 is a sequence diagram of image processing according to the second embodiment.
When the user opens the door of the refrigerator 1, the calculation unit 112 detects that the door has been opened by the door sensor 117. FIG. Thereby, the calculation unit 112 takes an image of the refrigerator compartment 116 in which the food is stored by the camera unit 10 (step S30). This camera image is an image before processing that can be viewed by the user.

The calculation unit 112 stores the unprocessed camera image (step S31). The calculation unit 112 corrects the lens distortion of the unprocessed camera image (step S32), and processes the camera image into a front view image by trapezoidal correction (step S33). In FIG. 23, correction of distortion by the lens is abbreviated as "lens correction".
Then, the calculation unit 112 deletes the unprocessed camera image (step S34). The calculation unit 112 requests the existing processed image from the storage server 6b (step S35), and receives the processed image from the storage server 6b (step S36).

After combining the existing processed image and the keystone-corrected image (step S37), the calculation unit 112 transmits the combined processed image to the server 2b (step S38). The server 2b transmits the post-processing image to the storage server 6b for storage (step S39). The storage server 6b stores the processed image received from the server 2b (step S40). Then, the storage server 6b transmits the processed image to the mobile terminal 3 in response to the request from the mobile terminal 3 (step S41). Accordingly, the customer can browse the image of the refrigerator compartment 116 using the mobile terminal 3 .

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. A part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

Some or all of the above configurations, functions, processing units, processing means, etc. may be realized by hardware such as integrated circuits. Each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be placed on recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as flash memory cards and DVDs (Digital Versatile Disks). can.

In each embodiment, control lines and information lines indicate those considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In fact, it may be considered that almost all configurations are interconnected.
Modifications of the present invention include, for example, the following (a) to (d).

(a) Any of the refrigerator 1, the server 2a, and the server 2b may perform the mosaic processing for preventing the customer and the customer's living environment from being unintentionally seen by others.
(b) Any of the refrigerator 1, the server 2a, and the server 2b may perform the masking process to prevent the customer and the customer's living environment from being unintentionally seen by others.
(c) Either mosaic processing or masking processing may be performed, or both may be performed.

(d) The present invention is not limited to viewing the refrigerating compartment of a refrigerator, but may be applied to viewing the storage area of any storage cabinet.

1 Refrigerator (Storage)
100 camera LEDs
10 camera unit 112 calculation unit 113 ROM
114 RAMs
115 Communication unit 116 Refrigerator (storage area)
117 Door sensor 12 Left door 124 Rotating partition 13 Right door 14 Ice making compartment 15 Upper freezer compartment 16 Vegetable compartment 17 Lower freezer compartment 2a Server (pre-processing image acquisition means, deletion means)
2b Server (processed image creating means)
21 CPUs
22 ROMs
23 RAM
24 storage unit 25 communication unit 3 portable terminal 31 CPU
32 ROMs
33 RAM
34 storage unit 35 communication unit 36 touch panel display 41 selection screen 411 new purchase button 412 other button 419 return button 42 warning screen 429 return button 43 continued registration user selection screen 431 check box 432 check box 433 check box 434 enter button 439 return button 44 Confirmation dialog box 441 Yes button 442 No button 45 Caution screen for transfer 451 Confirm button 459 Back button 46 Camera usage caution screen 469 Back button 47 Image utilization caution screen 479 Back button 50 Home screen 501 Check contents of refrigerator button 502 How to use concierge button 51 Refrigerator contents screen 512 Room selection button 6a Storage server (first storage means, deletion means)
6b Storage server (second storage means)
81 unprocessed image acquisition means 82 first storage means 83 deletion means 84 post-process image creation means 85 second storage means 86 registration means 87 registered user information presentation means 88 registered user authority deletion means

Claims

a storage area for storing articles;
a storage including an imaging unit that captures an image of an article range including the storage area;
a pre-processed image acquiring means for acquiring a pre-processed image captured by the imaging unit;
a first storage means for storing the unprocessed image;
post-processing image creation means for creating a post-processing image by deleting at least a part of the range other than the article range or reducing the amount of information on the pre-processing image;
a second storage means for storing the processed image;
a deletion means for deleting the pre-processed image after the post-processed image is created;
An image management system comprising:
The imaging unit is attached at a position where there is a possibility of imaging a living space range including a living space outside the storage,
The deleting means deletes the unprocessed image at least before the processed image is deleted.
2. The image management system according to claim 1, characterized by:
The post-processing image creation means creates the post-processing image by deleting or reducing the amount of information including a predetermined range from the pre-processing image.
2. The image management system according to claim 1, characterized by:
When a body part of a person is reflected in any of the pre-processed images, the post-processed image creation means deletes or reduces the amount of information including the body part of the person, and reproduces the post-processed image. create,
2. The image management system according to claim 1, characterized by:
registration means for registering a user who can view the processed image by linking it to the individual number of the storage or the imaging unit;
Registered user information presenting means for presenting information on an already registered user to the user when newly registering the user to be linked to the individual number;
The image management system of Claim 1, further comprising:
Registered user authority deletion means for causing the user to select an already registered user when newly registering a user to be associated with the individual number, and deleting the viewing authority of the selected registered user;
6. The image management system of claim 5, further comprising:
The IP address assigned to the first storage means is different from the IP address assigned to the second storage means,
2. The image management system according to claim 1, characterized by:
pre-processing image acquiring means for acquiring a pre-processing image of an article range including a storage area for storing the article, which is captured by an imaging unit;
a first storage means for storing the unprocessed image;
post-processing image creation means for creating a post-processing image by deleting at least a part of the range other than the article range or reducing the amount of information on the pre-processing image;
a second storage means for storing the processed image;
a deletion means for deleting the unprocessed image at least before the processed image is deleted;
An image management server characterized by comprising:
One or more computing units
A procedure for obtaining a pre-processing image of an article range including an article storage area of a storage by an imaging unit;
a procedure for storing the pre-processing image;
a procedure of creating a post-processing image by deleting at least a part of the range other than the article range or reducing the amount of information on the pre-processing image;
a procedure for storing the processed image;
a procedure of deleting the unprocessed image at least before the processed image is deleted;
An image management method characterized by executing
a storage area for storing articles;
a storage including an imaging unit that captures an image of an article range including the storage area;
a pre-processed image acquiring means for acquiring a pre-processed image captured by the imaging unit;
a first storage means for storing the unprocessed image;
post-processing image creation means for creating a post-processing image by deleting at least a part of the range other than the article range or reducing the amount of information on the pre-processing image;
a second storage means for storing the processed image;
a deletion means for deleting the pre-processed image after the post-processed image is created;
A method of producing an image management system comprising:
A method of producing the image management system by providing a control program for controlling the imaging unit to the repository or the terminal used by the user.