CN117677811A - Image management system, image management server, image management method, and production method of image display control system - Google Patents

Abstract

The present invention generates an appropriate in-store image when a customer is allowed to view a refrigerator having 2 doors. The image display control system includes: a refrigerator (1) comprising a left door (12) and a right door (13) for opening and closing a refrigerating chamber (116), a door sensor (117) for detecting the opening and closing of the left door (12) and the right door (13) separately, and a camera unit (10); a shooting control unit which causes the camera unit (10) to shoot an image in response to detection information of the door sensor (117); and an image update unit that updates a range on the left door (12) side in the storage area of the image displayed by the display unit using image data captured by the camera unit (10) in response to the opening of the left door (12), and updates a range on the right door (13) side in the storage area of the image displayed by the display unit using image data captured by the camera unit (10) in response to the opening of the right door (13).

Description

Image management system, image management server, image management method, and production method of image display control system

Technical Field

The present invention relates to an image management system, an image management server, an image management method, and a production method of an image display control system.

Background

In recent years, there has been provided a service that enables a user to browse a room with a portable terminal by uploading an image obtained in the room of a shooting refrigerator (refrigerator) to a server and performing image processing on the uploaded image. Patent document 1 describes an invention that enables shooting of a large number of places with a small number of cameras.

Prior art literature

Patent literature

Patent document 1: japanese patent laid-open No. 2003-42626

Disclosure of Invention

Technical problem to be solved by the invention

In the case of a refrigerator provided with left and right doors, there are cases where articles in the refrigerator are taken out by opening only the left door or opening only the right door. In this case, if the condition in the warehouse is to be taken with both doors open, the state in the warehouse cannot be taken when food is taken with only a single door continuously open. If the condition is to be satisfied that the interior of the warehouse is also photographed in a state in which a single door is opened, a state in which a portion of the other door that is closed is not photographed is brought.

Technical means for solving the technical problems

In order to solve the above technical problem, an image display control system of the present invention includes: a storage, comprising: a first door and a second door for opening and closing the storage area; a door sensor that differentially detects opening and closing of the first door and the second door; and a photographing section; a shooting control unit that causes the shooting unit to shoot an image in response to (in accordance with) detection information of the door sensor; a storage unit that stores an image including the storage area for display on a display unit; and an image updating unit that updates a range on the first door side in the storage area of the image for display on the display unit in response to opening of the first door using the image data captured by the capturing unit, and updates a range on the second door side in the storage area of the image for display on the display unit in response to opening of the second door using the image data captured by the capturing unit.

The image display control server of the present invention includes: a receiving unit that receives detection information of a door sensor that separately detects opening and closing of a first door and a second door that open and close a storage area, and image data obtained by capturing the storage area; a storage unit that stores an image including the storage area for display on a display unit; and an image updating unit that updates a range on the side of the first door in the storage area for displaying an image of the display section in response to opening of the first door using image data obtained by photographing the storage area, and updates a range on the side of the second door in the storage area for displaying an image of the display section in response to opening of the second door using image data obtained by photographing the storage area.

An image display control method of the present invention is an image display control method concerning a storage, the storage including: a first door and a second door for opening and closing the storage area; a door sensor that differentially detects opening and closing of the first door and the second door; and an imaging unit, wherein 1 or more arithmetic units execute: a step of causing the imaging unit to take an image in response to detection information of the door sensor; a step of causing a display unit to display an image including the storage area; a step of updating a range of the first door side in the storage area for display in an image of the display unit using image data captured by the capturing unit in response to opening of the first door; and updating a range of the second door side in the storage area for display in the image of the display section using the image data captured by the capturing section in response to the opening of the second door.

The present invention provides a production method (generation method) of an image display control system including: a storage, comprising: a first door and a second door for opening and closing the storage area; a door sensor that differentially detects opening and closing of the first door and the second door; and a photographing section; a photographing control unit that causes the photographing section to photograph an image in response to detection information of the door sensor; a display control unit that displays an image including the storage area on a display unit; and an image update unit that updates a range on the first door side in the storage area of the image for display on the display unit in response to opening of the first door using image data captured by the capturing unit, and updates a range on the second door side in the storage area of the image for display on the display unit in response to opening of the second door using image data captured by the capturing unit, the image display control system generating method including: at least a step of providing a program for causing the display control unit to execute to a terminal.

Other technical means are described in the embodiments of the present invention.

Drawings

Fig. 1 is a block diagram of a refrigerator according to the present embodiment.

Fig. 2 is a block diagram of a server of the 2 types.

Fig. 3 is a block diagram of a portable terminal.

Fig. 4 is a logical block diagram of an image management system.

Fig. 5 is a front view of the refrigerator.

Fig. 6 is a side view of the refrigerator.

Fig. 7 is a front view of a refrigerator in which left and right doors of a refrigerator compartment are opened.

Fig. 8 is a plan view of a refrigerator in which left and right doors of a refrigerator compartment are opened.

Fig. 9 is a perspective view of the camera unit of the refrigerator from obliquely below.

Fig. 10 is a sequence (sequence) diagram of the first image processing process.

Fig. 11 is a detailed sequence diagram of the image synthesis process.

Fig. 12 is a flowchart of an image synthesizing method according to the state of the door sensor.

Fig. 13 is a view showing a pre-processing image captured by a camera of a refrigerator.

Fig. 14 is a view showing an example of a processed image.

Fig. 15 is a view showing an example of a processed image in a state where the right door is closed.

Fig. 16 is a view showing an example of a processed image in a state where the left door is closed.

Fig. 17 is a view showing a content screen of the refrigerator.

Fig. 18 is a diagram showing an adjustment screen.

Fig. 19 is a diagram showing a preview adjustment screen.

Fig. 20 is a diagram showing a preview adjustment screen during adjustment.

Fig. 21 is a diagram showing a discard dialog for adjusting contents.

Fig. 22 is a sequence diagram of image processing according to the second embodiment.

Detailed Description

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The image management system of the present embodiment is configured to cooperatively provide services to users by the refrigerator 1 of fig. 1, the servers 2a and 2b of fig. 2, the storage servers 6a and 6b, and the mobile terminal 3 of fig. 3.

Referring to fig. 1, the structure of a refrigerator 1 will be described.

The refrigerator 1 includes, as hardware components, a camera unit 10, an arithmetic unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory: random access Memory) 114, a communication unit 115, and a camera LED (Light Emitting Diode: light emitting diode) 100. The refrigerator 1 further includes a left door 12, a right door 13, a door sensor 117, and a refrigerating chamber 116 shown in fig. 1.

The arithmetic unit 112 is CPU (Central Processing Unit), for example, and performs integrated control of the refrigerator 1. The RAM114 is a work area of the arithmetic unit 112. The RAM114 is a non-transitory or temporary recording medium storing various programs and data, and stores a control program, not shown. The arithmetic unit 112 implements a shooting (image capturing) control means for causing the camera unit 10 to capture an image by executing a control program.

The camera unit 10 is an imaging unit that images the refrigerator 116, and is, for example, a fisheye camera or a panoramic camera. The camera unit 10 is installed at an upper side of the refrigerating chamber 116. The camera LED100 is an indicator light that is turned on when the camera unit 10 shoots. The left door 12 and the right door 13 are a first door and a second door that open and close the refrigerator 116 as a storage area, and the open/closed states of the doors can be detected separately by the door sensor 117. When the user opens the left door 12 or/and the right door 13, the camera unit 10 operates to photograph the refrigerating compartment 116. The refrigerating compartment 116 is a storage compartment for storing articles typified by food.

The communication unit 115 is connected to a network, and transmits and receives data to and from the server 2a as an external device.

With reference to fig. 2, the structures of the servers 2a, 2b and the storage servers 6a, 6b are described. The servers 2a, 2b and the storage servers 6a, 6b each have a CPU21, a ROM22, a RAM23, a storage section 24, and a communication section 25.

The CPU21 comprehensively controls the servers 2a and 2 b. ROM22 is a non-volatile memory, for example, that holds a BIOS (Basic I/O System). The RAM23 is a work area of the CPU 21. The RAM23 is a non-transitory or temporary recording medium storing various programs and data, and stores programs not shown.

The storage unit 24 is a non-transitory or transitory recording medium storing various programs and data, and is constituted by, for example, HDD (Hard Disk Drive), flash memory, and the like. The storage units 24 of the storage servers 6a and 6b are preferably large-capacity and can be accessed at high speed. The communication unit 25 is connected to a network, and transmits and receives data to and from an external device.

Referring to fig. 3, the structure of the portable terminal 3 will be described. The portable terminal 3 has a CPU31, a ROM32, a RAM33, a storage section 34, a communication section 35, and a touch panel display 36, respectively.

The CPU31 performs integrated control of the portable terminal 3. ROM32 is a non-volatile memory, such as that holding a BIOS. The RAM33 is a work area of the CPU 31. The RAM33 is a non-transitory or temporary recording medium storing various programs and data, and stores programs not shown.

The storage unit 34 is a non-transitory or transitory recording medium storing various data and programs, and is constituted by, for example, a flash memory or the like. That is, the refrigerator management program is installed in the storage unit 34. The CPU31 executes a refrigerator management program to realize a display control unit for displaying an image including the refrigerator 116 on the touch panel display 36 and a setting unit for receiving an instruction to adjust the angle, range, and center position of the image in relation to the display mode of the image, and performing setting in advance.

The communication unit 35 is connected to a network, and transmits and receives data to and from an external device. The touch panel display 36 is configured by laminating touch panels on a flat display. The touch panel display 36 is a display section that displays characters, images, or graphics, and is also an input section that detects a user operation.

Fig. 4 is a logical block diagram of an image management system.

The image management system includes a shooting control unit 81, a receiving unit 82, an image updating unit 83, a storage unit 84, a display control unit 85, a pre-setting unit 86, and a display section 87.

The photographing control unit 81 causes the camera unit 10 to photograph an image in accordance with the detection information of the door sensor 117.

The receiving unit 82 receives: detection information of a door sensor 117 for detecting opening and closing of the left door 12 and the right door 13 for opening and closing the refrigerating chamber 116, and image data obtained by photographing the refrigerating chamber 116 are separately detected. The reception unit 82 is realized by the CPU21 of the server 2a executing a program.

The image update unit 83 updates the range on the first door side in the refrigerating compartment 116 for the image displayed on the display section 87 using the image data captured by the camera unit 10 in accordance with the opening of the left door 12, and updates the range on the right door 13 side in the refrigerating compartment 116 for the image displayed on the display section 87 using the image data captured by the camera unit 10 in accordance with the opening of the right door 13. The image updating unit 83 is realized by the CPU21 of the server 2b executing a program.

The storage unit 84 stores an image including the refrigerating compartment 116 for display on the display portion 87. The storage unit 84 is realized by executing a program by the CPU21 of the storage server 6b.

The display control unit 85 displays an image including the refrigerating compartment 116 on the display portion 87.

The advance setting unit 86 performs advance setting by receiving an adjustment instruction for at least one of the angle, the range, and the center position of the image with respect to the display mode for displaying the image including the refrigerator 116. The display control unit 85 causes the display unit 87 to display an image so as to reflect the adjustment instruction instructed by the advance setting unit 86. The display control unit 85 and the pre-setting unit 86 are realized by the CPU31 of the portable terminal 3 executing a refrigerator management program.

Fig. 5 is a front view of the refrigerator 1.

The refrigerator 1 is a device for cooling food or the like, and includes left and right doors 12 and 13 of a refrigerator compartment 116, and a camera unit 10. The refrigerator 1 has a plurality of storage compartments inside its cabinet. As the storage compartments of the refrigerator 1, a refrigerating compartment 116, an ice-making compartment 14 and an upper-stage freezing compartment 15, a vegetable compartment 16, and a lower-stage freezing compartment 17 are provided in this order from above.

The refrigerator 1 has a case in which a heat insulating material (not shown) such as foamed polyurethane is filled between an outer case made of steel sheet and an inner case made of resin (not shown). A plurality of openings corresponding to the respective chambers are provided on the front side (front side) of the cabinet (housing) of the refrigerator 1. For example, when food or the like is put in through the opening of the refrigerating chamber 116, the left door 12 and the right door 13 are opened. When the left door 12 and the right door 13 are closed, the opening of the refrigerator 116 is closed. Thus, the left door 12 and the right door 13 have a function of closing the opening of the refrigerating chamber 116, and the like. The same applies to other doors.

The refrigerator 1 includes a left door 12 and a right door 13 as side-by-side doors forming a refrigerating chamber 116 together with a cabinet. The left door 12 and the right door 13 correspond to a first door and a second door for opening and closing the storage area. The left door 12 is rotatable about the axis of a hinge 121 (see fig. 8) at the left end. The same applies to the right door 13. The refrigerator 1 includes an ice making compartment door, an upper-stage freezing compartment door, a vegetable compartment door, and a lower-stage freezing compartment door as drawer doors.

In the refrigerating chamber 116, a plurality of shelves for partitioning the refrigerating chamber 116 in a predetermined manner are provided. Inside the left door 12, a plurality of door shelves (door shelves) 123 for storing food and the like are provided (see fig. 7). The same applies to the left door 12. The ice making chamber 14 is provided with an ice making chamber container (not shown) that is drawn out integrally with an ice making chamber door. Similarly, an upper-stage freezing chamber container (not shown) is provided in the upper-stage freezing chamber 15, a vegetable chamber container (not shown) is provided in the vegetable chamber 16, and a lower-stage freezing chamber container (not shown) is provided in the lower-stage freezing chamber 17.

The refrigerator 1 includes a compressor (not shown), a radiator (condenser), a capillary tube (throttle mechanism), and an evaporator. The refrigerant circulates sequentially through the compressor, the radiator, the capillary tube, and the evaporator, and air in the storage chamber is cooled by heat exchange with the refrigerant flowing through the evaporator. The camera unit 10 shown in fig. 4 is provided on the top surface of the cabinet for photographing a storage room or the like of the refrigerator 1.

Fig. 6 is a side view of the refrigerator 1.

As shown in fig. 6, the camera unit 10 includes a main body portion 101 and a support portion 102. The main body 101 has a function of photographing a storage room or the like. The support 102 supports the main body 101 and is provided on the top surface of the refrigerator 1.

A lens 103 (see fig. 6) is provided near the front end of the main body 101. The lens 103 is an optical element for refracting and condensing light. As such a lens 103. For example using a fish eye lens. In order to be able to image the refrigerator compartment 116 (see fig. 5) with the camera unit 10 when the left door 12 and/or the right door 13 (see fig. 5) are opened, the lens 103 is oriented downward. The camera unit 10 is installed at a position where there is a possibility of photographing a living space range including a living space outside the refrigerating compartment 116.

As shown in fig. 6, the lens 103 is located on the front side of the front end of the case (opening of the case). More preferably, the lens 103 is located further to the front side than the front surfaces of the left door 12 and the right door 13 (refer to fig. 5) in the closed state. Thus, for example, when the left door 12 and the right door 13 of the refrigerating chamber 116 are opened, the refrigerating chamber 116 is easily brought into view of the lens 103. In addition, the vegetable compartment 16 and the like can be photographed by the camera unit 10 even in a state where the drawer door such as the vegetable compartment door is opened (see fig. 5). In this way, the lens 103 is provided at a position where each storage chamber can be viewed from above.

Fig. 7 is a front view of the refrigerator 1 in a state in which the left door 12 and the right door 13 are opened.

As described above, the inner panel 122 of the left door 12 is provided with the plurality of door frames 123 for accommodating food and the like (the same applies to the right door 13). When the left door 12 and the right door 13 are opened, food or the like in the refrigerating chamber 116, the door frames 123, 133 enters the field of view of the lens 103 of the camera 10 in plan view.

The rotary partition 124 shown in fig. 7 is a partition body for suppressing leakage of cool air from a gap between the left door 12 and the right door 13. In the example of fig. 7, a rotation spacer 124 is provided at an end of the left door 12 opposite to the axis of the hinge 121 (see fig. 8), and rotates in a predetermined manner as the left door 12 opens and closes. The camera unit 10 is disposed in an imaginary extension of the rotary partition 124 in the longitudinal direction when the left door 12 is in the closed state.

In addition, a photographing button 125 for a user to press when photographing the refrigerator compartment 116 manually using the camera unit 10 is provided at the left door 12, and a photographing button 135 is provided at the right door 13. In the example of fig. 7, an imaging button 135 is provided on the left door 12 at a lower portion of a surface opposite to a hinge 121 (see fig. 8) of the left door 12 (the same applies to the right door 13). From another point of view, in a state where the left door 12 and the right door 13 are closed (see fig. 5), the photographing button 125 is provided on a surface (in the vicinity of the rotary spacer 124 of fig. 7) of one of the left door 12 and the right door 13 facing the other.

The photographing performed when at least one of the photographing buttons 125 and 135 is pressed by the user is referred to as "manual photographing". The imaging performed by the computing unit 112 based on the elapsed time from the time when the left door 12 and the right door 13 start to open is referred to as "automatic imaging".

Fig. 8 is a plan view of the refrigerator 1 in a state in which the left door 12 and the right door 13 are opened.

In the example of fig. 8, the main body 101 of the camera unit 10 is disposed slightly to the left of the center of the refrigerator 1 in the left-right direction. More specifically, the main body 101 of the camera unit 10 is disposed immediately above the joint between the left door 12 and the right door 13 in the closed state (see fig. 5). By disposing the main body 101 in this manner, for example, when the left door 12 and the right door 13 are opened, food or the like of the door frames 123, 133 is likely to enter the field of view of the camera unit 10.

In addition, when using the refrigerator 1, the user often notices the joint (boundary portion) between the left door 12 and the right door 13. Therefore, by disposing the main body 101 directly above the joint between the left door 12 and the right door 13, it is possible to prevent the user from feeling unnatural, as compared with the case where the position of the main body 101 in the lateral direction is deviated from the joint.

Fig. 9 is a perspective view of the camera unit 10 from obliquely below.

The main body 101 of the camera unit 10 shown in fig. 9 includes the case 31b and the camera LED100 (Light Emitting Diode: illumination section) in addition to the lens 103 (see fig. 6). In the example of fig. 6, the case of the main body 101 has a substantially rectangular parallelepiped shape elongated in the front-rear direction. A circular hole (not shown) is provided in the bottom surface near the front end of the case of the main body 101, and the lens 103 is exposed through the hole. Further, a camera LED100 is provided on the rear side (back side) of the lens 103.

The camera LED100 is a light source that irradiates light to the refrigerating compartment 116 and the like so that the camera unit 10 can photograph the refrigerating compartment 116 (see fig. 7) and the shelves 123 and 133 (see fig. 7) with appropriate brightness. The camera LED100 (illumination unit) is provided outside the cabinet (see fig. 5) of the refrigerator 1. In the present embodiment, the camera unit 10 captures images of the refrigerator compartment 116 and the like (see fig. 7) while the in-compartment lamp of the refrigerator 1 is turned off and the camera LED100 is turned on, as will be described in detail later.

As shown in fig. 9, by providing the lens 103 on the front side of the camera LED100, reflected light from the refrigerator 116 and the like (see fig. 5) is likely to enter the lens 31a in a state where the left door 12 and the right door 13 are open. Therefore, the food stored in the refrigerator 116 or the like can be appropriately photographed. Further, the camera LED100 is preferably located on the front side of the front end (opening of the case) of the case (see fig. 6) of the refrigerator 1. Thus, light emitted from the camera LED100 is easily incident on the refrigerator 116 or the like, and a clear imaging result is easily obtained.

The refrigerator 1 is a storage compartment including a refrigerator compartment 116 and a camera unit 10, wherein the refrigerator compartment 116 is a storage area for storing articles, and the camera unit 10 is used for photographing an article range including the refrigerator compartment 116.

Fig. 10 is a sequence diagram of the image processing of the first embodiment.

When the user opens the door of the refrigerator 1, the arithmetic unit 112 detects that the door is opened by the door sensor 117. Thus, the computing unit 112 photographs the refrigerator 116 in which the food is stored with the camera unit 10 (step S10). That is, the operation unit 112 realizes a shooting control unit that causes the camera unit 10 to shoot an image in accordance with the detection information of the door sensor 117.

The arithmetic unit 112 uploads the captured camera image of the refrigerator compartment 116 to the server 2a (step S11). The camera image is an image before processing that is viewable by the user.

The CPU21 of the server 2a is a pre-processing image acquisition unit that acquires a pre-processing image captured by the camera unit 10. The server 2a transmits the camera image before processing to the storage server 6a (step S12). Servers 2a, 2b and storage servers 6a, 6b.

When the storage server 6a notifies the server 2b of an event (event) (step S13), the camera image before processing is stored (step S14). The storage server 6a functions as a first storage unit that stores the pre-processing image.

When the storage server 6a transmits the camera image before processing to the server 2b (step S15), the camera image before processing stored in itself is deleted (step S16). That is, the CPU21 of the storage server 6a functions as a deletion unit that deletes the camera image before processing.

The server 2b corrects distortion caused by the lens of the camera image before processing (step S17), and processes the camera image into an image for front viewing by trapezoidal correction (step S18). The server 2b requests the storage server 6b for the existing (original) processed image (step S19), and receives the processed image from the storage server 6b (step S20).

When the server 2b synthesizes the existing processed image and the trapezoid corrected image (step S21), the synthesized processed image is transmitted to the storage server 6b and stored (step S22). The storage server 6b holds the processed image received from the server 2b (step S23). The storage server 6b functions as a storage unit that stores an image including the refrigerating compartment 116 for display on the touch panel display 36 of the portable terminal 3.

The storage server 6b transmits the processed image to the mobile terminal 3 in response to a request from the mobile terminal 3 (step S24). When the mobile terminal 3 receives the processed image from the storage server 6b, the processed image is displayed on the touch panel display 36. Thereby, the user can view the image of the refrigerator compartment 116 with the mobile terminal 3.

The portable terminal 3 does not access the storage server 6a, and therefore does not know the IP address of the storage server 6 a. Thus, the access method for accessing the storage server 6a from the mobile terminal 3 cannot be known, and the customer and the living environment of the customer reflected in the image before processing can be prevented from being seen by others accidentally.

The storage server 6b erases (deletes) the processed image when a predetermined period of time elapses after the processed image is stored. That is, the storage server 6a erases the pre-processing image stored by itself before the storage server 6b erases the post-processing image. This prevents customers and living environments of customers reflected in the images before processing from being seen by others accidentally.

Fig. 11 is a detailed sequence diagram of the image synthesis process. Here, the processing between the storage server 6a, the server 2b, and the storage server 6b will be described in detail. The server 2b has 2 functional units, namely a door processing unit 26 for processing an image of a door portion and a storage internal processing unit 27 for processing an internal image of the refrigerator compartment 116.

The storage server 6a transmits the model number of the refrigerator 1 and the angle information of the left door 12 and the right door 13 to the server 2b (step S50). The gate processing unit 26 intercepts the image before processing according to the angle information (step S51), performs the synthesis processing (step S52), and generates the processed image. In this combination process, the door processing unit 26 detects a human body or the like reflected in the image, performs mosaic processing or concealment processing, and performs mosaic processing or concealment processing for a living space other than the inside of the door. Then, the door processing unit 26 transmits the processed door image to the storage server 6b (step S53).

In parallel with these processes, the library internal processing section 27 determines whether or not the left door 12 and the right door 13 are open (step S54). If the left door 12 and the right door 13 are open (Yes), the in-library processing section 27 transmits the processed in-library image to the storage server 6b (step S55). If one of the left door 12 and the right door 13 is not opened (No), the in-library processing unit 27 intercepts an in-library image corresponding to the opened door (step S56), performs a synthesis process (step S57), and generates a processed image. In this combination process, the library internal processing unit 27 detects a human body or the like reflected in the image, performs mosaic processing or concealment processing on the detected human body or the like, and performs mosaic processing or concealment processing on a living space other than the inside of the door. Then, the library internal processing unit 27 transmits the processed door image to the storage server 6b (step S58).

When the storage server 6b receives the processed image, the image is stored (step S59).

Fig. 12 is a flowchart of an image synthesizing method according to the state of the door sensor.

The in-house processing unit 27 receives the captured image of the refrigerator compartment 116 of the refrigerator 1 and the detection information of the door sensor 117 (step S70). Then, the library internal processing section 27 determines the detection state of the door sensor 117 (step S71).

When only the left door 12 is opened, the library internal processing unit 27 updates the range of the left door side in the processed image (step S72), and the process proceeds to step S75. That is, the in-house processing section 27 updates the range on the side of the left door 12 in the refrigerating compartment 116 for the image displayed on the portable terminal 3, using the image data captured by the camera unit 10 in accordance with the opening of the left door 12.

In addition, the left door 12 includes a rotating divider 124. The in-house processing unit 27 may update the range of the side of the left door 12 and the range of the rotation partition 124 in the refrigerating compartment 116 for the image displayed on the portable terminal 3 using the image data captured by the camera unit 10 in accordance with the opening of the left door 12.

When only the right door 13 is opened, the library internal processing unit 27 updates the range of the right door side in the processed image (step S73), and the process proceeds to step S75. That is, the in-house processing section 27 updates the range on the side of the right door 13 in the refrigerating compartment 116 for the image displayed on the portable terminal 3 using the image data captured by the camera unit 10 in accordance with the opening of the right door 13.

When both the left door 12 and the right door 13 are open, the library internal processing part 27 updates the ranges of the right door 13 side, the left door 12 side, and the rotation spacer 124 side in the processed image (step S74), and proceeds to step S75. That is, the in-house processing section 27 updates the range on the side of the left door 12 and the range on the side of the right door 13 in the refrigerating compartment 116 for the image displayed on the portable terminal 3 using the image data captured by the camera unit 10 in accordance with the opening of the left door 12 and the right door 13.

In step S75, when the in-store processing unit 27 transmits the processed image of the refrigerating compartment 116 of the refrigerator 1 to the storage server 6b and stores the processed image, the processing of fig. 12 is terminated. Thus, the in-store processing unit 27 can generate an appropriate in-store image when photographing the inside of the refrigerator provided with 2 doors and enabling the customer to view the inside.

Fig. 13 is a view showing a pre-processing image captured by a camera of the refrigerator 1. The refrigerating compartment 116 is shown in the lower half of the image. The inside of the right door 13 is shown on the left side of the image. The inside of the left door 12 is shown on the right side of the image. The living environment of the customer is reflected in the upper half of the image. Which is an unwanted part from the processed image.

Fig. 14 is a view showing an example of a processed image.

The server 2b corrects the distortion of the lens for the pre-processing image captured by the camera unit 10 of the refrigerator 1, and performs trapezoidal correction to the processed image as an image for front viewing. At this time, if a face is detected in the image, the server 2b performs mosaic processing on the area in which the face is reflected.

The mosaic processing executed by the system is processing for reducing the information amount of at least a part of the range out of the range of the article. Thus, the customer can be prevented from being seen by others accidentally. The face detection process and the mosaic process are executed by the server 2b, but may be executed by the server 2a or the refrigerator 1 on an image before processing.

The left door inside image 71 of fig. 14 is a processed image obtained by cutting out the region captured inside the left door 12. The in-store image 72 is a processed image obtained by capturing a region of the refrigerator compartment 116. The intra-library image 72 is divided into a region 721, a region 722, and a region 723.

The areas 721, 722 in the intra-library image 72 are occluded when the left door 12 is closed. The areas 722, 723 are blocked when the right door 13 is closed. Region 722 is the region corresponding to the rotating partition. The right door inside image 73 is a processed image obtained by cutting out the region captured inside the right door 13.

Fig. 15 is a view showing an example of the processed image 74 in a state where the right door 13 is closed. The right side of the processed image 74 is blocked by the right door 13, and the region 721 of the refrigerating chamber 116 is shown on the left side. At this time, the server 2b synthesizes the left area 721 with the photographed image of the past refrigerating chamber 116.

Fig. 16 is a view showing an example of the processed image 75 in a state where the left door 12 is closed. The left side of the processed image 75 is blocked by the left door 12, and the region 723 of the refrigerating compartment 116 is shown on the right side. At this time, the server 2b synthesizes the right-side area 723 with the photographed image of the past refrigerating chamber 116.

Fig. 17 to 21 show a flow of preview display setting. These figures are operation screens of the touch panel display 36 of the portable terminal 3 of the user. That is, the CPU31 of the mobile terminal 3 executes the refrigerator management program installed in the mobile terminal 3, and displays these operation screens.

Fig. 17 is a view showing a content screen 51 of the refrigerator displayed on the touch panel display 36 of the mobile terminal 3.

In the refrigerator contents screen 51, an in-refrigerator image 511 and an indoor selection button 512 are displayed. Here, the in-store image 511 reflects food and the like stored in the refrigerator 116. That is, the CPU31 executes a not-shown refrigerator management program to realize a display control means for displaying an image including the refrigerator compartment 116 on the touch panel display 36.

The indoor selection button 512 is used to select any one of an image of the refrigerator compartment 116, an image of the inner side of the left door 12, and an image of the inner side of the right door 13.

When the in-library image 511 is touched, the operation shifts to the pre-adjustment screen 52 shown in fig. 18.

Fig. 18 is a diagram showing the screen 52 before adjustment.

The in-library image 521 is displayed on the pre-adjustment screen 52, and a delete button 522, a preview adjustment button 523, and a download button 524 are arranged below the in-library image.

The delete button 522 is a button for deleting the in-library image 521 from the storage server 6b. The preview adjustment button 523 is a button for shifting to the preview adjustment screen 53 shown in fig. 19. The download button 524 is a button for downloading the in-library image 521 from the storage server 6b.

Fig. 19 is a diagram showing the preview adjustment screen 53.

The preview adjustment screen 53 includes a "restore to factory" button 532, an in-library image 531, a slide bar 533, a cancel button 534, and an overlay save button 535. The image 531 is seen in a hidden and transparent manner around the periphery of the in-library image.

The "restore to factory" button 532 is a button for bringing the adjustment parameter into a state of not being adjusted at the time of factory shipment. The "restore to factory" button 532 is inactive in a state where no adjustment is made. The in-library image 531 is an image when the in-library is viewed from the front, and is displayed together with an instruction for confirming the horizontal and vertical. The slide bar 533 is used to adjust the rotation angle of the image. The rotation angle of the image can also be adjusted by the user rotating the in-box image 531 while touching the in-box image 531 with 2 fingers.

The cancel button 534 is a button for returning to the screen before the preview adjustment button 523 is pressed. The overlay save button 535 is a button for adjusting parameters in the overlay setting. When the override save button 535 is touched, the setting adjustment parameter is overridden and is transferred to the pre-adjustment screen 52, and cannot be returned to the previous value.

Fig. 20 is a diagram showing a preview adjustment screen 53 during adjustment.

The preview adjustment screen 53 performs an operation of touching the image with 2 fingers and rotating the image. The adjusted rotation angle is 2 degrees, and "2 °" is shown on the sliding bar 533. In addition, the display range can be reduced and adjusted by a pinching (pin in), the display range can be enlarged and adjusted by a pinching out operation, the center position can be adjusted by a sliding (swipe) operation, and the like. When the cancel button 534 is touched after the adjustment is made in this way, the flow proceeds to the discard dialog 54 shown in fig. 21. That is, the CPU31 executes a not-shown refrigerator management program, and realizes a setting means for setting in advance by receiving an adjustment instruction of at least one of an angle, a range, and a center position of an image with respect to a display mode for displaying an image including the refrigerator 116.

Fig. 21 is a diagram showing a discard dialog 54 for adjusting contents.

The cancel button 541 is a button for canceling the discard of the adjustment content and returning to the immediately preceding preview adjustment screen 53. The OK button 542 is a button for discarding the adjustment content and returning to the unadjusted preview adjustment screen 53.

Fig. 22 is a sequence diagram of image processing according to the second embodiment.

When the user opens the door of the refrigerator 1, the arithmetic unit 112 detects that the door is opened by the door sensor 117. Thus, the computing unit 112 photographs the refrigerator 116 in which the food is stored with the camera unit 10 (step S30). The camera image is an image before processing to enable a user to browse.

The arithmetic unit 112 saves the camera image before processing (step S31). The computing unit 112 corrects distortion caused by the lens of the camera image before processing (step S32), and performs trapezoidal correction processing to an image viewed from the front (step S33). Then, the arithmetic unit 112 deletes the camera image before processing (step S34). The arithmetic unit 112 requests the storage server 6b for the existing processed image (step S35), and receives the processed image from the storage server 6b (step S36).

When the conventional processed image and the trapezoid-corrected image are combined (step S37), the computing unit 112 transmits the combined processed image to the server 2b (step S38). The server 2b transmits the processed image to the storage server 6b for storage (step S39). The storage server 6b holds the processed image received from the server 2b (step S40). The storage server 6b transmits the processed image to the mobile terminal 3 in response to a request from the mobile terminal 3 (step S41). Thereby, the user can view the image of the refrigerator compartment 116 with the mobile terminal 3.

The storage server 6b erases the processed image when a predetermined period of time has elapsed after the processed image is stored. That is, the storage server 6a erases the pre-processed image stored by itself before the storage server 6b erases the post-processed image. This prevents customers and living environments of customers reflected in the images before processing from being seen by others accidentally.

(modification)

The present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments are described in detail for the purpose of easily understanding the present invention, and are not limited to the configurations that are all described. In addition, a part of the structure of one embodiment may be replaced with the structure of another embodiment, and the structure of another embodiment may be added to the structure of one embodiment. In addition, other structures may be added, deleted, or replaced for a part of the structures of the respective embodiments.

For example, part or all of the above-described structures, functions, processing units, and the like may be implemented by hardware such as an integrated circuit. The above-described structures, functions, and the like may be implemented in software by a processor interpreting and executing a program for realizing the functions. Information such as programs, tables, and files for realizing the respective functions can be stored in a memory, a hard disk, a recording device such as SSD (Solid State Drive), or a recording medium such as a flash memory card or DVD (Digital Versatile Disk).

In each embodiment, the control lines and information lines are shown as deemed necessary for explanation, and not necessarily all of the control lines and information lines on the product are shown. In practice it is also possible to consider that almost all structures are interconnected.

As a modification of the present invention, for example, the following (a) to (c) are mentioned.

(a) The configuration of the server and the storage server is arbitrary and is not limited to the configuration shown in the embodiment.

(b) The present invention is not limited to browsing the refrigerator compartment of the refrigerator, and may be applied to browsing the storage area of any storage compartment.

(c) The combination of the image of the left side region and the image of the right side region of the refrigerator 116 is not limited to the server and the refrigerator, and for example, two images may be combined by a mobile terminal.

Description of the reference numerals

1. Refrigerator with a door

100. Camera LED

10. Camera (video camera) unit

112. Calculation unit

113 ROM

114 RAM

115. Communication unit

116. Refrigerating chamber

117. Door sensor

12. Left door

124. Rotary partition

13. Right door

14. Ice making chamber

15. Upper layer freezing chamber

16. Vegetable room

17. Lower layer freezing chamber

2a server

2b server

21 CPU

22 ROM

23 RAM

24. Storage unit

25. Communication unit

26. Door processing unit

27. In-house processing unit

3. Portable terminal

31 CPU

32 ROM

33 RAM

34. Storage unit

35. Communication unit

36. Touch panel display

51. Content screen of refrigerator

511. In-library image

512. Indoor selection button

52. Adjusting the front picture

521. In-library image

522. Delete button

523. Preview adjustment button

524. Download button

53. Previewing an adjustment screen

531. In-library image

532 Button for recovering to factory

533. Sliding bar (slide)

534. Cancel button

535. Cover save button

54. Abandon dialog box

541. Cancel button

542 OK button

6a storage server

6b storage server

71. Left door inside image

72. In-library image

721. Region(s)

722. Region(s)

723. Region(s)

73. Right door inside image

81. Shooting control unit

82. Receiving unit

83. Image update unit

84. Memory cell

85. Display control unit

86. Pre-setting unit

87. And a display unit.

Claims (7)

1. An image display control system, comprising:

a storage, comprising: a first door and a second door for opening and closing the storage area; a door sensor that differentially detects opening and closing of the first door and the second door; and a photographing section;

a photographing control unit that causes the photographing section to photograph an image in response to detection information of the door sensor;

a storage unit that stores an image including the storage area for display on a display unit; and

an image updating unit that updates a range on the first door side in the storage area of the image for display on the display unit in response to opening of the first door using image data captured by the capturing unit, and updates a range on the second door side in the storage area of the image for display on the display unit in response to opening of the second door using image data captured by the capturing unit.

2. The image display control system according to claim 1, wherein:

the first door includes a rotational partition,

the image updating unit updates a range on the side of the rotary divider in the storage area for the image displayed on the display unit using the image data captured by the capturing unit in response to the opening of the first door and the second door.

3. The image display control system according to claim 2, wherein:

the imaging unit is disposed in a virtual extension region in the longitudinal direction of the rotary spacer.

4. The image display control system according to claim 1, comprising:

a display control unit that displays an image including the storage area on a display unit; and

a setting means for setting in advance a display mode for displaying an image including the storage area by receiving an adjustment instruction for at least one of an angle, a range, and a center position of the image,

the display control means causes the display unit to display an image so as to reflect the adjustment instruction instructed by the advance setting means.

5. An image display control server, comprising:

a receiving unit that receives detection information of a door sensor that separately detects opening and closing of a first door and a second door that open and close a storage area, and image data obtained by capturing the storage area;

a storage unit that stores an image including the storage area for display on a display unit; and

an image updating unit that updates a range on the side of the first door in the storage area for displaying an image of the display section in response to opening of the first door using image data obtained by photographing the storage area, and updates a range on the side of the second door in the storage area for displaying an image of the display section in response to opening of the second door using image data obtained by photographing the storage area.

6. An image display control method for a storage container,

the storage house includes: a first door and a second door for opening and closing the storage area; a door sensor that differentially detects opening and closing of the first door and the second door; and a photographing part for photographing the object,

the image display control method is characterized in that:

1 or more arithmetic units execute:

a step of causing the photographing section to photograph an image in response to detection information of the door sensor;

a step of causing a display unit to display an image including the storage area;

a step of updating a range of the first door side in the storage area for display in an image of the display unit using image data captured by the capturing unit in response to opening of the first door; and

and updating a range on the second door side in the storage area for display in the image of the display unit using the image data captured by the capturing unit in response to the opening of the second door.

7. A method for producing an image display control system, characterized by:

the image display control system includes:

a storage, comprising: a first door and a second door for opening and closing the storage area; a door sensor that differentially detects opening and closing of the first door and the second door; and a photographing section;

a photographing control unit that causes the photographing section to photograph an image in response to detection information of the door sensor;

a display control unit that displays an image including the storage area on a display unit; and

an image updating unit that updates a range of the first door side in the storage area of the image for display on the display unit in response to opening of the first door using image data captured by the capturing unit, and updates a range of the second door side in the storage area of the image for display on the display unit in response to opening of the second door using image data captured by the capturing unit,

the production method of the image display control system comprises the following steps: at least a step of providing a program for causing the display control unit to execute to a terminal.