JP2024058082A - Imaging system, imaging device, and refrigerator - Google Patents

Abstract

[Problem] To select an image that appropriately captures the state inside a refrigerator from images captured by a camera. [Solution] An imaging system 7 includes a wide-angle camera 301 that captures an image of a refrigerator 2 having a refrigerator compartment 21 and a door 21A that opens and closes an opening 201 of the refrigerator compartment 21, and is arranged on a top surface 20A of the refrigerator 2, an object PS that is arranged near the opening 201 of the refrigerator compartment 21 or on the door 21A and whose image capturing state by the wide-angle camera 301 changes depending on the opening and closing angle of the door 21A, a state storage unit 364 that stores a first image capturing state CD1 that is an image capturing state of the object PS by the wide-angle camera 301 at a predetermined opening and closing angle of the door 21A, and a generation unit 356 that selects a first image capturing state P1 that is an image captured when the door 21A is open from an image capturing image PB1 captured by the wide-angle camera 301 based on the first image capturing state CD1. [Selected Figure] Fig. 8

Description

The present disclosure relates to an imaging system, an imaging device, and a refrigerator.

Patent Document 1 discloses a technology in which a camera is installed on top of a refrigerator, a photograph is taken when an object passing through the opening of the refrigerator is detected, and the food stored in the refrigerator is managed based on the photographed results.

JP 2019-070476 A

The present disclosure provides an imaging system, an imaging device, and a refrigerator that selects an image that appropriately captures the interior condition of a refrigerator from images captured by a camera.

The imaging system of the present disclosure includes a camera that photographs a refrigerator having a storage compartment and an opening/closing body that opens and closes the opening of the storage compartment and is arranged on the top surface of the refrigerator, an object to be photographed that is arranged near the opening of the storage compartment or on the opening/closing body and whose photographing state by the camera changes depending on the opening/closing angle of the opening/closing body, a state storage unit that stores the photographing state of the object to be photographed by the camera at a predetermined opening/closing angle of the opening/closing body, and a selection unit that selects an image to be photographed when the opening/closing body is in an open state from images photographed by the camera based on the photographing state of the object to be photographed by the camera stored in the state storage unit.

The imaging device disclosed herein also includes a camera that photographs a refrigerator having a storage compartment and an opening/closing body that opens and closes the opening of the storage compartment, and is arranged on the top surface of the refrigerator; a state storage unit that stores the state of an object photographed by the camera at a predetermined opening/closing angle of the opening/closing body; and a selection unit that selects an image photographed when the opening/closing body is in an open state from images photographed by the camera based on the state of the object photographed by the camera stored in the state storage unit, and the object is arranged near the opening of the storage compartment or on the opening/closing body, and the state of the object photographed by the camera changes depending on the opening/closing angle of the opening/closing body.

The refrigerator disclosed herein also includes a camera that photographs a refrigerator having a storage compartment and an opening/closing body that opens and closes the opening of the storage compartment and is disposed on the top surface of the refrigerator; an object to be photographed that is disposed near the opening of the storage compartment or on the opening/closing body and whose photographing state by the camera changes depending on the opening/closing angle of the opening/closing body; a state storage unit that stores the photographing state of the object to be photographed by the camera at a predetermined opening/closing angle of the opening/closing body; and a selection unit that selects an image to be photographed when the opening/closing body is in an open state from images photographed by the camera based on the photographing state of the object to be photographed by the camera stored in the state storage unit.

The imaging system, imaging device, and refrigerator disclosed herein can select an image captured when the opening/closing body is in the open state from the images captured by the camera, based on the state of the camera's image of the subject stored in the state memory unit. Therefore, an image that appropriately captures the interior condition of the refrigerator can be selected from the images captured by the camera.

FIG. 1 shows a configuration of an item management system according to a first embodiment. FIG. 1 is a perspective view of a refrigerator according to a first embodiment; FIG. 1 is a side view of the imaging device according to the first embodiment, seen from the right side. FIG. 1 is a perspective view of a restricting member abutting against a front edge of a main box body according to the first embodiment, as viewed from the left front side; FIG. 1 is a front view of a refrigerator and an image capturing device according to a first embodiment; FIG. 1 is a side view of a refrigerator and an imaging device according to a first embodiment, viewed from the right. FIG. 1 is a plan view of an image capturing device attached to a refrigerator according to a first embodiment, seen from above; FIG. 1 is a block diagram showing a configuration of an imaging device according to a first embodiment. FIG. 13 is a diagram showing a configuration of a fourth example of an object to be photographed in the first embodiment; FIG. 1 is a diagram showing a configuration of a first example of an object to be photographed in the first embodiment; FIG. 1 is a diagram showing a configuration of a first example of an object to be photographed in the first embodiment; FIG. 1 is a diagram showing a configuration of a second example of an object to be photographed in the first embodiment; FIG. 1 is a diagram showing a configuration of a second example of an object to be photographed in the first embodiment; FIG. 13 is a diagram showing a configuration of a third example of an object to be photographed in the first embodiment; FIG. 1 is a diagram showing a first example of a photographed image of a subject in the first embodiment; FIG. 13 is a diagram showing a second example of a photographed object image of the photographed object in the first embodiment; FIG. 1 is a block diagram showing a configuration of a terminal device and an item management server according to a first embodiment. A flowchart showing the operation of the photographing device in the first embodiment. A flowchart showing the operation of the photographing device and the article management server in the first embodiment. A flowchart showing the operation of the terminal device and the article management server in the first embodiment. FIG. 13 is a diagram showing a configuration of an item management system according to a second embodiment; FIG. 11 is a block diagram showing a configuration of a refrigerator according to a second embodiment. Flowchart showing the operation of the imaging device

(Knowledge and other information that forms the basis of this disclosure)
At the time the inventors came up with the idea of the present disclosure, there was technology available that took a photograph when an object was detected passing through the opening of a refrigerator, and managed the food stored in the refrigerator based on the photographic results.
Therefore, the inventors discovered a problem of selecting an image that appropriately captures the interior condition of a refrigerator, and in order to solve this problem, they came to form the subject of the present disclosure.
In view of this, the present disclosure provides an imaging system, an imaging device, and a refrigerator that select an image that appropriately captures the interior condition of a refrigerator from images captured by a camera.

Hereinafter, the embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially the same configurations may be omitted.
It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

(Embodiment 1)
[1-1. Configuration]
[1-1-1. Configuration of the Item Management System]
FIG. 1 is a diagram showing a configuration of an item management system 1 according to the first embodiment.
The item management system 1 is a system for managing items stored in a refrigerator 2. The item management system 1 of the present embodiment manages images showing items stored in the refrigerator 2 as an example of managing items.

In this embodiment, the items stored in the refrigerator 2 include food and containers for storing food. The items may be marked with a specified marker.

The item management system 1 includes a refrigerator 2 .
The refrigerator 2 includes a main box 20 with an opening at the front. The main box 20 is formed with a refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25. These storage compartments are cooled with cold air. A revolving door 21A is installed at an opening 201 at the front of the refrigerator compartment 21. The door 21A includes a revolving right door 21B and a revolving left door 21C. The ice-making compartment 22, the fresh-freezing compartment 23, the freezer compartment 24, and the vegetable compartment 25 are each provided with a drawer 22A, a drawer 23A, a drawer 24A, and a drawer 25A capable of storing items. Each of the drawers 22A to 25A has an opening at the top when installed. Hereinafter, when the drawers 22A, 23A, 24A, and 25A are collectively referred to without distinction, they will be referred to as "drawer 26A" with the reference "26A".
The refrigerator compartment 21 corresponds to an example of a "storage compartment" in the present disclosure. The door 21A corresponds to an example of an "opening/closing body" in the present disclosure.

The item management system 1 includes an imaging device 3. The imaging device 3 is provided on a top surface 20A of a main box 20 of the refrigerator 2. The top surface 20A of the main box 20 is made of metal. The imaging device 3 captures an image of the refrigerator 2.
The top surface 20A corresponds to an example of the "upper surface" in the disclosure.

The item management system 1 includes a terminal device 4. The terminal device 4 is a portable computer such as a smartphone. An application program related to management of items stored in the refrigerator 2 is installed in the terminal device 4, and the terminal device 4 communicates with an item management server 5 by using the function of the application program.
In the following description, this application program will be referred to as an "item management application" and will be given the reference number "411."

In FIG. 1, a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When the terminal device 4 is used by a user P who is at home, it communicates with the item management server 5 via the communication device 6 or without the communication device 6. When the terminal device 4 is used by a user P who has left home H and a communication connection with the communication device 6 cannot be established, it communicates with the item management server 5 without the communication device 6.

The communication device 6 is connected to a network NW configured by a public line network or the like, and communicates with the item management server 5 via the network NW. The communication device 6 functions as an interface device for connecting the refrigerator 2, the imaging device 3, and the terminal device 4 to the network NW. The communication device 6 establishes a local network in the home H.

The item management system 1 includes an item management server 5. The item management server 5 is a device that manages the items stored in the refrigerator 2. The item management server 5 stores images of items as part of the item management. The item management server 5 is connected to a network NW. Note that in each figure, the item management server 5 is represented by a single block, but this does not necessarily mean that the item management server 5 is composed of a single device.

[1-1-2. Configuration of refrigerator]
FIG. 2 is a perspective view of the refrigerator 2 in an installed state.
FIG. 2 illustrates an X-axis, a Y-axis, and a Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to each other. The Z-axis indicates the up-down direction, which corresponds to the height direction of the refrigerator 2. The X-axis and the Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction, which corresponds to the width direction of the refrigerator 2. The Y-axis indicates the front-rear direction. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the front direction. The positive direction of the Z-axis indicates the upward direction.

The refrigerator 2 comprises a main box 20, a door 21A, and a drawer 26A.
The main box body 20 is a box-shaped member forming the main body of the refrigerator 2. The main box body 20 includes an outer box 202 that opens at the front, an inner box 203 that is housed inside the outer box 202 and opens at the front, and a heat insulating material filled in the space between the outer box 202 and the inner box 203. The outer box 202 is formed of, for example, a metal such as iron or a hard resin such as ABS. The upper surface of the outer box 202, i.e., the top surface 20A of the main box 20, is formed of a metal. The inner box 203 is formed of, for example, a hard resin such as ABS. For example, a hard urethane foam is used as the heat insulating material. Note that the heat insulating material is not limited to this, and may be any material that has heat insulating properties and can be filled in the space between the outer box 202 and the inner box 203.

The internal space of the inner box 203 is divided into multiple spaces by partition members. In this embodiment, the internal space of the inner box 203 is divided into five spaces, and the five spaces function as a refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezer compartment 24, and a vegetable compartment 25, respectively.

The right door 21B is rotatably supported by a right door hinge member 204. The right door hinge member 204 is a hinge member installed between the top surface 20A of the main box body 20 and the right door 21B and the drawer 23A. The right door hinge member 204 arranged between the right door 21B and the drawer 23A is shown in FIG. 12. The left door 21C is rotatably supported by a left door hinge member 205. The left door hinge member 205 is a hinge member installed between the top surface 20A of the main box body 20 and the right door 21B and the drawer 22A.

A second wireless communication module 206 is installed in the right door hinge member 204. The second wireless communication module 206 is a module that receives control data for controlling the refrigerator 2 from an external device and transmits specific data to the external device.

[1-1-3. Configuration of the imaging device]
The imaging device 3 will be described with reference to FIGS.
3 to 7 show the same X-axis, Y-axis, and Z-axis as those shown in Fig. 2. The X-axis direction corresponds to the width direction and the left-right direction of the imaging device 3. The Y-axis direction corresponds to the front-rear direction of the imaging device 3. The Z-axis direction corresponds to the up-down direction of the imaging device 3.

Figure 3 is a side view of the imaging device 3 attached to the refrigerator 2, seen from the right. Figure 4 is a perspective view of the restricting member 34 abutting against the front edge 20B of the main box body 20, seen from the front left.

3 and 4, the imaging device 3 includes a main body 30 that is installed on the top surface 20A of the main box 20, and an imaging member 31 that extends forward from the main body 30. The imaging member 31 is provided above the front of the main body 30.

The main body 30 has a power button 32 on the front for turning the power of the imaging device 3 on and off. The power button 32 is provided with a first LED (light emitting diode) 33 that notifies predetermined information.

The main body 30 includes a restricting member 34. The restricting member 34 restricts the position of the imaging member 31 in the front-rear direction of the refrigerator 2 by contacting the front edge 20B of the main box body 20. The restricting member 34 is a plate-like member extending downward from the bottom surface 30A of the main body 30. The restricting member 34 extends in the width direction of the main body 30 (the width direction of the imaging device 3) on the front side of the main body 30, and has a mark 35 at the center of the width direction of the main body 30. The mark 35 indicates the center of the width direction of the imaging device 3. The restricting member 34 is formed so that the length extending from the bottom surface 30A is shorter than the length from the top surface 20A of the main box body 20 to the door packing provided approximately parallel to the top surface 20A at the upper inner surface of the door 21A.

Figure 5 is a front view of the refrigerator 2 and the imaging device 3 attached to the refrigerator 2, as viewed from the front. The refrigerator 2 shown in Figure 5 has its door 21A open. Figure 6 is a side view of the refrigerator 2 and the imaging device 3 attached to the refrigerator 2, as viewed from the right. The refrigerator 2 shown in Figure 6 has its drawer 25A open, i.e., pulled out. Figure 7 is a plan view of the imaging device 3 attached to the refrigerator 2, as viewed from above.

The imaging device 3 includes a wide-angle camera 301 and a narrow-angle camera 302. The wide-angle camera 301 and the narrow-angle camera 302 are installed at the front end of the bottom surface 31A of the imaging member 31. When the imaging device 3 is attached to the refrigerator 2 and the door 21A is closed, the wide-angle camera 301 and the narrow-angle camera 302 are located in front of the door 21A.

Wide-angle camera 301 has a wider angle than narrow-angle camera 302. With imaging device 3 attached to refrigerator 2, wide-angle camera 301 captures an image of refrigeration compartment 21 from above and in front of refrigerator 2.
For example, the shooting range of wide-angle camera 301 includes range A1 shown in Fig. 5 when viewed from the front of refrigerator 2. Range A1 is a range that includes door pockets 21E of right door 21B and left door 21C and opening 201 of refrigerator compartment 21 when right door 21B and left door 21C are in the maximum open state. Moreover, the shooting range of wide-angle camera 301 includes range A3 shown in Fig. 6 when viewed from the right of refrigerator 2. Range A3 is a range that includes, in the front-to-back direction, from the front ends of right door 21B and left door 21C in the open state to the front end of shelf 21F, and also includes opening 201 of refrigerator compartment 21 in the up-down direction.
The wide-angle camera 301 corresponds to an example of a "camera" in the present disclosure.

5 to 7, wide-angle camera 301 constitutes a part of imaging system 7 in embodiment 1. Imaging system 7 in embodiment 1 includes wide-angle camera 301, an object to be photographed PS, state storage unit 364, and generation unit 356. In other words, imaging system 7 includes refrigerator 2 including object to be photographed PS, and imaging device 3 including wide-angle camera 301, state storage unit 364, and generation unit 356.
The subject PS, the state storage unit 364, and the generation unit 356 will be described with reference to FIG.

Narrow-angle camera 302 has a narrower angle than wide-angle camera 301. With imaging device 3 attached to refrigerator 2, narrow-angle camera 302 captures an image of drawer 26A from above and in the front of refrigerator 2.
For example, the imaging range of narrow-angle camera 302 includes range A2 shown in Fig. 5 when viewed from the front of refrigerator 2. Range A2 is a range that includes drawer 26A when viewed from the front of refrigerator 2. Furthermore, for example, the imaging range of narrow-angle camera 302 includes range A4 shown in Fig. 6 when viewed from the right of refrigerator 2. Range A4 is a range that includes drawer 26A when it is pulled out to the maximum in the front-to-rear direction. The imaging range of narrow-angle camera 302 may be a range that includes drawers provided in refrigeration compartment 21.

The imaging device 3 includes a first photosensor 303 and a second photosensor 304. The first photosensor 303 and the second photosensor 304 are installed in the main body 30.
The first photosensor 303 is a sensor that detects whether the right door 21B is open or closed. The first photosensor 303 is a reflective photosensor. When the imaging device 3 is attached to the refrigerator 2, the first photosensor 303 projects light onto a predetermined position on the right door 21B. The first photosensor 303 then outputs a signal indicating the amount of received light.
The second photosensor 304 is a sensor that detects whether the left door 21C is open or closed. The second photosensor 304 is a reflective photosensor. When the imaging device 3 is attached to the refrigerator 2, the second photosensor 304 projects light onto a predetermined position on the left door 21C. The second photosensor 304 then outputs a detection value that indicates the amount of received light.

The imaging device 3 includes a thermopile 305. The thermopile 305 is a sensor that detects the opening and closing of the drawer 26A. The thermopile 305 is installed at the front end of the bottom surface 31A of the imaging member 31. The thermopile 305 outputs a detection value indicating the detected temperature.

The imaging device 3 is equipped with a human presence sensor 306. The human presence sensor 306 may be an infrared sensor or an ultrasonic sensor. The human presence sensor 306 outputs a detection value. The human presence sensor 306 is installed at the front end of the bottom surface 31A of the imaging member 31.

The imaging device 3 includes an illuminance sensor 307. The illuminance sensor 307 is a sensor for adjusting the exposure of the wide-angle camera 301 and the narrow-angle camera 302. The illuminance sensor 307 outputs a detection value. The illuminance sensor 307 is installed at the front end of the imaging member 31.

The imaging device 3 is equipped with a second LED 308 and a third LED 309. The second LED 308 is installed at the right front end of the bottom surface 31A of the imaging member 31. The third LED 309 is installed at the left front end of the bottom surface 31A of the imaging member 31. The second LED 308 and the third LED 309 illuminate the refrigerator 2 to which the imaging device 3 is attached from above.

As shown in FIG. 7, the imaging device 3 includes a first control board 311 and a second control board 312.

The first control board 311 is a board that controls the various sensors equipped in the imaging device 3 and the power supply of the imaging device 3. The first control board 311 is installed inside the imaging device 3. The power button 32, the first LED 33, the first photosensor 303, the second photosensor 304, the thermopile 305, the human presence sensor 306, and the illuminance sensor 307 are connected to the first control board 311. The first control board 311 controls these devices. The first control board 311 is installed behind the second control board 312 and the first wireless communication module 313.

The second control board 312 is a board that controls the wide-angle camera 301 and the narrow-angle camera 302, and also controls the first wireless communication module 313. The second control board 312 is installed inside the imaging device 3. The wide-angle camera 301, the narrow-angle camera 302, and the first wireless communication module 313 are connected to the second control board 312. The second control board 312 controls these devices. The second control board 312 is installed in front of and alongside the first control board 311.

The first wireless communication module 313 is a device equipped with hardware such as an antenna and a wireless communication circuit. The first wireless communication module 313 is installed inside the imaging member 31 to the left of the second control board 312. The first wireless communication module 313 is installed in the imaging device 3 so that it is located forward of the opening 201 of the main box 20 when the imaging device 3 is attached to the refrigerator 2. More specifically, the first wireless communication module 313 is installed forward of the regulating member 34 in the imaging device 3.

As shown in FIG. 7, when the imaging device 3 is attached to the refrigerator 2, a metal plate 314 is placed between the first wireless communication module 313 and the second wireless communication module 206 in the left-right direction.

FIG. 8 is a block diagram showing the configuration of the imaging device 3.
As shown in FIG. 8, the first control board 311 includes a first processor 320 such as a CPU (Central Processing Unit), a first memory 330, a first communication module 340, and an interface circuit to which other devices and sensors are connected.

The first memory 330 is a storage device that stores programs and data executed by the first processor 320. The first memory 330 is composed of a non-volatile storage device such as a ROM (Read Only Memory). The first memory 330 may also include a volatile storage device that constitutes the work area of the first processor 320, such as a RAM (Random Access Memory). The first memory 330 stores data processed by the first processor 320 and a control program 331 executed by the first processor 320.

The first communication module 340 has hardware such as a communication circuit and connectors that comply with a specified wired communication standard, and communicates with the second control board 312 under the control of the first processor 320.

The first processor 320 reads and executes the control program 331 to turn the power of the imaging device 3 on and off in response to the operation of the power button 32 .
In addition, the first processor 320 reads and executes a control program 331 to turn on and off the first LED 33 , the second LED 308 , and the third LED 309 .
In addition, the first processor 320 reads and executes the control program 331 to transmit the detection values output by each sensor connected to the first control board 311 to the second control board 312 via the first communication module 340.

The second control board 312 includes a second processor 350 such as a CPU, a second memory 360, a second communication module 370, and an interface circuit to which other devices and sensors are connected.

The second memory 360 is a storage device that stores programs and data executed by the second processor 350. The second memory 360 is configured by a non-volatile storage device such as a ROM. The second memory 360 may also include a volatile storage device such as a RAM that configures a work area for the second processor 350. The second memory 360 stores data processed by the second processor 350, a control program 361 executed by the second processor 350, and an imaging device ID (Identification) 362.
The imaging device ID 362 is information for uniquely identifying the imaging device 3 .

The second communication module 370 has hardware such as a communication circuit and connectors that comply with a specified wired communication standard, and communicates with the first control board 311 under the control of the second processor 350.

The second processor 350 functions as a first communication control unit 351, a second communication control unit 352, an open/close determination unit 353, a shooting control unit 354, a recording control unit 355, and a generation unit 356 by reading and executing the control program 361 stored in the second memory 360. The second processor 350 also causes the second memory 360 to function as a state storage unit 364 by reading and executing the control program 361 stored in the second memory 360.

The first communication control unit 351 communicates with the item management server 5 via the first wireless communication module 313.

The second communication control unit 352 communicates with the first control board 311 via the second communication module 370. The second communication control unit 352 receives detection values of various sensors connected to the first control board 311 from the first control board 311.

The open/close determination unit 353 determines whether the right door 21B is open or closed based on the detection value of the first photosensor 303 received by the second communication control unit 352. When the second communication control unit 352 receives a detection value different from the detection value corresponding to the closed state, the open/close determination unit 353 determines that the right door 21B is open. On the other hand, when the second communication control unit 352 receives a detection value corresponding to the closed state, the open/close determination unit 353 determines that the right door 21B is closed.

The open/close determination unit 353 determines whether the left door 21C is open or closed based on the detection value of the second photosensor 304 by the second communication control unit 352, in the same manner as the determination for the right door 21B.

The open/close determination unit 353 determines whether the drawer 26A is open or closed based on the detection value of the thermopile 305 received by the second communication control unit 352. When the second communication control unit 352 receives a detection value of the thermopile 305 indicating a temperature below a predetermined temperature, the open/close determination unit 353 determines that the drawer 26A is open. On the other hand, when the second communication control unit 352 receives a detection value of the thermopile 305 indicating a temperature above the predetermined temperature, the open/close determination unit 353 determines that the drawer 26A is closed.

When the opening/closing determination unit 353 determines that the door 21A or the drawer 26A is open, the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing. When the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start photographing, the second communication control unit 352 transmits lighting request information to the first control board 311. The lighting request information is information requesting that the second LED 308 and the third LED 309 be turned on. When the first processor 320 receives the lighting request information, it turns on the second LED 308 and the third LED 309 and notifies the user P that photographing has started.

When the opening/closing determination unit 353 determines that the door 21A and the drawer 26A are closed, the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography. When the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography, the second communication control unit 352 transmits turn-off request information to the first control board 311. The turn-off request information is information requesting that the second LED 308 and the third LED 309 be turned off. When the first processor 320 receives the turn-off request information, it turns off the second LED 308 and the third LED 309 and notifies the user P that photography has ended.

When the open/close determination unit 353 determines that the door 21A or the drawer 26A is open, the recording control unit 355 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302. That is, the recording control unit 355 starts recording the recording data 363 in the second memory 360. The recording data 363 includes the first recording data DV1 indicating the images captured by the wide-angle camera 301 and the second recording data DV2 indicating the images captured by the narrow-angle camera 302.

When the open/close determination unit 353 determines that the door 21A and the drawer 26A are closed, the recording control unit 355 ends recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302.

The state memory unit 364 stores a first shooting state CD1 and a second shooting state CD2 in advance. The first shooting state CD1 indicates a shooting state of the object PS to be shot by the wide-angle camera 301 at a predetermined opening and closing angle of the door 21A. When the opening and closing angle is zero degrees, it indicates that the door 21A is in a closed state. The larger the opening and closing angle, the more widely the door 21A is open. The predetermined opening and closing angle is, for example, within a range of 60 degrees to 100 degrees. The predetermined opening and closing angle is, for example, 90 degrees. In the following description, the state in which the door 21A is at a predetermined opening and closing angle (for example, 60 degrees to 100 degrees) is referred to as the "first reference state."
The first shooting state CD1 includes, for example, at least one of the position, size, and shape of an image corresponding to the object PS in the captured image PA1 captured by the wide-angle camera 301 in the first reference state. In the first embodiment, the first shooting state CD1 includes, for example, the size of the image corresponding to the object PS in the captured image PA1 captured by the wide-angle camera 301 in the first reference state.
The first shooting state CD1 corresponds to an example of "a shooting state of an object to be shot by a camera at a predetermined opening/closing angle of the opening/closing body" in the present disclosure.

The object PS to be photographed is placed near the opening of the refrigerator compartment 21, on the door 21A, and the state of photography by the wide-angle camera 301 changes depending on the opening and closing angle of the door 21A.
The subject PS will be further described with reference to FIGS.

The second shooting state CD2 indicates the shooting state of the object to be shot by narrow-angle camera 302 when drawer 26A is pulled out to a predetermined amount. When the pulled out amount is zero, this indicates that drawer 26A is in a closed state. The larger the predetermined pulled out amount, the more open drawer 26A is. A pulled out amount of 100% indicates that drawer 26A is in a fully open state. The predetermined pulled out amount is, for example, in the range of 70% to 100%. The predetermined pulled out amount is, for example, 80%. In the following description, the state in which drawer 26A is pulled out to a predetermined amount (for example, 70% to 100%) is referred to as the "second reference state."
The second shooting state CD2 includes, for example, at least one of the position, size, and shape of the image corresponding to the object in the captured image PA2 captured by the narrow-angle camera 302 in the second reference state. In the first and second embodiments, the second shooting state CD2 includes, for example, the size of the image corresponding to the object in the captured image PA2 captured by the narrow-angle camera 302 in the second reference state.
The object to be photographed is placed, for example, at the upper end of drawer 26A, and the state of photography by narrow-angle camera 302 changes depending on the amount that drawer 26A is pulled out.

The generating unit 356 selects a first captured image P1 based on the first shooting state CD1 from the captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363. The first captured image P1 is an image captured when the door 21A is in an open state.
The captured image PB1 corresponds to an example of an "image captured by a camera" in the present disclosure.
The first captured image P1 corresponds to an example of a "captured image when the opening/closing body is in the open state" in the present disclosure.
Moreover, the generating unit 356 selects a second captured image P2 based on the second shooting state CD2 from the captured image PB2 included in the moving image indicated by the second record data DV2 of the record data 363. The second captured image P2 is an image captured when the drawer 26A is in the open state.
Then, the generation unit 356 generates management data D1 from the first captured image P1 and the second captured image P2. The management data D1 is data to be uploaded to the item management server 5 and is data for managing the items stored in the refrigerator 2. The first communication control unit 351 transmits the management data D1 generated by the generation unit 356 to the item management server 5.
The generating unit 356 corresponds to an example of a "selecting unit" of the present disclosure.
In this embodiment, a case will be described in which the image capturing device 3 includes the generating unit 356, but the present invention is not limited to this. For example, the item management server 5 may include the generating unit 356. In this case, the management data D1 includes the first recording data DV1 and the second recording data DV2.

In the first embodiment, the generation unit 356 selects the first captured image P1 based on the first shooting state CD1 from the captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363, but is not limited to this. The generation unit 356 may select, from the captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363, the captured image PB1 in which the user P is least reflected based on the first shooting state CD1 as the first captured image P1.
In this case, the generation unit 356 further selects the first captured image P1 from the first captured images P1 selected based on the first shooting state CD1 as follows: That is, the generation unit 356 recognizes an image of the user P by a method such as pattern matching, and selects the captured image PB1 having the smallest area of the recognized image of the user P as the first captured image P1.

Similarly, in the first embodiment, the generation unit 356 selects the second captured image P2 from the captured image PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363 based on the second shooting state CD2, but this is not limited to this. The generation unit 356 may select the captured image PB2 that has the least amount of user P reflected therein based on the second shooting state CD2 as the second captured image P2 from the captured image PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363.

[1-1-4. Configuration of the photographed subject]
Next, an example of the configuration of the object PS will be described with reference to Fig. 9. Fig. 9 is a diagram showing the configuration of a fourth example of the object PS in the first embodiment. In the following description, the object PS in the fourth example will be referred to as object PS4. The object PS4 is formed by a connecting member that connects the left end of the right door 21B and the right end of the opening of the refrigerator compartment 21. The larger the opening/closing angle of the right door 21B, the larger the left-right length LD of the object PS4.

In this case, the state memory unit 364 stores, as an example of the first shooting state CD1, the length LD of the captured image PA1 captured by the wide-angle camera 301 when the opening/closing angle is between 60 degrees and 100 degrees. Furthermore, based on the length LD of the captured image PA1 stored in the state memory unit 364, the generation unit 356 selects the first captured image P1 when the right door 21B is in the open state from the captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363.

The generating unit 356 may select the first captured image P1 in which the right door 21B is in the open state from the captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363, based on the area of the photographed object PS4 in the captured image PA1 stored in the state storage unit 364. In this case, the state storage unit 364 may store, as an example of the first photographing state CD1, the area of the photographed object PS4 in the captured image PA1 captured by the wide-angle camera 301 when the opening/closing angle is set to 60 to 100 degrees.

In Figure 9, a case has been described in which the object to be photographed PS4 is configured with a connecting member that connects between the left end of the right door 21B and the right end of the opening of the refrigerator compartment 21, but the object to be photographed PS4 may also be configured with a connecting member that connects between the right end of the left door 21C and the left end of the opening of the refrigerator compartment 21.
9, 10, 12, and 14 show the same X-axis, Y-axis, and Z-axis as those shown in FIG.

Next, the configuration of the photographed object PS will be described with reference to Figures 10 to 16. In the following description, the photographed object PS is placed near the opening of the refrigerator compartment 21 or on the right door 21B, and is photographed by the wide-angle camera 301.
In the following description, a case will be described in which the state storage unit 364 stores the first object image MK1 as the first shooting state CD1.
The first object image MK1 will be described with reference to FIGS.
The first subject image MK1 corresponds to an example of a "subject image" in the present disclosure.

FIGS. 10 and 11 are diagrams showing the configuration of a first example of the object to be photographed PS in embodiment 1. In the following description, the object to be photographed PS in the first example will be referred to as object to be photographed PS1. The left diagram in FIG. 10 is a front view of the refrigerator 2 and the imaging device 3 as seen from the front, and corresponds to the upper part of FIG. 5.

The right diagram of Fig. 10 is an enlarged view of an area AR1 in the left diagram of Fig. 10. As shown in the right diagram of Fig. 10, the subject PS1 is placed in the door pocket 21E provided at the top of the right door 21B. In Figs. 10 and 11, a case is described in which the subject PS1 is placed in the door pocket 21E of the right door 21B, but the subject PS1 may also be placed in the door pocket 21E of the left door 21C.
The photographed object PS1 is configured, for example, as a rectangular flat plate member AT3. The flat plate member AT3 is fixed to the upper end of a side surface member 21S of the door pocket 21E via a connecting member AT2 and an attachment member AT1. The side surface member 21S is a member that configures the side surface of the door pocket 21E that is close to the opening of the refrigerator compartment 21 (the left side in FIG. 10). The flat plate member AT3, the connecting member AT2, and the attachment member AT1 are configured integrally as the photographed object AT. The photographed object AT is formed, for example, from resin.
The connecting member AT2 and the mounting member AT1 correspond to an example of a "fixing member" in the present disclosure.

The photographing plane PF1 is the surface of the photographed object PS1 that is photographed by the wide-angle camera 301. The photographing plane PF1 is fixed to the side member 21S of the door pocket 21E so that the right door 21B is approximately perpendicular to the photographing direction DR of the wide-angle camera 301 when the right door 21B is at a predetermined opening/closing angle (e.g., 90 degrees). In other words, the photographed object PS1 is placed on the right door 21B so that the wide-angle camera 301 is positioned on the normal line at the center of the photographing plane PF1, i.e., on the dashed line indicating the photographing direction DR in FIG. 10.

10, since the photographed object PS1 is placed in the door pocket 21E provided at the top of the right door 21B, the distance between the wide-angle camera 301 and the photographed object PS1 can be made shorter than, for example, when the photographed object PS1 is placed in another door pocket 21E. As a result, the wide-angle camera 301 can appropriately capture the first subject image MK1 formed on the photographing plane PF1 of the photographed object PS1 in the first reference state.
10, the object PS1 is disposed on the side member 21S of the door pocket 21E of the right door 21B, so that the distance between the wide-angle camera 301 and the object PS1 can be made shorter than, for example, when the object PS1 is disposed at another position in the door pocket 21E. The other position in the door pocket 21E is, for example, the other side of the door pocket 21E (the right side in FIG. 10). As a result, the wide-angle camera 301 can properly capture the first target image MK1 formed on the imaging plane PF1 of the object PS1 in the first reference state.
10, the object PS1 to be photographed is disposed at the upper end of the side member 21S of the door pocket 21E of the right door 21B, which prevents the wide-angle camera 301 and the object PS1 from being blocked by items stored in the refrigerator 2. As a result, the wide-angle camera 301 can properly capture the first subject image MK1 formed on the photographing plane PF1 of the object PS1 in the first reference state.

Figure 11 is a perspective view of the photographed object PS1. As indicated by the dashed line in Figure 11, a first photographed object image MK1 including identification information indicating the type of item stored in the refrigeration chamber 21 is formed on the photographed surface PF1 of the photographed object PS1. For example, a sticker on which the first photographed object image MK1 is formed is affixed to the photographed surface PF1.

The state storage unit 364 stores a first object image MK1. The first object image MK1 corresponds to an example of a first object image capturing state CD1.
The generation unit 356 selects a first photographed image P1 in a case where the right door 21B is in an open state from a captured image PB1 included in a moving image represented by the first recorded data DV1 of the recorded data 363, based on the first photographed object image MK1 stored in the state storage unit 364. Note that the generation unit 356 may select a first photographed image P1 in a case where the right door 21B is in an open state from a photographed image PB2 included in a moving image represented by the second recorded data DV2 of the recorded data 363, based on the first photographed object image MK1 stored in the state storage unit 364.
The first object image MK1 will be further described with reference to FIGS.

As shown in FIG. 11, the attachment member AT1 includes a first member AT11, a second member AT12, and an opening/closing member AT13. The opening/closing member AT13 is formed of a spring or the like, and opens and closes the first member AT11 and the second member AT12. That is, the first member AT11 and the second member AT12 are opened and inserted into the side member 21S of the door pocket 21E, and the first member AT11 and the second member AT12 are closed, whereby the photographed object PS1 is fixed to the side member 21S of the door pocket 21E.

12 and 13 are diagrams showing a configuration of a second example of the object to be photographed PS in embodiment 1. In the following description, the object to be photographed PS in the second example will be referred to as object to be photographed PS2.
As shown in Fig. 12, the photographed object PS2 is configured as a hinge cover AS. The left diagram of Fig. 12 shows a state in which the hinge cover AS is not attached, and the right diagram of Fig. 12 shows a state in which the hinge cover AS is attached. In the left diagram of Fig. 12, since the hinge cover AS is not attached, the right door hinge member 204 is exposed. The right door hinge member 204 axially supports the right door 21B so as to be rotatable.
The right door hinge member 204 corresponds to an example of a "hinge member."

The hinge cover AS is formed in an L-shape in a plan view, and an imaging surface PF2 is formed on the upper surface. A first object image MK1 including identification information indicating the type of item stored in the refrigeration chamber 21 is formed on the imaging surface PF2. For example, a sticker on which the first object image MK1 is formed is affixed to the imaging surface PF2.
The first object image MK1 will be further described with reference to FIGS.

Fig. 13 is a perspective view of the hinge cover AS. As shown in Fig. 13, a notch AS1 is formed in the hinge cover AS. The notch AS1 fits into the upper part of the frame of the drawer 23A. The hinge cover AS is made of, for example, resin. The hinge cover AS may be the same color as the main box body 20 and the hinge, or may be a different color.
Moreover, the state storage unit 364 and the generation unit 356 execute the same processes as those described with reference to FIGS.

Fig. 14 is a diagram showing a configuration of a third example of the object PS in the first embodiment. In the following description, the object PS in the third example will be referred to as object PS31, object PS32, and object PS33. Each of the object PS31, object PS32, and object PS33 corresponds to an example of the object PS in the third example. As shown in Fig. 14, each of the object PS31, object PS32, and object PS33 is disposed near the opening of the refrigeration chamber 21.
The photographed object PS31 is disposed on the lower right side of the inner surface of the refrigeration compartment 21. In other words, the photographed object PS31 is configured as a part of the lower right side of the inner surface of the refrigeration compartment 21. The photographed surface PF31 is configured integrally with the photographed object PS31. A first photographed object image MK1 including identification information indicating the type of item stored in the refrigeration compartment 21 is formed on the photographed surface PF31. For example, a sticker on which the first photographed object image MK1 is formed is affixed to the photographed surface PF31.

The photographed object PS32 is placed at the right end of the lower surface of the inner surface of the refrigerator compartment 21. In other words, the photographed object PS32 is configured as part of the right end of the lower surface of the inner surface of the refrigerator compartment 21. The photographed surface PF32 is configured integrally with the photographed object PS32. A first photographed object image MK1 including identification information indicating the type of item stored in the refrigerator compartment 21 is formed on the photographed surface PF32. For example, a sticker on which the first photographed object image MK1 is formed is affixed to the photographed surface PF32.

The photographed object PS33 is disposed at the right end of the upper surface of the front member of the drawer 23A of the fresh freezing compartment 23, which is disposed below the refrigerator compartment 21. In other words, the photographed object PS33 is configured as a part of the right end of the upper surface of the front member of the drawer 23A of the fresh freezing compartment 23, which is disposed below the refrigerator compartment 21. The photographed surface PF33 is configured integrally with the photographed object PS33. A first photographed object image MK1 including identification information indicating the type of item stored in the refrigerator compartment 21 is formed on the photographed surface PF33. For example, a sticker on which the first photographed object image MK1 is formed is affixed to the photographed surface PF33.

In addition, the state storage unit 364 and the generation unit 356 execute the same processing as that described with reference to Figures 10 and 11.

In FIG. 14, the photographed object PS31 is placed on the lower right side of the inner surface of the refrigerator compartment 21, but the photographed object PS31 may be placed on the lower left side of the inner surface of the refrigerator compartment 21. Also, in FIG. 14, the photographed object PS32 is placed on the right end of the lower surface of the inner surface of the refrigerator compartment 21, but the photographed object PS32 may be placed on the left end of the lower surface of the inner surface of the refrigerator compartment 21. In FIG. 14, the photographed object PS33 is placed on the right end of the upper surface of the front member of the drawer 23A of the fresh freezing compartment 23 placed below the refrigerator compartment 21, but is not limited to this. The photographed object PS33 may be placed on the left end of the upper surface of the front member of the drawer 22A of the ice making compartment 22 placed below the refrigerator compartment 21.

Next, the first object image MK1 will be described with reference to Fig. 15 and Fig. 16. Fig. 15 is a diagram showing a first example of the first object image MK1 of the object PS in the first embodiment.
15, the first subject image MK1 includes a rectangular outer frame image MF and an identification image MD formed inside the outer frame image MF. The identification image MD corresponds to identification information indicating the type of item stored in the refrigeration compartment 21 of the refrigerator 2. The outer frame image MF is formed in, for example, black color.

The identification image MD is composed of a first rectangular image PD1 in a first color and a second rectangular image PD2 in a second color different from the first color, and the arrangement pattern of the first rectangular image PD1 and the second rectangular image PD2 indicates the identification information. The first color is, for example, white, and the second color is, for example, black.
The size of the first rectangular image PD1 is the same as the size of the second rectangular image PD2, and each of the first rectangular image PD1 and the second rectangular image PD2 is, for example, a square. In other words, the width WP1 of the first rectangular image PD1 and the width WP2 of the second rectangular image PD2 are the same as the height HP of the first rectangular image PD1 and the second rectangular image PD2.

The thickness HF of the outer frame image MF in the short side direction (up and down direction) and the thickness WF of the outer frame image MF in the long side direction (left and right direction) are each the same as, for example, the width WP1 of the first rectangular image PD1, the width WP2 of the second rectangular image PD2, the height HP of the first rectangular image PD1, and the height HP of the second rectangular image PD2.
Each of the width HF in the short side direction (up-down direction) of the outer frame image MF and the width WF in the long side direction (left-right direction) of the outer frame image MF is determined based on the resolution of the wide-angle camera 301 and the distance between the wide-angle camera 301 and the first subject image MK1 at a predetermined opening/closing angle (e.g., 90 degrees) of the right door 21B. Also, each of the width HF in the short side direction of the outer frame image MF and the width WF in the long side direction of the outer frame image MF is determined based on the imaging range or angle of view of the wide-angle camera 301.
For example, the larger the resolution of the wide-angle camera 301, the smaller the width HF of the outer frame image MF in the long side direction and the width WF of the outer frame image MF in the short side direction can be. Also, the longer the distance between the wide-angle camera 301 and the first captured target image MK1 at a predetermined opening/closing angle (e.g., 90 degrees) of the right door 21B, the larger the width HF of the outer frame image MF in the long side direction and the width WF of the outer frame image MF need to be. Also, the larger the imaging range or angle of view of the wide-angle camera 301, the larger the width HF of the outer frame image MF in the long side direction and the width WF of the outer frame image MF need to be.
In the first subject image MK1 shown in FIG. 15, the width HF of the outer frame image MF in the long side direction and the width WF of the outer frame image MF in the short side direction are each, for example, 6 mm.

The first rectangular image PD1 and the second rectangular image PD2 are arranged along the longitudinal direction (here, the left-right direction) of the outer frame image MF. In Fig. 15, five first rectangular images PD1 and four second rectangular images PD2 are arranged as the identification image MD. The first rectangular image PD1 represents, for example, "0," and the second rectangular image PD2 represents, for example, "1." In other words, the identification image MD represents a nine-digit binary number.
The identification image MD is composed of an information image MDA and a code image MDB. The information image MDA is composed of five rectangular images PD, and the code image MDB is composed of four rectangular images PD. The rectangular images PD are the first rectangular images PD1 or the second rectangular images PD2. The information image MDA corresponds to identification information indicating the type of item stored in the refrigeration chamber 21 of the refrigerator 2. The code image MDB indicates a code for error correction of the identification information indicated by the information image MDA.
Since the information image MDA is composed of five rectangular images PD, it represents a five-digit binary number. That is, the information image MDA represents 2 to the power of 5, that is, 32 types of identification information. For example, the information image MDA shown in FIG. 15 represents a five-digit binary number, "01010."

The size of the first rectangular image PD1 and the size of the second rectangular image PD2 are determined based on the resolution of the wide-angle camera 301 and the distance between the wide-angle camera 301 and the first captured object image MK1 at a predetermined opening/closing angle (e.g., 90 degrees) of the right door 21B. The size of the first rectangular image PD1 and the size of the second rectangular image PD2 are determined based on the imaging range or angle of view of the wide-angle camera 301.
For example, the larger the resolution of the wide-angle camera 301, the smaller the size of the first rectangular image PD1 and the size of the second rectangular image PD2 can be. Also, the longer the distance between the wide-angle camera 301 and the first captured object image MK1 at a predetermined opening/closing angle (e.g., 90 degrees) of the right door 21B, the larger the size of the first rectangular image PD1 and the size of the second rectangular image PD2 must be. Also, the larger the imaging range or angle of view of the wide-angle camera 301, the larger the size of the first rectangular image PD1 and the size of the second rectangular image PD2 must be.
In the first subject image MK1 shown in FIG. 15, the size of the first rectangular image PD1 and the size of the second rectangular image PD2 are squares with one side measuring 6 mm.

Figure 16 is a diagram showing a second example of a first photographed object image MK1 of the photographed object PS in embodiment 1. The first photographed object image MK1 shown in Figure 16 differs from the first photographed object image MK1 shown in Figure 15 in that the identification image MD has two rows of rectangular images PD in the vertical direction.

The information image MDA is composed of ten rectangular images PD, and the code image MDB is composed of eight rectangular images PD. Of the eight rectangular images PD that make up the code image MDB, the four rectangular images PD in the top row indicate codes for error correction of the identification information indicated by the five rectangular images PD in the top row out of the ten rectangular images PD that make up the information image MDA. Also, of the eight rectangular images PD that make up the code image MDB, the four rectangular images PD in the bottom row indicate codes for error correction of the identification information indicated by the five rectangular images PD in the bottom row out of the ten rectangular images PD that make up the information image MDA.

In FIG. 16, a case is described in which the identification image MD has two rows of rectangular images PD in the vertical direction, but this is not limited to the case. The identification image MD may have three or more rows of rectangular images PD in the vertical direction.

In Figures 15 and 16, a case where the outer frame image MF is formed in black is described, but this is not limited to this. For example, the outer frame image MF may be formed in blue. Also, a case where the first color is white and the second color is black is described, but this is not limited to this. For example, the first color may be yellow and the second color may be blue.

Each of the photographed objects PS1, PS2, PS31, PS32, and PS33 described with reference to Figures 10 to 14 has a first photographed object image MK1. Also, as shown in Figures 15 and 16, the first photographed object image MK1 includes a rectangular outer frame image MF and an identification image MD formed inside the outer frame image MF. Therefore, the generation unit 356 executes the following process.

That is, the generating unit 356 first judges whether or not an outer frame image MF can be detected from each captured image PB1 included in the moving image indicated by the first recorded data DV1 of the recorded data 363. Then, when the outer frame image MF can be detected from the captured image PB1, the generating unit 356 judges whether or not the identification information indicated by the identification image MD inside the detected outer frame image MF can be read. Next, when the identification information indicated by the identification image MD can be read and the read identification information matches the identification information included in the first photographed image MK1 stored in the state storage unit 364, the generating unit 356 selects the captured image as the first photographed image P1.
Furthermore, for example, when the generation unit 356 can detect an outer frame image MF from each captured image PB1 included in the moving image indicated by the first recording data DV1, the generation unit 356 notifies the user P by lighting or blinking an LED, sound, or the like that the object PS has been captured by the wide-angle camera 301. Note that, for example, when the generation unit 356 can detect an outer frame image MF from each captured image PB2 included in the moving image indicated by the second recording data DV2, the generation unit 356 may notify the user P by lighting or blinking an LED, sound, or the like that the object PS has been captured by the narrow-angle camera 302.
The case where the outer frame image MF can be detected from each captured image PB1 corresponds to an example of "the case where it is determined that the camera has captured an image of the subject".

In addition, in order for the generating unit 356 to easily detect the outer frame image MF from each captured image PB1 included in the moving image indicated by the first recording data DV1, it is preferable that the color of the outer frame image MF is different from the color of the inner surface of the refrigerator compartment 21 and the color of the items stored in the refrigerator compartment 21. Therefore, it is preferable that the color of the outer frame image MF is, for example, black, blue, etc.

In addition, an unillustrated object to be photographed is also placed at the upper end of drawer 26A, and the object to be photographed has a second object to be photographed image MK2 having a configuration similar to that of the first object to be photographed image MK1 described with reference to Figures 15 and 16.
Therefore, the state storage section 364 stores the second object image MK2 as the second shooting state CD2. The generation section 356 also executes the following process.

That is, the generating unit 356 first judges whether or not an outer frame image MF can be detected from each captured image PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363. Then, when the outer frame image MF can be detected from the captured image PB2, the generating unit 356 judges whether or not the identification information indicated by the identification image MD inside the detected outer frame image MF can be read. Next, when the identification information indicated by the identification image MD can be read and the read identification information matches the identification information included in the second photographed image MK2 stored in the state storage unit 364, the generating unit 356 selects the captured image PB2 as the second photographed image P2.
In addition, for example, when the generation unit 356 can detect an outer frame image MF from each of the captured images PB2 included in the moving image indicated by the second recording data DV2, the generation unit 356 notifies the user P that the object PS has been captured by the narrow-angle camera 302 by lighting or blinking an LED, or by sound, etc.
The case where an outer frame image MF can be detected from each captured image PB2 corresponds to an example of "a case where it is determined that the camera has captured an image of the subject".

[1-1-5. Configuration of item management server]
FIG. 17 is a block diagram showing the configuration of the terminal device 4 and the item management server 5.
The item management server 5 includes a server processor 50, which is a processor such as a CPU, a server memory 51, and a server communication module 52. The server processor 50 is connected to the server memory 51 and the server communication module 52.

The server memory 51 is a storage device that stores programs and data executed by the server processor 50. The server memory 51 is configured with a non-volatile storage device. The server memory 51 may also include a non-volatile storage device that configures a work area of the server processor 50. The server memory 51 stores data processed by the server processor 50, a control program 511 executed by the server processor 50, and an item management DB (database) 512.
The item management DB 512 has one record for each imaging device 3. This one record includes the imaging device ID 362, image data of the first captured image P1, and image data of the second captured image P2.

The server communication module 52 is equipped with a communication circuit and connector that conforms to a specific communication standard, and communicates with the refrigerator 2, the imaging device 3, and the terminal device 4 via the network NW.

The server processor 50 reads and executes a control program 511 stored in the server memory 51 , thereby functioning as a server communication control unit 501 and a server processing unit 502 .
The server communication control unit 501 communicates with the refrigerator 2, the imaging device 3, and the terminal device 4 via the server communication module 52.
The server processing unit 502 performs various information processes using the imaging device 3 and the terminal device 4 as clients. The server processing unit 502 may also use the refrigerator 2 as a client.

[1-1-6. Configuration of terminal device]
Next, the configuration of the terminal device 4 will be described.
The terminal device 4 includes a terminal processor 40 , which is a processor such as a CPU, a terminal memory 41 , a terminal communication module 42 , and a touch panel 43 .

The terminal memory 41 is a storage device that non-volatilely stores programs and data executed by the terminal processor 40. The terminal memory 41 is composed of a non-volatile storage device. The terminal memory 41 may also include a RAM that constitutes the work area of the terminal processor 40. The terminal memory 41 stores data processed by the terminal processor 40, control programs executed by the terminal processor 40, and an item management application 411.

The terminal communication module 42 includes a communication circuit and a connector that comply with a predetermined communication standard, and communicates with the item management server 5 under the control of the terminal processor 40 .
The touch panel 43 includes a display panel and a touch sensor that is provided over or integral with the display panel.

The terminal processor 40 functions as an application execution unit 400 .
The application execution unit 400 communicates with the item management server 5 via the terminal communication module 42 .
The application execution unit 400 displays a user interface related to management of items stored in the refrigerator 2 on the touch panel 43. In the following description, this user interface is referred to as an "application UI." The display control unit 402 of this embodiment displays images of items stored in the refrigerator 2 by displaying the application UI.
The application execution unit 400 receives various inputs from the user P via the application UI.

[1-2. motion]
Next, the operation of each part of the item management system 1 will be described.

[1-2-1. Recording-related operations]
First, the operation of the image capture device 3 relating to recording will be described.
FIG. 18 is a flowchart FA showing the operation of the imaging device 3.

The open/close determination unit 353 determines whether the door 21A or the drawer 26A is in the open state (step SA1).

If the open/close determination unit 353 determines that the door 21A or the drawer 26A is not open (step SA1: NO), the second processor 350 returns the process to step SA1.

If the open/close determination unit 353 determines that the door 21A or the drawer 26A is open (step SA1: YES), the image capture control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start capturing images (step SA2).

Next, the recording control unit 355 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302 (step SA3).

Next, the open/close determination unit 353 determines whether the door 21A and the drawer 26A are closed (step SA4). If the open/close determination unit 353 determines that the door 21A and the drawer 26A are not closed (step SA4: NO), it performs the determination in step SA4 again.

On the other hand, if it is determined that the door 21A and the drawer 26A are closed (step SA4: YES), the recording control unit 355 ends recording of the captured image (step SA5). As a result, recording data 363 is generated in the second memory 360.

Next, the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end shooting (step SA6).

[1-2-2. Operations related to transmission of management data]
Next, the operation relating to the transmission of the management data D1 will be described.
FIG. 19 is a flowchart showing the operation of the imaging device 3 and the item management server 5.
In FIG. 19, a flowchart FB shows the operation of the imaging device 3, and a flowchart FC shows the operation of the item management server 5.

The generating unit 356 determines whether or not the recording data 363 is stored in the second memory 360 (step SB1). If the generating unit 356 determines that the recording data 363 is not stored (step SB1: NO), it performs the determination of step SB1 again.

On the other hand, if the generation unit 356 determines that the recording data 363 has been stored (step SB1: YES), it selects the first captured image P1 from the captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363 stored in the second memory 360, based on the first shooting state CD1 stored in the state memory unit 364 (step SB2).

Step SB2 will now be described in detail.
The first captured image P1 is an image captured when the right door 21B or the left door 21C is open at a predetermined opening/closing angle (for example, 90 degrees).

The generation unit 356 first determines whether or not an outer frame image MF can be detected from each captured image PB1 included in the moving image indicated by the first recording data DV1 of the recording data 363. Then, if the outer frame image MF can be detected from the captured image PB1, the generation unit 356 determines whether or not the identification information indicated by the identification image MD inside the detected outer frame image MF can be read. Next, if the identification information indicated by the identification image MD can be read and the read identification information matches the identification information included in the first photographed image MK1 stored in the state memory unit 364, the generation unit 356 selects the captured image PB1 as the first photographed image P1.

Returning to the explanation of the flowchart FB, the generation unit 356 selects a second captured image P2 from each of the captured images PB2 included in the moving image indicated by the second recording data DV2 of the stored recording data 363 (step SB3).

Step SB3 will now be described in detail.
The second captured image P2 is an image captured when the drawer 26A is in the open state.

The generation unit 356 first determines whether or not an outer frame image MF can be detected from each captured image PB2 included in the moving image indicated by the second recording data DV2 of the recording data 363. Then, if the outer frame image MF can be detected from the captured image PB2, the generation unit 356 determines whether or not the identification information indicated by the identification image MD inside the detected outer frame image MF can be read. Next, if the identification information indicated by the identification image MD can be read and the read identification information matches the identification information included in the second photographed image MK2 stored in the state memory unit 364, the generation unit 356 selects the captured image PB2 as the second photographed image P2.

The generation unit 356 determines whether or not at least one of the first captured image P1 and the second captured image P2 has been selected (step SB4).

If the generation unit 356 determines that the first captured image P1 and the second captured image P2 could not be selected (step SB4: NO), it performs the process of step SB7.

On the other hand, if the generation unit 356 determines that at least one of the first captured image P1 and the second captured image P2 has been selected (step SB4: YES), it generates management data D1 (step SB5).

When the first captured image P1 and the second captured image P2 are selected, the generation unit 356 generates management data D1 including image data of the first captured image P1, image data of the second captured image P2, and the photographing device ID 362.
If the first captured image P1 can be selected but the second captured image P2 cannot be selected, the generation unit 356 generates management data D1 including image data of the first captured image P1 and the photographing device ID 362.
If the second captured image P2 can be selected but the first captured image P1 cannot be selected, the generation unit 356 generates management data D1 including image data of the second captured image P2 and the photographing device ID 362.

Next, the first communication control unit 351 transmits the management data D1 generated by the generation unit 356 to the item management server 5 (step SB6).

Next, the generation unit 356 erases the recording data 363 that was processed in steps SB2 and after from the second memory 360 (step SB7).

As shown in the flowchart FC, the server communication control unit 501 determines whether or not it has received management data D1 from the imaging device 3 (step SC1). If the server communication control unit 501 determines that it has received management data D1 (step SC1: YES), the server processing unit 502 updates the item management DB 512 based on the management data D1 received by the server communication control unit 501 (step SC2).

Step SC2 will now be described in detail.
The server processing unit 502 identifies a record from the item management DB 512 that includes the shooting device ID 362 included in the management data D1, and updates the image data of the first captured image P1 in the identified record to the image data of the first captured image P1 included in the management data D1.
The server processing unit 502 identifies a record from the item management DB 512 that includes the shooting device ID 362 included in the management data D1, and updates the image data of the second captured image P2 of the identified record to the image data of the second captured image P2 included in the management data D1.

[1-2-3. Application UI-related operations]
Next, the operation of the item management system 1 related to the application UI will be described.
20 is a flowchart showing the operations of the terminal device 4 and the item management server 5. In FIG.

As shown in the flowchart FD, the application execution unit 400 determines whether to request image data stored in the item management DB 512 from the item management server 5 (step SD1). For example, the terminal communication control unit 401 makes a positive determination in step SD1 when the item management application 411 is launched.

When the application execution unit 400 determines that image data stored in the item management DB 512 is to be requested (step SD1: YES), it transmits request data requesting the image data to the item management server 5 (step SD2). The request data includes the imaging device ID 362 stored in the terminal memory 41. The terminal device 4 acquires the imaging device ID 362 at a predetermined timing.

As shown in flow chart FE, the server communication control unit 501 receives the request data (step SE1). Next, the server processing unit 502 acquires image data corresponding to the imaging device ID 362 included in the request data received by the server communication control unit 501 from the item management DB 512 (step SE2). Next, the server communication control unit 501 transmits the image data acquired by the server processing unit 502 to the terminal device 4 as a response to the request data (step SE3).

Referring to the flowchart FD, the application execution unit 400 receives image data from the item management server 5 (step SD3). Next, the application execution unit 400 displays an application UI including an image indicated by the image data received in step SD3 on the touch panel 43 (step SD4).

[1-3. Effects, etc.]
As described above with reference to Figures 1 to 20, imaging system 7 in embodiment 1 comprises wide-angle camera 301 arranged on top surface 20A of refrigerator 2 to capture an image of refrigerator 2 having refrigerator compartment 21 and door 21A for opening and closing opening 201 of refrigerator compartment 21, object PS to be photographed that is arranged near opening 201 of refrigerator compartment 21 or on door 21A and whose image capture state by wide-angle camera 301 changes depending on the opening and closing angle of door 21A, state memory unit 364 that stores first image capture state CD1 which is the image capture state of object PS by wide-angle camera 301 at a predetermined opening and closing angle of door 21A, and generation unit 356 that selects first image capture image P1 which is the image captured when door 21A is in an open state from image capture image PB1 captured by wide-angle camera 301 based on the first image capture state CD1.

According to this, a first captured image P1 when the door 21A is in an open state is selected from captured images PB1 captured by the wide-angle camera 301 based on a first captured state CD1, which is a captured state of the photographed object PS by the wide-angle camera 301 at a predetermined opening and closing angle of the door 21A. Therefore, by appropriately setting the predetermined opening and closing angle, an image that appropriately captures the interior situation of the refrigerator 2 can be selected as the first captured image P1, which is an image captured when the door 21A is in an open state. The predetermined opening and closing angle is, for example, 60 degrees to 100 degrees.

In addition, in the imaging system 7, the object to be photographed PS1 is placed on the door 21A so that the imaging plane PF1 of the object to be photographed PS1 is approximately perpendicular to the imaging direction of the wide-angle camera 301 when the door 21A is at the specified opening and closing angle.

As a result, when the door 21A is at the predetermined opening/closing angle, the imaging plane PF1 of the object PS1 is approximately perpendicular to the imaging direction of the wide-angle camera 301, so that the object PS1 can be properly imaged by the wide-angle camera 301 when the door 21A is at the predetermined opening/closing angle.

In addition, in the imaging system 7, the photographed object PS1 is placed in the door pocket 21E of the door 21A, and includes a connecting member AT2 and an attachment member AT1 that fix the photographed object PS1 to the door pocket 21E.

By doing this, the photographed object PS1 is fixed to the door pocket 21E, so the photographed state by the wide-angle camera 301 changes depending on the opening and closing angle of the door 21A. Therefore, the photographed object PS1 can be positioned in an appropriate position.

The imaging system 7 also includes a right door hinge member 204 that rotatably supports the right door 21B, and the photographed object PS2 is configured as a hinge cover AS of the right door hinge member 204.

According to this, the photographed object PS2 is configured as the hinge cover AS of the right door hinge member 204, so the photographed state by the wide-angle camera 301 changes depending on the opening and closing angle of the door 21A. Therefore, the photographed object PS2 can be positioned in an appropriate position.

In addition, in the imaging system 7, a notch AS1 is formed in the hinge cover AS, and the notch AS1 fits into the upper part of the frame of the drawer 23A below the right door.

As a result, the notch AS1 formed in the hinge cover AS fits into the upper part of the frame of the lower drawer 23A of the right door, so that the hinge cover AS can be fixed to the right door hinge member 204. Therefore, the photographed object PS2 can be positioned in the appropriate position.

In addition, in the imaging system 7, a first photographed subject image MK1 is formed on the imaging surface PF1 of the photographed subject PS1, the first photographed subject image MK1 including identification information indicating at least one of the type of item stored in the refrigerator compartment 21 and the refrigerator compartment 21 in which the item is stored, the state memory unit 364 stores the first photographed subject image MK1, and the generation unit 356 selects a first photographed image P1 when the door 21A is in the open state from the photographed image PB1 captured by the wide-angle camera 301 based on the first photographed subject image MK1 stored in the state memory unit 364.

According to this, the first captured image P1 can be appropriately selected by, for example, selecting an image PB1 in which the first subject image MK1 is appropriately captured from the captured images PB1 captured by the wide-angle camera 301 as the first captured image P1.

In addition, in the imaging system 7, the first subject image MK1 includes a rectangular outer frame image MF and an identification image MD formed inside the outer frame image MF and corresponding to identification information indicating the type of item stored in the refrigerator compartment 21 of the refrigerator 2.

According to this, the first captured image P1 can be appropriately selected by selecting, for example, an image PB1 in which the outer frame image MF and the identification image MD are appropriately captured from the captured images PB1 captured by the wide-angle camera 301 as the first captured image P1.

In addition, in the imaging system 7, the identification image MD is composed of a first rectangular image PD1 in a first color and a second rectangular image PD2 in a second color different from the first color, and the arrangement pattern of the first rectangular image PD1 and the second rectangular image PD2 indicates the identification information.

By appropriately setting each of the first and second colors, it is possible to accurately detect the identification information indicated by the arrangement pattern of the first rectangular image PD1 and the second rectangular image PD2.

In addition, in the imaging system 7, the size of the first rectangular image PD1 is the same as the size of the second rectangular image PD2, and the first rectangular image PD1 and the second rectangular image PD2 are arranged along the longitudinal direction of the outer frame image MF.

With this, the size of the first rectangular image PD1 is the same as the size of the second rectangular image PD2, so the identification information can be efficiently expressed by the arrangement pattern of the first rectangular image PD1 and the second rectangular image PD2. Furthermore, since the first rectangular image PD1 and the second rectangular image PD2 are arranged along the longitudinal direction of the outer frame image MF, the first rectangular image PD1 and the second rectangular image PD2 can be arranged compactly.

In addition, in the imaging system 7, the size of the first rectangular image PD1 and the size of the second rectangular image PD2 are determined based on the resolution of the wide-angle camera 301 and the distance between the wide-angle camera 301 and the first target image MK1 at a specified opening and closing angle of the door 21A.

According to this, for example, the higher the resolution of the wide-angle camera 301, the smaller the size of the first rectangular image PD1 and the size of the second rectangular image PD2. Also, for example, the longer the distance between the wide-angle camera 301 and the first subject image MK1 at a specified opening/closing angle of the door 21A, the larger the size of the first rectangular image PD1 and the size of the second rectangular image PD2. In this way, the size of the first rectangular image PD1 and the size of the second rectangular image PD2 can be appropriately determined.

In addition, in the imaging system 7, the generation unit 356 can detect the outer frame image MF included in the captured image PB1 captured by the wide-angle camera 301, and selects the captured image PB1 as the first captured image P1 when the identification information indicated by the identification image MD of the first subject image MK1 included in the captured image PB1 captured by the wide-angle camera 301 matches the identification information indicated by the identification image MD of the first subject image MK1 stored in the state memory unit 364.

According to this, if the outer frame image MF included in the captured image PB1 captured by the wide-angle camera 301 can be detected, the captured image PB1 is an appropriately captured image. Furthermore, if the identification information indicated by the identification image MD of the first subject image MK1 included in the captured image PB1 matches the identification information indicated by the identification image MD of the first subject image MK1 stored in the state memory unit 364, the captured image PB1 is an even more appropriately captured image. Therefore, the first captured image P1 can be easily and appropriately selected.

In addition, in the imaging system 7, when the generation unit 356 determines that the wide-angle camera 301 or the narrow-angle camera 302 has captured an image of the subject PS, it notifies the user P that the wide-angle camera 301 or the narrow-angle camera 302 has captured an image of the subject PS.

By this, the user P is notified that the wide-angle camera 301 or the narrow-angle camera 302 has captured an image of the subject PS, so the user P can easily recognize that the wide-angle camera 301 or the narrow-angle camera 302 has captured an image of the subject PS. This improves the convenience for the user P.

The imaging device 3 in the first embodiment also includes a wide-angle camera 301 arranged on the top surface 20A of the refrigerator 2 to capture an image of the refrigerator 2 having a refrigeration compartment 21 and a door 21A that opens and closes the opening 201 of the refrigeration compartment 21, a state storage unit 364 that stores a first capture state CD1, which is a capture state of the object PS1 captured by the wide-angle camera 301 at a predetermined opening and closing angle of the door 21A, and a generation unit 356 that selects a first captured image P1, which is a capture image when the door 21A is in an open state, from the captured image PB1 captured by the wide-angle camera 301 based on the first capture state CD1. The object PS1 is arranged near the opening 201 of the refrigeration compartment 21 or on the door 21A, and the capture state captured by the wide-angle camera 301 changes depending on the opening and closing angle of the door 21A.

As a result, the imaging device 3 in embodiment 1 achieves the same effect as the imaging system 7 in embodiment 1.

(Embodiment 2)
Next, a description will be given of embodiment 2. In the description of embodiment 2, the same components as those of each part of the item management system 1 of embodiment 1 will be given the same reference numerals and detailed description will be omitted as appropriate.

[2-1. Configuration]
FIG. 21 is a diagram showing the configuration of an item management system 1A in the second embodiment.
The item management system 1A in the second embodiment includes a refrigerator 2A, an item management server 5, and a terminal device 4. The refrigerator 2 and the image capturing device 3 in the first embodiment described above are separate entities, but the refrigerator 2A in the second embodiment includes an image capturing device 3A. That is, the refrigerator 2A and the image capturing device 3A in the present embodiment are not separate entities.

FIG. 22 is a block diagram showing the configuration of a refrigerator 2A in this embodiment.
The refrigerator 2A includes a third control board 220. The third control board 220 includes a third processor 230 such as a CPU, a third memory 240, a third communication module 250, and an interface circuit to which other devices and sensors are connected.

The third memory 240 is a storage device that stores programs and data executed by the third processor 230. The third memory 240 is configured from a non-volatile storage device. The third memory 240 may also include a volatile storage device that configures the work area of the third processor 230. The third memory 240 stores data processed by the third processor 230 and a control program 241 executed by the third processor 230.

The third communication module 250 has communication hardware such as a communication circuit and a connector that conforms to a specified wired communication standard, and communicates with the first control board 311A under the control of the third processor 230.

The third control board 220 is connected to a second wireless communication module 206, a cooling unit 207, a first opening/closing sensor 208, a second opening/closing sensor 209, a third opening/closing sensor 210, a fourth opening/closing sensor 211, a fifth opening/closing sensor 212, and a sixth opening/closing sensor 213.

The cooling unit 207 includes mechanisms for cooling each storage compartment of the refrigerator 2A, such as a compressor, a condenser, a capillary tube, a cooler, a cooling fan that sends the cold air generated by the cooler to each storage compartment, and a damper that divides the cold air sent by the cooling fan. The cooling unit 207 cools each storage compartment of the refrigerator 2A according to the control of the third control board 220.

The first open/close sensor 208 to the sixth open/close sensor 213 are each formed of, for example, a Hall IC.
The first opening/closing sensor 208 is a sensor for detecting whether the right door 21B is opened or closed.
The second opening/closing sensor 209 is a sensor for detecting whether the left door 21C is opened or closed.
The third open/close sensor 210 is a sensor for detecting whether the drawer 22A is open or closed.
The fourth open/close sensor 211 is a sensor for detecting whether the drawer 23A is open or closed.
The fifth open/close sensor 212 is a sensor for detecting whether the drawer 24A is open or closed.
The sixth opening/closing sensor 213 is a sensor for detecting whether the drawer 25A is opened or closed.
Each opening/closing sensor outputs a detection value to the third control board 220 .

The third processor 230 reads and executes the control program 241 to communicate with an external device via the second wireless communication module 206 .
In addition, the third processor 230 reads and executes the control program 241, thereby cooling each storage compartment of the refrigerator 2A by the cooling unit 207.
Furthermore, the third processor 230 reads and executes the control program 241 to transmit the detection values output by each of the first opening/closing sensor 208 to the sixth opening/closing sensor 213 to the first control board 311A.

Refrigerator 2A is equipped with imaging device 3A. Compared to imaging device 3, imaging device 3A does not have first photosensor 303, second photosensor 304, thermopile 305, human presence sensor 306, second LED 308, and third LED 309.

The object PS to be photographed is placed near the opening of the refrigeration compartment 21 of the refrigerator 2A, on the right door 21B, or on the left door 21C, and the state photographed by the wide-angle camera 301 changes depending on the opening and closing angles of the right door 21B and the left door 21C.
Furthermore, the subject PS constitutes part of the refrigerator 2A according to the present disclosure.

The imaging device 3A in the second embodiment has a first control board 311A instead of the first control board 311. The power button 32, the first LED 33, and the illuminance sensor 307 are connected to the first control board 311A. The imaging device 3A in the second embodiment also has a second control board 312A instead of the second control board 312. The wide-angle camera 301, the narrow-angle camera 302, and the first wireless communication module 313 are connected to the second control board 312A.

The first control board 311A is installed in the imaging device 3A at a position similar to that of the first control board 311 in the imaging device 3.
Further, the second control board 312A is installed in the imaging device 3A at a position similar to that of the second control board 312 in the imaging device 3.
In this embodiment, the wide-angle camera 301, the narrow-angle camera 302, the power button 32, the first LED 33, the illuminance sensor 307, and the first wireless communication module 313 are installed in the same positions as those in the first embodiment.

The first control board 311A includes a first processor 320A such as a CPU, a first memory 330A, a first communication module 340A, and an interface circuit to which other devices and sensors are connected.

The first memory 330A is a memory similar to the first memory 330. The first memory 330A stores data processed by the first processor 320A and the control program 331A executed by the first processor 320A.

The first communication module 340A has communication hardware such as a communication circuit and a connector that conforms to a specific wired communication standard, and communicates with the second control board 312A and the third control board 220 via cables.

The first processor 320A reads and executes the control program 331A to turn the power of the imaging device 3A on and off in response to the operation of the power button 32.
The first processor 320A reads and executes the control program 331A to turn the first LED 33 on and off.
The first processor 320A reads and executes the control program 331A, thereby transmitting the detection value output by the illuminance sensor 307 to the second control board 312A via the first communication module 340A.
The first processor 320A reads and executes the control program 331A, and transmits the detection values of the first opening/closing sensor 208 to the sixth opening/closing sensor 213 received from the third control board 220 to the second control board 312A via the first communication module 340A.

The second control board 312A includes a second processor 350A such as a CPU, a second memory 360A, a second communication module 370A, and an interface circuit to which other devices and sensors are connected.

The second memory 360A is a memory similar to the first memory 330. The second memory 360A stores data processed by the second processor 350A and a control program 361A executed by the second processor 350A.

The second communication module 370A has communication hardware such as a communication circuit and a connector that conforms to a specific communication standard, and communicates with the first control board 311A under the control of the second processor 350A.

The second processor 350A reads out and executes the control program 361A stored in the second memory 360A, thereby functioning as the first communication control unit 351, the second communication control unit 352A, the open/close determination unit 353A, the shooting control unit 354, the recording control unit 355, and the generation unit 356. The second processor 350A also reads out and executes the control program 361A stored in the second memory 360A, thereby causing the second memory 360A to function as a state storage unit 364.

The refrigerator 2A in the second embodiment includes a wide-angle camera 301, an object to be photographed PS, a state memory unit 364, and a generation unit 356.

The second communication control unit 352A communicates with the first control board 311A via the second communication module 370A. The second communication control unit 352A receives detection values of various sensors from the first control board 311A.

The open/close determination unit 353A determines whether the right door 21B is open or closed based on the detection value of the first open/close sensor 208 received by the second communication control unit 352A. When the second communication control unit 352A receives a detection value different from the detection value corresponding to the closed state, the open/close determination unit 353 determines that the right door 21B is open. On the other hand, when the second communication control unit 352A receives a detection value corresponding to the closed state, the open/close determination unit 353 determines that the right door 21B is closed.

The opening/closing determination unit 353A determines whether the left door 21C is open or closed based on the detection value of the second opening/closing sensor 209 by the second communication control unit 352, in the same manner as the determination for the right door 21B.

The open/close determination unit 353A determines whether the drawer 26A is open or closed based on the detection values of the third open/close sensor 210 to the sixth open/close sensor 213 received by the second communication control unit 352. The open/close determination unit 353A performs the same determination as for the right door 21B.

[2-2. motion]
The operation of each part of the article management system 1A in the second embodiment is the same as the operation of each part of the article management system 1 in the first embodiment. The operation of the imaging device 3A is the same as that of the imaging device 3. Moreover, the operation of the imaging device 3A relating to the management data D1 is the same as that of the imaging device 3 in the first embodiment.

[2-3. Effects, etc.]
As described above with reference to Figures 21 and 22, refrigerator 2A in embodiment 2 comprises wide-angle camera 301 arranged on top surface 20A of refrigerator 2 to photograph refrigerator 2 having refrigerator compartment 21 and door 21A which opens and closes opening 201 of refrigerator compartment 21, object PS which is arranged near opening 201 of refrigerator compartment 21 or on door 21A and whose photographing state by wide-angle camera 301 changes depending on the opening and closing angle of door 21A, state memory unit 364 which stores first photographing state CD1 which is the photographing state of object PS by wide-angle camera 301 at a predetermined opening and closing angle of door 21A, and generation unit 356 which selects first photographed image P1 which is the photographed image when door 21A is in an open state from photographed image PB1 photographed by wide-angle camera 301 based on the first photographing state CD1.

As a result, the refrigerator 2A in the second embodiment achieves the same effects as the imaging system 7 in the first embodiment.

Other Embodiments
As described above, the above-mentioned first and second embodiments have been described as examples disclosed in the present application. However, the technology in the present disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. have been made. In addition, it is also possible to combine the components described in the above-mentioned first and second embodiments to form new embodiments. Therefore, other embodiments will be exemplified below.

In the above-mentioned first and second embodiments, the case where the photographed object PS1 is placed in the door pocket 21E of the right door 21B is described, but the photographed object PS1 may also be placed in the door pocket 21E of the left door 21C.

In the above-mentioned first and second embodiments, the case where the photographed object PS1 is placed in the door pocket 21E provided at the top of the door 21A has been described, but it is sufficient that the photographed object PS1 is placed in the door pocket 21E of the door 21A. For example, the photographed object PS1 may be placed in the door pocket 21E that is the second or third door pocket from the top of the door 21A.

In the above-mentioned first and second embodiments, the case where the photographed object PS1 is fixed to the side member 21S of the door pocket 21E of the door 21A has been described, but the photographed object PS1 may be placed in the door pocket 21E of the door 21A. For example, the photographed object PS1 may be fixed to a front member. The front member is the member on the positive side of the Y axis of the door pocket 21E in FIG. 5, i.e., the front side. Also, for example, the photographed object PS1 may be fixed to a bottom member. The bottom member is the member on the negative side of the Z axis of the door pocket 21E in FIG. 5, i.e., the lower side.

In the above-described first and second embodiments, the identification image MD is composed of the information image MDA and the code image MDB, but the identification image MD may be composed of only the information image MDA. In other words, the identification image MD does not have to include the code image MDB.

In the above-described first and second embodiments, the first rectangular image PD1 and the second rectangular image PD2 are described as being square, but the first rectangular image PD1 and the second rectangular image PD2 may also be rectangular.

In the above-described first and second embodiments, the outer frame image MF is formed in black, but is not limited thereto. For example, the outer frame image MF may be formed in blue. In addition, the first color is white and the second color is black, but is not limited thereto. For example, the first color may be yellow and the second color may be blue.
In order for the generating unit 356 to easily detect the outer frame image MF from each captured image PB1 included in the moving image indicated by the first recording data DV1, it is preferable that the color of the outer frame image MF is different from the color of the inner surface of the refrigeration compartment 21 and the color of the items stored in the refrigeration compartment 21. Thus, the color of the outer frame image MF is preferably, for example, black, blue, or the like.

In the above-described second embodiment, the first photosensor 303, the second photosensor 304, the thermopile 305, the human sensor 306, the second LED 308, and the third LED 309 are not included. In other embodiments related to the second embodiment, any of these devices may be included.

In the above-described first and second embodiments, the right door 21B and the left door 21C have different widths. In other embodiments, the right door 21B and the left door 21C may have the same width, or the left door 21C may have a greater width than the right door 21B.

In the above-described first and second embodiments, the imaging device 3, 3A is configured to include the second LED 308 and the third LED 309. In other embodiments, the second LED 308 and the third LED 309 may be configured as a single LED.

In the above-mentioned second embodiment, the first opening/closing sensor 208 to the sixth opening/closing sensor 213 are configured with Hall ICs. However, the first opening/closing sensor 208 and the second opening/closing sensor 209 may be other sensors such as an angle sensor, a camera, or a temperature sensor. Also, the third opening/closing sensor 210 to the sixth opening/closing sensor 213 may be other sensors such as an acceleration sensor, a mechanical switch, an angle sensor, a camera, or a temperature sensor.

In another embodiment, the imaging device 3 may include a subject detection unit. In this case, the recording control unit 355 may control the start and end of recording based on the subject detection unit being able to detect the subject. This can reduce the volume of recorded data and improve the processing speed.
The subject detection unit corresponds to an example of a "selection unit."
The operation of the imaging device 3 relating to recording based on the object to be photographed will be described below.

The open/close determination unit 353 determines whether the door 21A or the drawer 26A is open (step SF1).

If the open/close determination unit 353 determines that the door 21A or the drawer 26A is not open (step SF1: NO), the second processor 350 returns the process to step SF1.

If the open/close determination unit 353 determines that the door 21A or the drawer 26A is open (step SF1: YES), the photography control unit 354 causes the wide-angle camera and the narrow-angle camera 302 to start photographing (step SF2).

Next, the subject detection unit determines whether or not the subject has been detected (step SF3).

If the subject detection unit determines that the subject cannot be detected (step SF3: NO), the second processor 350 proceeds to step SF7.

If the subject detection unit determines that the subject has been detected (step SF3: YES), the recording control unit 355 starts recording the results of the wide-angle camera 301 and the narrow-angle camera 302 (step SF4). It is to be noted that recording of the results of the wide-angle camera 301 alone may be started based on the subject corresponding to the wide-angle camera 301. Similarly, recording of the results of the narrow-angle camera 302 alone may be started based on the subject corresponding to the narrow-angle camera 302. In addition, if the subject detection unit determines that the subject has been detected, the subject detection unit may notify the user that the subject has been detected (recording of the photographing results has started). In this way, the user is notified that the subject has been detected (recording of the photographing results has started), so that the user can easily recognize that recording of the photographing results has started. This leads to a reduction in noise in the photographing, and improves the user's convenience.

After the recording control unit 355 starts recording the images captured by the wide-angle camera 301 and the narrow-angle camera 302, the photographed object detection unit determines whether or not the photographed object has been detected (step SF5).

If the photographed object detection unit determines that the photographed object has been detected (step SF5: YES), it performs the determination in step SF5 again.

On the other hand, if the subject detection unit determines that the subject could not be detected (step SF5: NO), the recording control unit 355 ends the recording of the shooting results (step SF6). As a result, the recording data 363 is generated in the second memory 360. Note that the recording data 363 may include multiple recording data. For example, multiple recording data may be generated in the second memory 360 in one shooting. In this case, the generation unit 356 may perform the processes of steps SB1 to SB7 for each of the multiple recording data. Also, if the subject detection unit determines that the subject could not be detected, the subject detection unit may notify the user P that the subject could not be detected (the recording of the shooting results has ended). In this way, the user P is notified that the subject could not be detected (the recording of the shooting results has ended), so that the user P can easily recognize that the recording of the shooting results has ended. Therefore, the convenience of the user P can be improved.

Next, the open/close determination unit 353 determines whether the door 21A and the drawer 26A are closed (step SF7). If the open/close determination unit 353 determines that the door 21A and the drawer 26A are not closed (step SF7: NO), the second processor 350 returns the process to step SF3.

On the other hand, if the open/close determination unit 353 determines that the door 21A and the drawer 26A are closed (step SF7: YES), the photography control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to end photography (step SF8).

In another embodiment, the recording control unit 355 may end the recording of the captured image a predetermined time (e.g., one minute) after the start of recording the captured image. This can also be used to deal with malfunctions of the image capture device 3.

In another embodiment, the generation unit 356 may compare the image captured by the camera with pre-stored images of the door in an open state and select an image of the door in an open state.

In another embodiment, the photographing device 3 may include a photographing device confirmation unit. The photographing device confirmation unit is for confirming that the photographing device 3 has been installed in a predetermined position of the refrigerator 2 when the user installs the photographing device 3 in the refrigerator 2. In this case, when the photographing device confirmation unit detects an object to be photographed within a pre-stored area, the photographing device confirmation unit may notify the user that the photographing device 3 has been installed in a predetermined position of the refrigerator 2 (for example, the center position in the left-right direction of the refrigerator 2) by turning on or blinking an LED, or by issuing a sound or a notification to the terminal device 4. This allows the user to easily recognize that the photographing device 3 has been installed in the predetermined position of the refrigerator 2 and to move on to the next setting task. This improves user convenience.
Furthermore, when the photographing object cannot be detected within the area stored in advance, the photographing device confirmation unit may notify the user that the photographing device 3 has not been installed at the predetermined position of the refrigerator 2 by turning on or blinking an LED, or by issuing a sound, or by notifying the terminal device 4. This allows the user to easily recognize that the photographing device 3 has not been installed at the predetermined position of the refrigerator 2. This improves the user's convenience. Furthermore, the photographing device confirmation unit may notify the user in a manner different from that when the photographing device 3 has been installed at the predetermined position of the refrigerator 2 and that when the photographing device 3 has not been installed at the predetermined position of the refrigerator 2. For example, the photographing device confirmation unit may turn on an LED when the photographing device 3 has been installed at the predetermined position of the refrigerator 2, and may blink the LED when the photographing device 3 has not been installed at the predetermined position of the refrigerator 2. This allows the user to easily recognize that the photographing device 3 has been installed at the predetermined position of the refrigerator 2 and that when the photographing device 3 has not been installed at the predetermined position of the refrigerator 2, this improves the user's convenience.

In another embodiment, the photographing device 3 may include a photographed object confirmation unit. The photographed object confirmation unit is for confirming that the photographed object has been placed in the specified position of the refrigerator 2 when the user places the photographed object in the refrigerator 2. In this case, when the information of the photographed object (information obtained from the photographed object) matches the information of the storage room stored in advance, the photographed object confirmation unit may notify the user that the photographed object has been placed in the specified position of the refrigerator 2 by turning on or blinking an LED, or by issuing a sound or a notification to the terminal device 4. This allows the user to easily recognize that the photographed object has been placed in the specified position of the refrigerator 2 and to move on to the next setting task. This improves user convenience.
In addition, when the information of the photographed object (information obtained from the photographed object) does not match the information of the storage room stored in advance, the photographed object confirmation unit may notify the user that the photographed object has not been placed in the specified position of the refrigerator 2 by lighting or blinking an LED, or by sound, or by notifying the terminal device 4 (for example, by displaying information about the photographed object). This allows the user to easily recognize that the photographed object has not been placed in the specified position of the refrigerator 2. This improves the user's convenience. In addition, the photographed object confirmation unit may notify the user in a different manner depending on whether the photographed object has been placed in the specified position of the refrigerator 2 or has not been placed in the specified position of the refrigerator 2. For example, the LED may be turned on when the photographed object has been placed in the specified position of the refrigerator 2, and the LED may be blinked when the photographed object has not been placed in the specified position of the refrigerator 2. This allows the user to easily recognize that the photographed object has been placed in the specified position of the refrigerator 2 and that the photographed object has not been placed in the specified position of the refrigerator 2. This improves the user's convenience.

In other embodiments, the object to be photographed may include text, illustrations, or the like that provide information about the storage compartment in which it should be placed (for example, the text "refrigerator compartment" or an illustration of a refrigerator compartment). This allows the user to easily identify the storage compartment in which the object to be photographed should be placed when placing it in the refrigerator 2. This can therefore reduce installation errors by the user, improving user convenience.

In other embodiments, in addition to the functional units of the second processor 350, 350A in embodiments 1 and 2, the second processor 350, 350A may function as a detection unit that detects an item. During the time from when the recording control unit 355 starts recording with the wide-angle camera 301 and the narrow-angle camera 302 to when it ends the recording, the detection unit may cut out an image of an item being put in or taken out of the refrigerator compartment 21 from the video. Also, if a marker is attached to the item, the detection unit may detect the item with the marker.

In another embodiment, in addition to the functional units of second processor 350 in embodiment 1, second processor 350 may function as a person presence determination unit. The person presence determination unit determines whether or not a person is present around refrigerator 2 based on the detection value of human presence sensor 306. More specifically, when the detection value of human presence sensor 306 corresponds to the presence of a person, the person presence determination unit determines that a person is present around refrigerator 2, and when not, determines that no person is present around refrigerator 2.
When the second processor 350 functions as a person presence determination unit, the shooting control unit 354 causes the wide-angle camera 301 and the narrow-angle camera 302 to start shooting when the person presence determination unit determines that a person is present, and causes the wide-angle camera 301 and the narrow-angle camera 302 to end shooting when the person presence determination unit determines that a person is not present. Also, when the second processor 350 functions as a person presence determination unit, the recording control unit 355 starts recording when the person presence determination unit determines that a person is present, and ends recording when the person presence determination unit determines that a person is not present. This allows the shooting results of the wide-angle camera 301 and the narrow-angle camera 302 to be appropriately recorded even if the opening/closing determination unit 353 makes an erroneous determination.

In another embodiment, in addition to the functional units of the second processor 350, 350A in the first and second embodiments, the second processor 350, 350A may function as an entry/exit determination unit. In this case, the entry/exit determination unit changes the boundary line used to determine entry/exit depending on the opening/closing angle of the left door 21C and the right door 21B, and determines whether an item is entry or exit. Also, the entry/exit determination unit changes the boundary line used to determine entry/exit depending on the degree to which the drawer 26A is pulled out, and determines whether an item is entry or exit.

In another embodiment, the first captured image P1 may be corrected and uploaded to the item management server 5. An example of image correction is viewpoint conversion correction.

In the above-described embodiment, the first captured image P1 and the second captured image P2 are used for display purposes on the terminal device 4. However, the first captured image P1 and the second captured image P2 may be used for other purposes, such as detecting items stored in the refrigerators 2 and 2A.

In other embodiments, the main box 20 of the refrigerator 2, 2A may have a number of storage compartments other than the five types described above, or may have fewer types. Also, in other embodiments, the number of doors provided at the opening 201 of the refrigerator compartment 21 may be one.

The first processor 320, 320A, the second processor 350, 350A, the server processor 50, and the terminal processor 40 may be configured with a single processor or with multiple processors. These processors may be hardware programmed to realize the corresponding functional units. That is, these processors may be configured with, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The configurations of the imaging devices 3, 3A, refrigerators 2, 2A, terminal devices 4, and item management server 5 shown in Figures 8, 17, and 22 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware that corresponds to each unit individually, and it is also possible to implement a configuration in which one processor executes a program to realize the functions of each unit. Also, some of the functions realized by software in the above-mentioned embodiments may be hardware, or some of the functions realized by hardware may be software.

The step units of the operations shown in Figures 18, 19, 20, and 23 are divided according to the main processing content in order to make the operations easier to understand, and the operation is not limited by the way in which the processing units are divided or their names. The operations may be divided into more step units depending on the processing content. Furthermore, one step unit may be divided so that it contains even more processing. Furthermore, the order of the steps may be changed as appropriate within the scope of the purpose of this disclosure.

The above-described embodiments are intended to illustrate the technology disclosed herein, and various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents.

(Additional Note)
The above description of the embodiments discloses the following techniques.

(Technology 1) An imaging system comprising: a camera that photographs a refrigerator having a storage compartment and an opening/closing body that opens and closes an opening of the storage compartment and is arranged on a top surface of the refrigerator; an object to be photographed that is arranged near the opening of the storage compartment or on the opening/closing body and whose photographing state by the camera changes depending on the opening/closing angle of the opening/closing body; a state memory unit that stores the photographing state of the object to be photographed by the camera at a predetermined opening/closing angle of the opening/closing body; and a selection unit that selects, from images photographed by the camera, an image to be photographed when the opening/closing body is in an open state, based on the photographing state of the object to be photographed by the camera stored in the state memory unit.
According to this configuration, an image captured when the opening/closing body is in the open state is selected from images captured by the camera based on the state of the object captured by the camera at a predetermined opening/closing angle of the opening/closing body. Therefore, by appropriately setting the predetermined opening/closing angle, an image that appropriately captures the interior state of the refrigerator can be selected as the image captured when the opening/closing body is in the open state.

(Technology 2) The imaging system according to Technology 1, wherein the object is placed on the opening/closing body so that the imaging surface of the object is approximately perpendicular to the imaging direction of the camera at the specified opening/closing angle.
With this configuration, when the opening/closing body is at the specified opening/closing angle, the imaging surface of the object to be photographed is approximately perpendicular to the imaging direction of the camera, so that the object to be photographed can be properly imaged by the camera when the opening/closing body is at the specified opening/closing angle.

(Technology 3) The imaging system according to Technology 2, wherein the object to be photographed is placed in a door pocket of the opening/closing body, and the imaging system further includes a fixing member that fixes the object to the door pocket.
According to this configuration, the object to be photographed is fixed to the door pocket of the opening/closing body, so that the state of the object to be photographed by the camera changes depending on the opening/closing angle of the opening/closing body, and therefore the object to be photographed can be placed in an appropriate position.

(Technology 4) The imaging system according to Technology 1, further comprising a hinge member that rotatably supports the opening/closing body, and the object to be photographed is configured as a hinge cover of the hinge member.
According to this configuration, the object to be photographed is configured as a hinge cover of the hinge member, so that the state of the object to be photographed by the camera changes depending on the opening and closing angle of the opening and closing body. Therefore, the object to be photographed can be placed in an appropriate position.

(Technology 5) An imaging system according to Technology 4, wherein the hinge cover has a cutout formed therein, and the cutout fits into an upper portion of a frame of a drawer below the storage chamber.
According to this configuration, the notch formed in the hinge cover fits into the upper part of the frame of the drawer below the storage chamber, so that the hinge cover can be fixed to the hinge member, and therefore the subject can be positioned in an appropriate position.

(Technology 6) An imaging system described in any one of Technology 1 to Technology 5, wherein an image of a subject to be photographed is formed on an imaging surface of the subject to be photographed, the image corresponding to identification information indicating at least one of the type of item stored in the storage chamber and the storage chamber in which the item is stored, the state memory unit stores the image of the subject to be photographed, and the selection unit selects an image of the subject to be photographed when the opening/closing body is in an open state from the images photographed by the camera based on the image of the subject to be photographed stored in the state memory unit.
According to this configuration, by selecting, from the images captured by the camera, for example, an image in which the image of the subject to be photographed is appropriately captured as the image captured when the opening/closing body is in the open state, it is possible to appropriately select the image captured when the opening/closing body is in the open state.

(Technology 7) The imaging system described in Technology 6, wherein the subject image includes a rectangular outer frame image and an identification image formed inside the outer frame image and corresponding to identification information indicating a type of item stored in the storage compartment of the refrigerator.
According to this configuration, by selecting, from the images captured by the camera, for example, an image in which the outer frame image and the identification image are appropriately captured, as the image captured when the opening/closing body is in the open state, it is possible to appropriately select the image captured when the opening/closing body is in the open state.

(Technology 8) An imaging system described in Technology 7, wherein the identification image is composed of a first rectangular image of a first color and a second rectangular image of a second color different from the first color, and the arrangement pattern of the first rectangular image and the second rectangular image indicates the identification information.
According to this configuration, by appropriately setting each of the first color and the second color, it is possible to accurately detect the identification information indicated by the arrangement pattern of the first rectangular image and the second rectangular image.

(Technology 9) An imaging system described in Technology 8, wherein the size of the first rectangular image is the same as the size of the second rectangular image, and the first rectangular image and the second rectangular image are arranged along the longitudinal direction of the outer frame image.
According to this configuration, since the size of the first rectangular image is the same as the size of the second rectangular image, the identification information can be efficiently expressed by the arrangement pattern of the first rectangular image and the second rectangular image. Furthermore, since the first rectangular image and the second rectangular image are arranged along the longitudinal direction of the outer frame image, the first rectangular image and the second rectangular image can be arranged compactly.

(Technology 10) An imaging system described in Technology 8 or Technology 9, in which the size of the first rectangular image and the size of the second rectangular image are determined based on the resolution of the camera and the distance between the camera and the image of the subject to be captured at a predetermined opening and closing angle of the opening and closing body.
According to this configuration, for example, the higher the resolution of the camera, the smaller the size of the first rectangular image and the size of the second rectangular image are made. Also, for example, the longer the distance between the camera and the image to be photographed at a predetermined opening/closing angle of the opening/closing body, the larger the size of the first rectangular image and the size of the second rectangular image are made. In this way, the size of the first rectangular image and the size of the second rectangular image can be appropriately determined.

(Technology 11) An imaging system described in any one of Technology 7 to Technology 10, wherein the selection unit is capable of detecting the outer frame image included in the image captured by the camera, and when identification information indicated by an identification image of the image to be captured included in the image captured by the camera matches identification information indicated by the identification image of the image to be captured stored in the state memory unit, selects the image captured by the camera as the image to be captured when the opening/closing body is in the open state.
According to this configuration, when the outer frame image included in the image captured by the camera can be detected, the image captured by the camera is an appropriately captured image. Also, when the identification information indicated by the identification image of the image to be captured included in the image captured by the camera matches the identification information indicated by the identification image of the image to be captured stored in the state storage unit, the image captured by the camera is an even more appropriately captured image. Therefore, the captured image when the opening/closing body is in the open state can be easily and appropriately selected.

(Technology 12) An imaging system described in any one of Technology 1 to Technology 11, wherein the selection unit notifies a user that the camera has photographed the subject when it determines that the camera has photographed the subject.
According to this configuration, when it is determined that the camera has photographed the subject, the camera notifies the user that the camera has photographed the subject, so that the user can easily recognize that the camera has photographed the subject, thereby improving user convenience.

(Technology 13) An imaging device that photographs a refrigerator having a storage compartment and an opening/closing body that opens and closes an opening of the storage compartment, and that comprises: a camera arranged on a top surface of the refrigerator; a state memory unit that stores a state of an object to be photographed by the camera at a predetermined opening/closing angle of the opening/closing body; and a selection unit that selects an image taken when the opening/closing body is in an open state from images photographed by the camera based on the state of the object to be photographed by the camera stored in the state memory unit, wherein the object to be photographed is arranged near the opening of the storage compartment or on the opening/closing body, and the state of the object to be photographed by the camera changes depending on the opening/closing angle of the opening/closing body.
According to this configuration, the same effects as those of the imaging system described in the first technique can be achieved.

(Technology 14) A refrigerator comprising: a camera that photographs a refrigerator having a storage compartment and an opening/closing body that opens and closes an opening of the storage compartment, and that is arranged on a top surface of the refrigerator; an object to be photographed that is arranged near the opening of the storage compartment or on the opening/closing body and whose photographed state by the camera changes depending on the opening/closing angle of the opening/closing body; a state memory unit that stores the photographed state of the object to be photographed by the camera at a predetermined opening/closing angle of the opening/closing body; and a selection unit that selects an image to be photographed when the opening/closing body is in an open position from images photographed by the camera, based on the photographed state of the object to be photographed by the camera stored in the state memory unit.
According to this configuration, the same effects as those of the imaging system described in the first technique can be achieved.

As described above, the imaging system, imaging device, and refrigerator according to the present invention can be used to select an image that appropriately captures the interior condition of the refrigerator from images captured by a camera.

1, 1A Item management system 2, 2A Refrigerator 3, 3A Imaging device 4 Terminal device 5 Item management server 7 Imaging system 21 Refrigerator (storage room)
21A Door (opening and closing body)
21B Right door 21C Left door 21E Door pocket 201 Opening 20A Top surface (upper surface)
204 Right door hinge member (hinge member)
301 Wide-angle camera (camera)
302 Narrow-angle camera 350, 350A Second processor 351 First communication control unit 352, 352A Second communication control unit 353, 353A Open/close determination unit 354 Shooting control unit 355 Recording control unit 356 Generation unit (selection unit)
360, 360A Second memory 361, 361A Control program 362 Photographing device ID
363 Recording data 364 State memory unit AS Hinge cover AS1 Cutout AT Imaged member AT1 Flat member AT2 Connecting member (part of fixed member)
AT3 Mounting material (part of the fixing material)
CD1 First shooting state (shooting state of the object to be shot by the camera at a predetermined opening/closing angle of the opening/closing body)
CD2 Second shooting state D1 Management data DV1 First recording data DV2 Second recording data MK1 First shooting target image (shooting target image)
MK2: Second photographed object image MD: Identification image MDA: Information image MDB: Code image PD1: First rectangular image PD2: Second rectangular image MF: Outer frame image P1: First photographed image (photographed when the opening/closing body is in the open state)
P2: Second captured image PB1: Captured image (image captured by camera)
PF1 Shooting surface PS1, PS2, PS31, PS32, PS33, PS Shooting object

Claims (14)

A camera that photographs a refrigerator having a storage compartment and an opening/closing body that opens and closes an opening of the storage compartment and is disposed on an upper surface of the refrigerator;
an object to be photographed, the object being disposed near an opening of the storage chamber or on the opening/closing body, and the photographing state of which is changed according to the opening/closing angle of the opening/closing body;
a state storage unit that stores a photographing state of the subject by the camera at a predetermined opening/closing angle of the opening/closing body;
a selection unit that selects a captured image in which the opening/closing body is in an open state from the images captured by the camera based on the captured state of the object by the camera stored in the state storage unit;
An imaging system comprising: the object is placed on the opening/closing body so that an imaging surface of the object is approximately perpendicular to an imaging direction of the camera when the opening/closing body is at the predetermined opening/closing angle;
The imaging system according to claim 1 . The subject is placed in a door pocket of the opening/closing body,
The imaging system according to claim 2 , further comprising a fixing member for fixing the object to be photographed to the door pocket. A hinge member is provided to rotatably support the opening/closing body,
The object to be photographed is configured as a hinge cover of the hinge member.
The imaging system according to claim 1 . The hinge cover has a notch formed therein,
The notch fits into the upper part of the frame of the drawer below the storage chamber.
The imaging system according to claim 4 . a photographing target image including identification information indicating at least one of the type of item stored in the storage chamber and the storage chamber in which the item is stored is formed on an imaging surface of the photographing target;
The state storage unit stores the image of the subject to be photographed,
The selection unit selects a captured image in which the opening/closing body is in an open state from the images captured by the camera based on the captured target image stored in the state storage unit.
The imaging system according to claim 1 . The image of the object to be photographed is
A rectangular outer frame image;
an identification image formed inside the outer frame image and corresponding to identification information indicating a type of item stored in the storage compartment of the refrigerator;
The imaging system of claim 6 . the identification image is composed of a first rectangular image in a first color and a second rectangular image in a second color different from the first color,
an arrangement pattern of the first rectangular image and the second rectangular image indicates the identification information;
The imaging system according to claim 7 . a size of the first rectangular image is the same as a size of the second rectangular image;
the first rectangular image and the second rectangular image are arranged along a longitudinal direction of the outer frame image;
The imaging system according to claim 8 . a size of the first rectangular image and a size of the second rectangular image are determined based on a resolution of the camera and a distance between the camera and the image of the subject at a predetermined opening/closing angle of the opening/closing body;
The imaging system according to claim 9 . The selection unit is capable of detecting the outer frame image included in the image captured by the camera, and when identification information indicated by an identification image of the image to be photographed included in the image captured by the camera matches identification information indicated by the identification image of the image to be photographed stored in the state storage unit, selects the image captured by the camera as the image to be photographed when the opening/closing body is in the open state.
The imaging system according to claim 7 . When it is determined that the camera has photographed the object to be photographed, the selection unit notifies a user that the camera has photographed the object to be photographed.
The imaging system according to claim 1 . A camera that photographs a refrigerator having a storage compartment and an opening/closing body that opens and closes an opening of the storage compartment and is disposed on an upper surface of the refrigerator;
a state storage unit that stores a photographing state of an object to be photographed by the camera at a predetermined opening/closing angle of the opening/closing body;
a selection unit that selects a captured image in which the opening/closing body is in an open state from the images captured by the camera based on the captured state of the object by the camera stored in the state storage unit;
Equipped with
The object to be photographed is placed near the opening of the storage chamber or on the opening/closing body, and the state of photography by the camera changes depending on the opening/closing angle of the opening/closing body. A camera that photographs a refrigerator having a storage compartment and an opening/closing body that opens and closes an opening of the storage compartment and is disposed on an upper surface of the refrigerator;
an object to be photographed, the object being disposed near an opening of the storage chamber or on the opening/closing body, and the photographing state of which is changed according to the opening/closing angle of the opening/closing body;
a state storage unit that stores a photographing state of the subject by the camera at a predetermined opening/closing angle of the opening/closing body;
a selection unit that selects a captured image in which the opening/closing body is in an open state from the images captured by the camera based on the captured state of the object by the camera stored in the state storage unit;
Equipped with a refrigerator.