JP2023127929A - Information processing system, and photographing device - Google Patents

Abstract

To provide an information processing system that can select a photographic image appropriately showing an interior condition of a storage cabinet photographed by a photographing device, from images taken by the photographing device, regardless of the kind of the storage cabinet, and the photographing device.SOLUTION: An information processing system comprises a photographing device, and a terminal device for communicating with the photographing device. The photographing device comprises: a camera for photographing a storage cabinet comprising a box body in which a storage chamber is formed, and an opening/closing body for opening/closing an opening of the storage chamber; a registration unit for registering a predetermined open state of the opening/closing body; and a selection unit for selecting a photographic image in which the opening/closing body is in the open state, from images taken by the camera, on the basis of the predetermined open state registered by the registration unit.SELECTED DRAWING: Figure 9

Description

The present disclosure relates to an information processing system and a photographing device.

Patent Document 1 discloses a technology in which a camera is provided on the top of a refrigerator, photographs are taken when an object passing through an opening of the refrigerator is detected, and food stored in the refrigerator is managed based on the photographed results.

JP2019-070476A

The present disclosure provides an information processing system and a photographing device that can select a photographed image that appropriately depicts the internal situation of a storage from images photographed by a camera, regardless of the model of the storage that the photographing device photographs.

An information processing system according to the present disclosure includes a photographing device and a terminal device that communicates with the photographing device, and the photographing device includes a box in which a storage chamber is formed, and an opening/closing body that opens and closes an opening of the storage chamber. a registration unit that registers a predetermined open state of the opening/closing body; and a registration unit that registers a predetermined open state of the opening/closing body; A selection unit that selects a captured image when the body is in an open state.

Further, the photographing device according to the present disclosure includes a camera that photographs a storage having a box in which a storage chamber is formed, an opening/closing body that opens and closes an opening of the storage chamber, and a predetermined open state of the opening/closing body. The device includes a registration unit, and a selection unit that selects a photographed image when the opening/closing body is in the open state from among images photographed by the camera, based on the predetermined open state registered by the registration unit.

The information processing system and the photographing device in the present disclosure select a captured image when the opening/closing body is in the open state based on the registered predetermined open state, so that the information processing system and the photographing device select a photographed image when the opening/closing body is in the open state. You can select the captured image. Therefore, it is possible to select a captured image that appropriately depicts the internal situation of the storage from the images captured by the camera, regardless of the model of the storage captured by the imaging device.

A diagram showing the configuration of an article management system in Embodiment 1 A perspective view of a refrigerator in Embodiment 1 A plan view of the photographing device installed in the refrigerator in Embodiment 1 seen from the right side. A perspective view of the regulating member in contact with the front edge of the box in Embodiment 1, seen from the front left. A plan view of the refrigerator and the photographing device in Embodiment 1 seen from the front. A plan view of the refrigerator and photographing device in Embodiment 1 seen from the right side. A plan view from above of the photographing device installed in the refrigerator in Embodiment 1. A plan view from above of the photographing device installed in the refrigerator in Embodiment 1. Block diagram showing the configuration of an imaging device in Embodiment 1 Block diagram showing the configuration of a terminal device and an article management server in Embodiment 1 Flowchart showing the operation of the information processing system in Embodiment 1 Flowchart showing the operation of the information processing system in Embodiment 1 A diagram showing an example of a first guide image in Embodiment 1 Flowchart showing the operation of the information processing system in Embodiment 1 A diagram showing an example of a second guide image in Embodiment 1 Flowchart showing the operation of the imaging device in Embodiment 1 Flowchart showing the operation of the article management system in Embodiment 1 Flowchart showing the operation of the article management system in Embodiment 1 A diagram showing the configuration of an article management system in Embodiment 2 Plan view of the refrigerator in Embodiment 2 seen from the right side Block diagram showing the configuration of a refrigerator in Embodiment 2 Flowchart showing the operation of the article management system in Embodiment 2

(Findings, etc. that formed the basis of this disclosure)
At the time the inventors came up with the present disclosure, there was a technology in which photography was performed when an object passing through an opening of a refrigerator was detected, and food stored in the refrigerator was managed based on the photography results. Ta. By the way, depending on the model of the storage, the size of the opening/closing body that opens and closes the opening of the storage room and the range in which the opening/closing body can be opened and closed may differ, so the criteria for selecting the photographed images are those that correspond to the model of the storage. It is requested that However, in the conventional technology, this request is not taken into consideration, and the inventors discovered the problem that captured images that do not properly depict the internal situation of the storage warehouse may be selected. This has led to the subject matter of the present disclosure.
Therefore, the present disclosure provides an information processing system and a photographing device that can select a photographed image that appropriately depicts the internal situation of a storage from images taken by a camera, regardless of the model of the storage that the photographing device photographs. do.

Hereinafter, embodiments will be described in detail with reference to the drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of well-known matters or redundant explanations of substantially the same configurations may be omitted.
The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter recited in the claims.

(Embodiment 1)
[1-1. composition]
[1-1-1. Configuration of goods management system]
FIG. 1 is a diagram showing the configuration of an article management system 1A in the first embodiment.
The article management system 1A is a system that manages articles stored in the refrigerator 2A. The article management system 1A of this embodiment manages images showing articles stored in the refrigerator 2A, as an example of article management.
The refrigerator 2A corresponds to a "storage" in the present disclosure.

In this embodiment, the articles include foods, ingredients, containers for storing foods, and the like. A predetermined marker may be attached to the article. An example of the marker is a code such as a QR code (registered trademark).

The article management system 1A includes a refrigerator 2A.
The refrigerator 2A includes a box body 20 with an open front. A refrigerator compartment 21, an ice-making compartment 22, a fresh-freezing compartment 23, a freezing compartment 24, and a vegetable compartment 25 are formed in the box body 20. These containment chambers are cooled with cold air. A chilled compartment 26 whose internal temperature is lower than that of the refrigerator compartment 21 is formed within the refrigerator compartment 21 . A drawer 26A capable of storing articles is installed in the chilled chamber 26. A rotary door 21A is installed in the front opening of the refrigerator compartment 21. The door 21A is a double door, and includes a rotating left door 21B and a rotating right door 21C. Each of the ice making compartment 22, the fresh freezing compartment 23, the freezing compartment 24, and the vegetable compartment 25 is provided with drawers 22A, 23A, 24A, and 25A that can accommodate articles. Each of the drawers 22A to 26A has an open top surface in the installed state.
The refrigerator compartment 21 corresponds to the "first storage compartment" of the present disclosure.

In the following description, when the refrigerator compartment 21, the ice-making compartment 22, the fresh-freezing compartment 23, the freezing compartment 24, and the vegetable compartment 25 are not distinguished, they will be referred to as "storage compartments" and will be given the numeral "27".
In the following description, if the ice making compartment 22, fresh freezing compartment 23, freezing compartment 24, vegetable compartment 25, and chilled compartment 26 are not distinguished, they will be referred to as "drawer compartments" and will be given the reference numeral "28". In this embodiment, the drawer chamber 28 corresponds to the "second storage chamber" of the present disclosure.
Further, in the following description, when the drawers 22A, 23A, 24A, 25A, and 26A are not distinguished from each other, they are referred to as "drawer 28A."
Moreover, in the following description, when the door 21A and the drawer 28A are not distinguished, they will be referred to as "opening/closing body" and will be given the reference numeral "29A".

The article management system 1A includes a photographing device 3. The photographing device 3 is provided on the upper surface 20A of the box 20 of the refrigerator 2A. The photographing device 3 photographs the refrigerator 2A.

The article management system 1A includes a terminal device 4. The terminal device 4 is a portable computer such as a smartphone. The terminal device 4 has installed therein an application program related to the management of articles stored in the refrigerator 2A, and communicates with the article management server 5 through the functions of this application program.
In the following description, this application program will be referred to as an "article management application" and will be given the reference numeral "411".

In FIG. 1, the user P who is at home is shown by a solid line, and the user P who is out of his home H is shown by a dotted line. When the terminal device 4 is used by a user P who is at home, the terminal device 4 communicates with the article management server 5 via the communication device 6 or not via the communication device 6 . Further, when the terminal device 4 is used by the user P who has gone out from home H and a communication connection cannot be established with the communication device 6, the terminal device 4 communicates with the article management server 5 without going through the communication device 6.

The communication device 6 is connected to a network NW composed of a public line network, a private line, and other communication circuits, and communicates with the article management server 5 via the network NW. The communication device 6 functions as an interface device for connecting the refrigerator 2A, the photographing device 3, and the terminal device 4 to the network NW. The communication device 6 constructs a local network at home H.

The article management system 1A includes an article management server 5. The article management server 5 is a device that manages articles stored in the refrigerator 2A. The article management server 5 stores images of articles for article management. The article management server 5 connects to the network NW. Note that although the article management server 5 is represented by one block in each figure, this does not necessarily mean that the article management server 5 is constituted by a single device.

The article management system 1A includes an information processing system 7A. The information processing system 7A includes a photographing device 3 and a terminal device 4, and the photographing device 3 and the terminal device 4 perform information processing.

[1-1-2. Refrigerator configuration]
FIG. 2 is a perspective view of the refrigerator 2A in an installed state.
In FIG. 2, an X-axis, a Y-axis, and a Z-axis are illustrated. The X-axis, Y-axis, and Z-axis are orthogonal to each other. The Z axis indicates the vertical direction and corresponds to the height direction of the refrigerator 2A. The X-axis and Y-axis are parallel to the horizontal direction. The X axis indicates the left and right direction. The left-right direction corresponds to the width direction of the refrigerator 2A. The Y-axis indicates the front-back direction. The positive direction of the X axis indicates the right direction. The positive direction of the Y axis indicates the forward direction. The positive direction of the Z axis indicates the upward direction.

The refrigerator 2A includes a box 20, a door 21A, and a drawer 28A.
The box body 20 is a box-shaped member that forms the main body of the refrigerator 2A. The box body 20 includes an outer box 201 that opens on the front, an inner box 202 that is housed inside the outer box 201 and opens on the front, and a heat insulating material filled in the space between the outer box 201 and the inner box 202. Equipped with. The outer box 201 is made of, for example, a metal plate such as iron or a hard resin such as ABS. Inner box 202 is made of, for example, hard resin such as ABS. For example, rigid urethane foam is used as the heat insulating material. Note that the heat insulating material is not limited to this, and any material can be used as long as it has heat insulating properties and can be filled into the space between the outer box 201 and the inner box 202.

The internal space of the inner box 202 is partitioned into a plurality of spaces by partition members. In this embodiment, the internal space of the inner box 202 is partitioned into five spaces, and each of the five spaces functions as a refrigerator compartment 21, an ice-making compartment 22, a fresh-freeze compartment 23, a freezer compartment 24, and a vegetable compartment 25. do.

The left door 21B is rotatably supported by a first hinge member 203A installed on the upper surface 20A of the box body 20. The right door 21C is rotatably supported by a second hinge member 203B installed on the upper surface 20A of the box body 20.

In a top view of the refrigerator 2A in the installed state, the left door 21B has a first notch 204A formed in the front surface of the end opposite to the end connected to the first hinge member 203A. The first notch 204A is formed to extend in the height direction of the refrigerator 2A. The first notch 204A functions as a handle for opening and closing the left door 21B.
In a top view of the refrigerator 2A in the installed state, the right door 21C has a second notch 204B formed in the front surface of the end opposite to the end connected to the second hinge member 203B. The second notch 204B is formed to extend in the height direction of the refrigerator 2A. The second notch 204B functions as a handle for opening and closing the right door 21C.
In a top view of the refrigerator 2A in the installed state, a recess 205 is formed in the front surface of the door 21A in the closed state by the first notch 204A and the second notch 204B. The recess 205 of this embodiment is formed approximately at the center of the refrigerator 2A in the width direction when viewed from above of the refrigerator 2A in the installed state.

[1-1-3. Configuration of imaging device]
The photographing device 3 will be explained with reference to FIGS. 3 to 8.
3-8 illustrate the same X-axis, Y-axis, and Z-axis as the X-axis, Y-axis, and Z-axis shown in FIG. 2. The X-axis direction corresponds to the width direction of the imaging device 3. The Y-axis direction corresponds to the front-rear direction of the photographing device 3.

FIG. 3 is a plan view of the photographing device 3 installed in the refrigerator 2A viewed from the right side. FIG. 4 is a perspective view of the regulating member 303 in contact with the front edge 20B of the box body 20, viewed from the front left.

As shown in FIGS. 3 and 4, the photographing device 3 includes a main body 301 installed on the upper surface 20A of the box 20, and a photographing member 302 extending forward from the main body 301. The photographing member 302 is provided above the front part of the main body 301.
The main body 301 includes a regulating member 303 that regulates the position of the photographing member 302 in the front-rear direction of the refrigerator 2A by coming into contact with the front edge 20B of the box body 20. The regulating member 303 is a piece that extends downward from the bottom surface 301A of the main body 301 in the installed state. The regulating member 303 extends in the width direction of the main body 301 (the width direction of the imaging device 3) on the front surface of the main body 301, and has a mark 304 at the center of the main body 301 in the width direction. The mark 304 indicates the center of the photographing device 3 in the width direction.

In the installed state, the regulating member 303 is formed to be shorter than the length from the upper surface 20A of the box body 20 to the door packing (hereinafter referred to as "upper door packing") provided approximately parallel to the upper surface 20A at the upper inner surface of the door 21A. has been done.

FIG. 5 is a plan view of the refrigerator 2A and the photographing device 3 installed in the refrigerator 2A, viewed from the front. In the refrigerator 2A shown in FIG. 5, the left door 21B and the right door 21C are in an open state. FIG. 6 is a plan view of the refrigerator 2A and the photographing device 3 installed in the refrigerator 2A, viewed from the right. In the refrigerator 2A shown in FIG. 6, the drawer 25A is in an open state, that is, in a pulled out state.

The photographing device 3 includes a wide-angle camera 305 and a narrow-angle camera 306. A wide-angle camera 305 and a narrow-angle camera 306 are installed at the front end of the bottom surface 302A of the photographing member 302.
In the following description, if the wide-angle camera 305 and the narrow-angle camera 306 are not distinguished, they will be referred to as "cameras" and will be given the reference numeral "307".
In the photographing device 3, a camera 307 is provided in front of the door 21A when the door 21A is closed.
The wide-angle camera 305 corresponds to the "first camera" of the present disclosure. The narrow-angle camera 306 corresponds to the "second camera" of the present disclosure.

The wide-angle camera 305 has a wider angle than the narrow-angle camera 306. The wide-angle camera 305 photographs the refrigerator compartment 21 from above and in front of the refrigerator 2A when the photographing device 3 is installed in the refrigerator 2A.
For example, the photographing range of the wide-angle camera 305 includes a range A1 shown in FIG. 5 when viewed from the front of the refrigerator 2A. The range A1 is a range that includes the door pockets 211 of the left door 21B and the right door 21C and the opening of the refrigerator compartment 21 when the left door 21B and the right door 21C are in the open state at the maximum opening degree. Further, the photographing range of the wide-angle camera 305 includes a range A3 shown in FIG. 6 when viewed from the right side of the refrigerator 2A. Range A3 is a range that includes from the front ends of the left door 21B and right door 21C in the open state to the front end of the shelf 212 in the front-rear direction, and includes the opening of the refrigerator compartment 21 in the vertical direction. be.

The narrow-angle camera 306 has a narrower angle than the wide-angle camera 305. The narrow-angle camera 306 photographs the drawer 28A from the upper front of the box body 20 when the photographing device 3 is installed in the refrigerator 2A.
For example, the photographing range of the narrow-angle camera 306 includes a range A2 shown in FIG. 5 when viewed from the front of the refrigerator 2A. The range A2 includes the drawer 28A when viewed from the front of the refrigerator 2A. Further, for example, the photographing range of the narrow-angle camera 306 includes a range A4 shown in FIG. 6 when viewed from the right side of the refrigerator 2A. The range A4 is a range including the drawer 28A in the most pulled out state in the front-rear direction.

The photographing device 3 includes a left door distance measurement sensor 308, a right door distance measurement sensor 309, a drawer distance measurement sensor 310, and a human sensor 311. A left door distance measurement sensor 308 and a right door distance measurement sensor 309 are installed in the main body 301. The draw-out distance sensor 310 and the human sensor 311 are installed at the front end of the bottom surface 302A of the photographing member 302. Although the left door distance measurement sensor 308, the right door distance measurement sensor 309, and the drawer distance measurement sensor 310 in this embodiment are optical sensors, the sensing method is not limited.
In the following description, if the left door distance measurement sensor 308 and the right door distance measurement sensor 309 are not distinguished, they will be referred to as "distance measurement sensors" and will be given the reference numeral "312".

The left door distance sensor 308 detects the distance between the left door 21B and itself, and outputs a detection value indicating the detected distance. The sensing target of the left door distance measurement sensor 308 is a portion of the left door 21B located above the upper door packing.
The right door distance sensor 309 detects the distance between the right door 21C and itself, and outputs a detection value indicating the detected distance. The sensing target of the right door distance measuring sensor 309 is a portion of the right door 21C located above the upper door packing.
The drawer distance sensor 310 detects the distance between the drawer 28A and itself, and outputs a detection value indicating the detected distance.
The human sensor 311 may be an infrared sensor or an ultrasonic sensor.

The arrangement of the camera 307 will be explained.
FIG. 7 is a top plan view of the photographing device 3 installed in the refrigerator 2A.

As shown in FIG. 7, the wide-angle camera 305, the narrow-angle camera 306, the draw-out range sensor 310, and the human sensor 311 are arranged in the width direction of the photographing device 3 at the front end of the bottom surface 302A of the photographing member 302. is set up. The wide-angle camera 305 and the narrow-angle camera 306 are installed closer to the center than the draw-out range sensor 310 and the human sensor 311 in the width direction of the photographing device 3 . The length of the imaging device 3 in the width direction in this embodiment is shorter than the length of the recess 205 in the left-right direction. Therefore, when the photographing device 3 is installed in the refrigerator 2A with the mark 304 located approximately at the center in the width direction of the refrigerator 2A, the wide-angle camera 305 and the narrow-angle camera 306 are positioned at positions overlapping with the recess 205 in the width direction of the refrigerator 2A. Located in Further, in this case, the drawer distance sensor 310 and the human sensor 311 are also located at a position overlapping the recess 205 in the width direction of the refrigerator 2A.

Note that the length of the photographing device 3 in the width direction may be equal to or longer than the length of the recess 205. In this configuration, the wide-angle camera 305 and the narrow-angle camera 306 are installed at an interval shorter than the length of the recess 205 in the left-right direction at the front end of the bottom surface 302A.

The arrangement of the distance measuring sensor 312 will be explained.
FIG. 8 is a plan view of the photographing device 3 installed in the refrigerator 2A viewed from above.

The distance measurement sensor 312 is installed at the front lower part of the main body 301.
The left door distance measurement sensor 308 is installed with an irradiation surface that irradiates light and a light reception surface that receives reflected light facing toward the left front of the photographing device 3. That is, when the left door distance measuring sensor 308 is installed in the refrigerator 2A, the sensing direction is on the first hinge member 203A side in the width direction of the photographing device 3, in other words, the rotation axis of the left door 21B. It is installed on the main body 301 so as to face to the side.
Specifically, the left door ranging sensor 308 connects the optical axis AX1 indicating the sensing direction and the left door opened at the maximum opening degree when viewed from above of the photographing device 3 when the photographing device 3 is installed in the refrigerator 2A. It is installed in the main body 301 so that the inner surface 21B1 of 21B intersects with the inner surface 21B1. Note that the installation angle is appropriately determined through prior tests and simulations.
Further, in the left door distance measurement sensor 308, the irradiation surface that irradiates light is installed closer to the left door 21B than the light reception surface that receives reflected light. That is, the left door distance measuring sensor 308 is installed so that the irradiation surface is in the positive direction of the Y axis (the near side when the refrigerator is installed) rather than the light receiving surface. This makes it easier for light to hit the surface of the left door 21B, making it possible to widen the detection range of the left door distance measurement sensor 308.

The right door ranging sensor 309 is installed with its irradiation surface and light receiving surface facing toward the right front of the photographing device 3 . That is, when the photographing device 3 is installed in the refrigerator 2A, the right door distance measuring sensor 309 has a sensing direction on the second hinge member 203B side in the width direction of the photographing device 3, in other words, the rotation axis of the right door 21C. It is installed on the main body 301 so as to face to the side.
Specifically, the right door ranging sensor 309 connects the optical axis AX2 indicating the sensing direction and the right door opened at the maximum opening degree when viewed from above of the photographing device 3 when the photographing device 3 is installed in the refrigerator 2A. It is installed in the main body 301 so that the inner surface 21C1 of 21C intersects with the inner surface 21C1. Note that the installation angle is appropriately determined through prior tests and simulations.
Further, in the right door distance measurement sensor 309, the irradiation surface that irradiates light is installed closer to the right door 21C than the light reception surface that receives reflected light. That is, the right door distance measurement sensor 309 is installed so that the irradiation surface is in the positive direction of the Y axis (the front side when the refrigerator is installed) rather than the light receiving surface. This makes it easier for light to hit the surface of the right door 21C, making it possible to widen the detection range of the right door distance measurement sensor 309.

FIG. 9 is a block diagram showing the configuration of the photographing device 3. As shown in FIG.
As shown in FIG. 9, the photographing device 3 includes a sensor control board 313, a left door distance sensor 308, a right door distance sensor 309, a drawer distance sensor 310, and a human sensor 311.

The sensor control board 313 includes a first processor 320 such as a CPU (Central Processing Unit), a first memory 330, a first communication module 340, and an interface circuit to which other devices and sensors are connected. The sensor control board 313 is provided on the bottom surface 301A of the main body 301.

The first memory 330 is a storage device that non-volatilely stores programs and data executed by the first processor 320. The first memory 330 is composed of a nonvolatile storage device. Further, the first memory 330 may include a RAM (Random Access Memory) that constitutes a work area of the first processor 320. The first memory 330 stores data processed by the first processor 320 and a control program 331 executed by the first processor 320.

The first communication module 340 includes communication hardware such as a communication circuit and a connector according to a predetermined wired communication standard, and communicates with the camera control board 314 via a cable.

A left door distance measurement sensor 308 , a right door distance measurement sensor 309 , a drawer distance measurement sensor 310 , and a human sensor 311 are connected to the sensor control board 313 . Each of these sensors outputs a detected value to the sensor control board 313.

The first processor 320 functions as the first communication control unit 321 by reading and executing the control program 331 stored in the first memory 330.
The first communication control unit 321 communicates with the camera control board 314 via the first communication module 340. The first communication control unit 321 transmits detection values output by each sensor connected to the sensor control board 313 to the camera control board 314.

The photographing device 3 includes a camera control board 314, a third communication module 315, a fourth communication module 316, a wide-angle camera 305, a narrow-angle camera 306, a first LED (Light Emitting Diode) 317, and a second LED 318.

The camera control board 314 includes a second processor 350 such as a CPU, a second memory 370, a second communication module 380, and an interface circuit to which other devices and sensors are connected. The camera control board 314 is provided on the bottom surface 301A of the main body 301.

The second memory 370 is a storage device that nonvolatilely stores programs and data executed by the second processor 350. The second memory 370 is configured by a nonvolatile storage device. Further, the second memory 370 may include a RAM that constitutes a work area of the second processor 350. The second memory 370 stores data processed by the second processor 350, a control program 371 executed by the second processor 350, an imaging device ID (Identification) 372 that uniquely identifies the imaging device 3, and setting data 373. do.

The setting data 373 is data related to the settings of the photographing device 3, and includes first registration data 373A, second registration data 373B, and third registration data 373C.
The first registration data 373A is data in which a predetermined open state of the door 21A is registered. The predetermined open state is an open state that serves as a reference in selection by the selection unit 358, which will be described later. In this embodiment, the predetermined open state of the door 21A is a state in which the left door 21B is opened at 90 degrees, and a state in which the right door 21C is opened at 90 degrees. The first registration data 373A records the detection value of the distance measurement sensor 312 when the door 21A is in a predetermined open state. Further, the first registered data 373A records the detection value of the distance measurement sensor 312 when the door 21A is in the closed state. Note that the angle regarding the left door 21B is 0° when the left door 21B is in the closed state, and is positive when the rotation direction in which the left door 21B is in the open state. Further, the angle regarding the right door 21C is 0° when the right door 21C is in the closed state, and is positive when the rotation direction in which the right door 21C is in the open state.
The second registration data 373B is data in which a predetermined open state of the drawer 28A is registered. In this embodiment, the predetermined open state of the drawer 28A is the state when the drawer 28A is pulled out the most. The second registration data 373B records the position of the front edge of the drawer 28A when the drawer 28A is in a predetermined open state.
The third registration data 373C is data in which the SSID and password of the local network constructed by the communication device 6 are recorded.

The second communication module 380 includes communication hardware such as a communication circuit and a connector according to a predetermined communication standard, and communicates with the sensor control board 313 under the control of the second processor 350.

A third communication module 315, a fourth communication module 316, a wide-angle camera 305, a narrow-angle camera 306, a first LED 317, and a second LED 318 are connected to the camera control board 314.
The third communication module 315 includes a communication circuit, a connector, etc. in accordance with a predetermined communication standard, and communicates with the article management server 5 via the communication device 6 and the network NW. The third communication module 315 is provided on the bottom surface 302A of the photographing member 302.
The fourth communication module 316 includes a communication circuit, a connector, and the like according to a predetermined short-range wireless communication standard, and communicates with the terminal device 4 . Note that the fourth communication module 316 may be equipped with a communication circuit, a connector, etc. in accordance with a predetermined communication standard, and may communicate with the terminal device 4 via the communication device 6 and the network NW. The fourth communication module 316 is provided on the bottom surface 302A of the photographing member 302.
Note that the third communication module 315 and the fourth communication module 316 are provided at positions separated from each other by a predetermined distance or more. Thereby, interference of radio waves between communication modules can be suppressed. The predetermined distance is, for example, 100 mm.
Further, the third communication module 315 and the fourth communication module 316 may be configured to communicate with the terminal device 4 or the article management server 5 as one communication module.
Each of the wide-angle camera 305 and the narrow-angle camera 306 outputs the shooting results to the camera control board 314. The photographed result is a moving image. Note that the photographing result is not limited to a moving image, and may be an image. Alternatively, the image may be a plurality of images shot continuously. Furthermore, a moving image may include the concept of a plurality of still images.

The second processor 350 reads and executes the control program 371 stored in the second memory 370, thereby controlling the second communication control section 351, the third communication control section 352, the fourth communication control section 353, and the person presence determination section. 354, an opening/closing determination section 355, a photographing control section 356, a recording control section 357, a selection section 358, a generation section 359, and a registration section 360.

The second communication control unit 351 communicates with the sensor control board 313 via the second communication module 380. The second communication control unit 351 receives detection values of various sensors connected to the sensor control board 313 from the sensor control board 313. The second communication control unit 351 outputs the detection value of the human sensor 311 received from the sensor control board 313 to the person presence determination unit 354. The second communication control unit 351 outputs the detection value of each ranging sensor received from the sensor control board 313 to the open/close determination unit 355, the recording control unit 357, and the registration unit 360.

The third communication control unit 352 communicates with the article management server 5 via the third communication module 315.
The fourth communication control unit 353 communicates with the terminal device 4 via the fourth communication module 316.

The person presence determination unit 354 determines whether or not there is a person around the refrigerator 2A based on the detection value of the human sensor 311. For example, if the detection value of the human sensor 311 is equal to or higher than a predetermined value, the person presence determination unit 354 determines that there is a person around the refrigerator 2A, and if it is less than the predetermined value, the person presence determination unit 354 determines that there is a person around the refrigerator 2A. It is determined that it does not.

The opening/closing determination unit 355 determines whether the left door 21B is in an open state or a closed state based on the first registration data 373A and the detected value of the left door ranging sensor 308. The open/close determining unit 355 determines that the left door 21B is in the open state when the left door ranging sensor 308 outputs a detected value different from the detected value corresponding to the closed state. The open/close determining unit 355 determines that the left door 21B is in the closed state when the left door ranging sensor 308 outputs a detection value corresponding to the closed state.
The opening/closing determination unit 355 determines whether the right door 21C is in an open state or a closed state based on the first registration data 373A and the detected value of the right door ranging sensor 309. The open/close determining unit 355 determines that the right door is in the open state when the right door ranging sensor 309 outputs a detection value that is different from the detection value corresponding to the closed state. The open/close determining unit 355 determines that the right door 21C is in the closed state when the right door ranging sensor 309 outputs a detection value corresponding to the closed state.
The open/close determining unit 355 determines whether the drawer 28A is in an open state or a closed state, depending on the distance indicated by the detection value of the drawer distance measurement sensor 310. Note that the open/close determining unit 355 may determine whether the drawer 28A is in an open state or a closed state by a method such as image analysis of a moving image captured by the narrow-angle camera 306.

When the person presence determination section 354 determines that there is a person around the refrigerator 2A, the photographing control section 356 causes the wide-angle camera 305 and the narrow-angle camera 306 to start photographing. When the shooting control unit 356 causes the wide-angle camera 305 and the narrow-angle camera 306 to start shooting, it lights up the first LED 317 and notifies the user P that shooting has started. After the wide-angle camera 305 and the narrow-angle camera 306 start shooting, if the person presence determining unit 354 determines that there is no person around the refrigerator 2A, the shooting control unit 356 causes the wide-angle camera 305 and the narrow-angle camera 306 to take a photo. Terminate it. The photographing control unit 356 turns off the first LED 317 when the wide-angle camera 305 and the narrow-angle camera 306 finish photographing.

When the opening/closing determination unit 355 determines that the opening/closing body 29A is in the open state, the recording control unit 357 starts recording the photographic results of the wide-angle camera 305 and the narrow-angle camera 306. Specifically, the recording control unit 357 starts recording the first recording data 374 indicating the shooting result of the wide-angle camera 305 into the second memory 370 . Furthermore, the recording control unit 357 starts recording second recording data 375 indicating the photographing result of the narrow-angle camera 306 into the second memory 370 . Further, when the opening/closing determination unit 355 determines that the opening/closing body 29A is in the open state, the recording control unit 357 lights up the second LED 318 to illuminate the inside of the storage chamber 27. When the opening/closing determining unit 355 determines that both opening/closing bodies 29A are in the closed state, the recording control unit 357 ends recording of the photographic results of the wide-angle camera 305 and the narrow-angle camera 306. Furthermore, when the opening/closing determination unit 355 determines that all of the opening/closing bodies 29A are in the closed state, the recording control unit 357 turns off the second LED 318.

Note that the recording control unit 357 may start recording the photographed result of the wide-angle camera 305 when the opening/closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is in an open state. Specifically, the recording control unit 357 starts recording the first recording data 374 indicating the shooting result of the wide-angle camera 305 into the second memory 370 . In this case, recording of the photographed results of the narrow-angle camera 306 may be started at the same time. Further, the recording control unit 357 may turn on the second LED 318 when the opening/closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is in an open state. The recording control unit 357 may end the recording of the photographed result of the wide-angle camera 305 when the open/close determination unit 355 determines that both the left door 21B and the right door 21C are in the closed state. In this case, recording of the photographic results of the narrow-angle camera 306 may be terminated at the same time. Further, the recording control unit 357 may turn off the second LED 318 when the opening/closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is in a closed state.
Further, the recording control unit 357 may start recording the photographed result of the narrow-angle camera 306 when the open/close determination unit 355 determines that the drawer 28A is in the open state. Specifically, the recording control unit 357 starts recording second recording data 375 indicating the photographing result of the narrow-angle camera 306 into the second memory 370 . In this case, recording of the shooting results of the wide-angle camera 305 may be started at the same time. Further, the recording control unit 357 may turn on the second LED 318 when the open/close determination unit 355 determines that the drawer 28A is in the open state. The recording control unit 357 may end the recording of the photographed results of the narrow-angle camera 306 when the open/close determination unit 355 determines that all of the drawers 28A are in the closed state. In this case, recording of the shooting results of the wide-angle camera 305 may be finished at the same time. Further, the recording control unit 357 may turn off the second LED 318 when the open/close determination unit 355 determines that the drawer 28A is in the closed state.

During recording, the detection value of the distance measurement sensor 312 is input to the recording control unit 357. The recording control unit 357 adds the detection value of the left door distance sensor 308 to the shooting results of the wide-angle camera 305 during recording and the shooting results of the narrow-angle camera 306 during recording, frame by frame or at intervals of a predetermined number of frames. and the detection value of the right door distance measurement sensor 309. Note that during recording, the recording control unit may associate the withdrawal degree with the photographic results of the wide-angle camera 305 and the narrow-angle camera 306 that are recording.

The selection unit 358 selects a captured image from the moving images captured by the camera 307. The selection section 358 includes a first selection section 358A and a second selection section 358B. The first selection unit 358A selects a first photographed image showing the refrigerator compartment 21 when the door 21A is open from the moving image indicated by the first recorded data 374. The second selection unit 358B selects a second photographed image from the moving images indicated by the second recorded data 375. The second photographed image of this embodiment is a photographed image in which the drawer 28A is shown in an open state. Details of the first captured image and the second captured image will be described later.

The generation unit 359 generates upload data. The upload data is data to be uploaded to the article management server 5. The generation unit 359 outputs the generated upload data to the third communication control unit 352.

The registration unit 360 registers a predetermined open state of the door 21A by generating first registration data 373A and storing it in the second memory 370. Further, the registration unit 360 registers the predetermined open state of the drawer 28A by generating second registration data 373B and storing it in the second memory 370. Furthermore, the registration unit 360 registers information for connecting to the local network by generating third registration data 373C and storing it in the second memory 370.

[1-1-4. Article management server configuration]
FIG. 10 is a block diagram showing the configuration of the terminal device 4 and the article management server 5.
The article management server 5 includes a server processor 50 that is a processor such as a CPU, a server memory 51, and a server communication module 52. A server memory 51 and a server communication module 52 are connected to the server processor 50 .

The server memory 51 is a storage device that stores programs and data executed by the server processor 50 in a non-volatile manner. Server memory 51 is composed of a nonvolatile storage device. Further, the server memory 51 may include a RAM that constitutes a work area of the server processor 50. The server memory 51 stores data processed by the server processor 50, a control program 511 executed by the server processor 50, and an article management DB (database) 512.
The article management DB 512 has one record for each photographing device 3. This one record includes the photographing device ID 372, image data of the first photographed image, and image data of the second photographed image.

The server communication module 52 includes a communication circuit, a connector, and the like according to a predetermined communication standard, and communicates with the refrigerator 2A, the photographing device 3, and the terminal device 4 via the network NW.

The server processor 50 functions as a server communication control unit 501 and a server processing unit 502 by reading and executing a control program 511 stored in the server memory 51.
The server communication control unit 501 communicates with the refrigerator 2A, the photographing device 3, and the terminal device 4 via the server communication module 52.
The server processing unit 502 performs various information processing using the refrigerator 2A, the photographing device 3, and the terminal device 4 as clients.

[1-1-5. Terminal device configuration]
Next, the configuration of the terminal device 4 will be explained.
The terminal device 4 includes a terminal processor 40 that is a processor such as a CPU, a terminal memory 41, a terminal communication module 42, and a touch panel 43.

The terminal memory 41 is a storage device that stores programs and data executed by the terminal processor 40 in a non-volatile manner. Terminal memory 41 is composed of a nonvolatile storage device. Further, the terminal memory 41 may include a RAM that constitutes a work area of the terminal processor 40. The terminal memory 41 stores data processed by the terminal processor 40, a control program executed by the terminal processor 40, and an article management application 411.

The terminal communication module 42 includes a communication circuit, a connector, etc. in accordance with a predetermined communication standard, and communicates with the article management server 5 under the control of the terminal processor 40.
The touch panel 43 includes a display panel and a touch sensor provided overlappingly or integrally with the display panel.

As described above, the terminal processor 40 functions as the application execution unit 400. The application execution unit 400 functions as a terminal communication control unit 401, a display control unit 402, and a reception unit 403.
The terminal communication control unit 401 communicates with the article management server 5 via the terminal communication module 42.
Display control unit 402 causes touch panel 43 to display a user interface related to management of articles stored in refrigerator 2A. In the following description, this user interface will be referred to as an "application UI." The display control unit 402 of the present embodiment displays a guide image that guides the opening/closing body 29A to a predetermined open state by displaying the application UI.
The reception unit 403 receives various inputs from the user P via the application UI.
The terminal communication control unit 401 corresponds to the “communication control unit” of the present disclosure.

[1-2. motion]
Next, the operation of each part of the article management system 1A will be explained.

For example, after purchasing the photographing device 3, the user P installs the photographing device 3 in the refrigerator 2A and performs various settings for the photographing device 3.

[1-2-1. Adjustment of photographic equipment]
In installing the photographing device 3, the user P adjusts the position of the photographing device 3 in the width direction of the refrigerator 2A. The user P can adjust the position of the photographing device 3 using at least one of the first adjustment method, the second adjustment method, the third adjustment method, the fourth adjustment method, and the fifth adjustment method.

The first adjustment method is an adjustment method using an application UI. In the first adjustment method, the application UI displays the shooting results of either the wide-angle camera 305 or the narrow-angle camera 306, and the user P makes adjustments based on the displayed shooting results.
The second adjustment method is an adjustment method using recording paper. On this recording paper, a mark indicating the left end of the refrigerator 2A, a mark indicating the right end of the refrigerator 2A, and a mark indicating the center of the refrigerator 2A in the left-right direction are recorded. In the second adjustment method, the user P uses recording paper to determine where the center of the refrigerator 2A in the width direction is. In the second adjustment method, the user P adjusts the position of the photographing device 3 so that the mark 304 of the regulating member 303 is located at the determined center.
A third adjustment method is a method using marks attached to the refrigerator 2A. The mark is a mark indicating the center in the width direction of the refrigerator 2A. In the third adjustment method, the user P adjusts the position of the photographing device 3 so that the position of the mark on the refrigerator 2A matches the position of the mark 304 on the regulating member 303.
A fourth adjustment method is a method using a protrusion provided on the upper surface 20A of the refrigerator 2A. The protrusion is provided in accordance with the shape of the photographing device 3. For example, at least a protrusion parallel to the X-axis and a protrusion parallel to the Y-axis are provided on the upper surface 20A of the refrigerator 2A. In the fourth adjustment method, the user P adjusts the position of the photographing device 3 by placing the photographing device 3 in line with a protrusion provided on the upper surface 20A of the refrigerator 2A. Note that the position of the photographing device 3 may be adjusted by placing the photographing device 3 in a recess provided in the upper surface 20A of the refrigerator 2A to match the shape of the photographing device 3.
A fifth adjustment method is a method using a claw provided on the photographing device 3 and a hole provided on the upper surface 20A of the refrigerator 2A. The position of the photographing device 3 is adjusted by placing the photographing device 3 so that the nail fits into the hole. Note that a hole may be provided in the photographing device 3, and a claw may be provided in the upper surface 20A of the refrigerator 2A.

[1-2-2. Shooting device settings]
In the settings of the photographing device 3, a predetermined open state of the door 21A is registered. Further, in the settings of the photographing device 3, a predetermined open state of the drawer 28A is registered. Furthermore, in the settings of the photographing device 3, information for connecting to the local network is registered.

[1-2-2-1. [Registering information for connecting to local network]
First, registration of information for connecting to a local network will be explained.
FIG. 11 is a flowchart showing the operation of the information processing system 7A. Flowchart FA is a flowchart showing the operation of the terminal device 4, and flowchart FB is a flowchart showing the operation of the photographing device 3.
At the start of the flowchart shown in FIG. 11, the terminal device 4 and the photographing device 3 are connected to each other so as to be capable of short-range wireless communication.

As shown in the flowchart FA, the reception unit 403 receives input of the SSID and password of the local network constructed by the communication device 6 via the application UI (step SA1). Next, the terminal communication control unit 401 transmits the SSID and password received in step SA1 to the photographing device 3 (step SA2).

As shown in the flowchart FB, the fourth communication control unit 353 receives the SSID and password from the terminal device 4 (step SB1). Next, the registration unit 360 registers the SSID and password received in step SB1 (step SB2). That is, in step SB2, the registration unit 360 generates third registration data 373C in which the SSID and password received in step SB1 are recorded, and stores the generated third registration data 373C in the second memory 370.

When the registration unit 360 stores the third registration data 373C in the second memory 370, the photographing device 3 connects to the local network using the SSID and password recorded in the third registration data 373C.

[1-2-2-2. Registration of predetermined open state of door]
Next, registration of a predetermined open state of the door 21A will be explained.
FIG. 12 is a flowchart showing the operation of the information processing system 7A. Flowchart FC shows the operation of the terminal device 4, and flowchart FD shows the operation of the photographing device 3.
In the flowchart shown in FIG. 12, the terminal device 4 and the photographing device 3 communicate with each other, and the photographing device 3 transmits the photographing result of the wide-angle camera 305 to the terminal device 4.

As shown in the flowchart FC, the display control unit 402 causes the touch panel 43 to display information prompting the user to close the door 21A (step SC1).

Next, the reception unit 403 determines whether an input indicating that the door 21A has been closed has been received from the user P (step SC2). If the reception unit 403 determines that the input indicating that the door 21A has been closed has not been received (step SC2: NO), the reception unit 403 performs the determination in step SC2 again.

On the other hand, if the receiving unit 403 determines that the input indicating that the camera is in the closed state has been received (step SC2: YES), the terminal communication control unit 401 transmits instruction information to the photographing device 3 (step SC3). The instruction information is information that instructs recording of the detected value in the second memory 370.

As shown in the flowchart FD, the fourth communication control unit 353 receives instruction information from the terminal device 4 (step SD1).

Next, the registration unit 360 records the detection value currently output by the distance measurement sensor 312 in the second memory 370 (step SD2).

As shown in the flowchart FD, the fourth communication control unit 353 transmits completion information indicating that recording of the detected values has been completed to the terminal device 4 (step SD3).

As shown in the flowchart FC, the terminal communication control unit 401 receives completion information from the photographing device 3 (step SC4). Next, the display control unit 402 displays the first guide image AG1 superimposed on the photographed result of the wide-angle camera 305 (step SC5).

FIG. 13 is a diagram showing an example of the first guide image AG1 superimposed on the photographed result of the wide-angle camera 305.
The first guide image AG1 is a guide image that guides the door 21A to a predetermined open state, and has a plurality of lines L. The first guide image AG1 shown in FIG. 13 has 12 lines L, lines L1 to L12. Lines L1, L2, and L3 are lines L corresponding to the edges of the shelves 212 in the refrigerator compartment 21. Lines L4, L5, and L6 are lines L corresponding to the front edge of the box body 20. Line L4 is line L corresponding to the left edge. Line L5 is line L corresponding to the right edge. Line L6 is line L corresponding to the lower edge. Lines L7 to L9 are lines L corresponding to the edges of the door pocket 211 installed in the left door 21B. Lines L10 to L12 are lines L corresponding to the edges of the door pocket 211 provided in the right door 21C.

Each of the lines L included in the first guide image AG1 is displayed so as to correspond to each edge when the door 21A is in a predetermined open state. Note that the display position of the line L included in the first guide image AG1 differs depending on the model of the refrigerator 2A in which the photographing device 3 is installed. The display control unit 402 receives image data of a first guide image AG1 corresponding to the model of the refrigerator 2A from the article management server 5 in advance, and displays the first guide image AG1 indicated by the received image data.

The user P changes the rotation angle of the door 21A so that the lines L7 to L12 of the first guide image AG1 match the edge of the door pocket 211 shown in the photographic result displayed by the terminal device 4.

Returning to the explanation of the flowchart shown in FIG. 12, the receiving unit 403 determines whether or not it has received an input from the user P indicating that the opening state has been set to a predetermined open state (step SC6). If the reception unit 403 determines that the input indicating that the predetermined open state has been set has not been received (step SC6: NO), the reception unit 403 performs the determination in step SC6 again.

When the reception unit 403 determines that an input indicating that the camera is in the predetermined open state has been received (step SC6: YES), the terminal communication control unit 401 transmits instruction information to the photographing device 3 (step SC7).

As shown in the flowchart FD, the fourth communication control unit 353 receives instruction information from the terminal device 4 (step SD4).

Next, the registration unit 360 records the detection value currently output by the distance measurement sensor 312 in the second memory 370 (step SD5).

Next, the registration unit 360 generates first registration data 373A (step SD6).
Step SD6 will be explained in detail.
The registration unit 360 determines the detection value of the distance measurement sensor 312 recorded in step SD2 as the detection value of the distance measurement sensor 312 when the rotation angle of the door 21A is 0°, in other words, when the door 21A is in the closed state. do. Next, the registration unit 360 determines the detection value of the distance measurement sensor 312 recorded in step SD5 as the detection value of the distance measurement sensor 312 when the door 21A is in the predetermined open state. Next, the registration unit 360 associates the angle information indicating 0° with the detection value of the ranging sensor 312 determined as 0°, and also associates the angle information indicating 90° with the angle information indicating 90° when the door 21A is in a predetermined open state. First registration data 373A is generated in which the detection value of the distance measurement sensor 312 in a certain case is associated with the detected value.

The registration unit 360 stores the first registration data 373A generated in step SD6 in the second memory 370 (step SD7).

[1-2-2-3. Registration of specified open state of drawer]
Next, registration of a predetermined open state of the drawer 28A will be explained. In this description, the drawer 25A is illustrated as the drawer 28A.
FIG. 14 is a flowchart showing the operation of the information processing system 7A. In FIG. 14, a flowchart FE shows the operation of the terminal device 4, and a flowchart FF shows the operation of the photographing device 3.
In the flowchart shown in FIG. 14, the terminal device 4 and the photographing device 3 communicate with each other, and the photographing device 3 transmits the photographing result of the narrow-angle camera 306 to the terminal device 4.

As shown in the flowchart FE, the display control unit 402 displays the second guide image AG2 superimposed on the photographed result of the narrow-angle camera 306 (step SE1).

FIG. 15 is a diagram showing an example of the second guide image AG2 superimposed on the photographed result of the narrow-angle camera 306.
The second guide image AG2 is a guide image that guides the drawer 25A to a predetermined open state. Further, the second guide image AG2 is also an image that guides the storage case included in the drawer 25A to the most drawn out state. The second guide image AG2 shown in FIG. 15 has five lines L, lines L13 to L17. Lines L13 to L16 are lines L corresponding to the outer edge of the drawer 25A. Line L17 is a line L corresponding to the front edge of the storage case.

Each of the lines L13 to L16 included in the second guide image AG2 is displayed to correspond to each of the edges when the drawer 25A is in a predetermined open state. Moreover, the line L17 included in the second guide image AG2 is displayed so as to correspond to the front edge of the storage case of the drawer 25A when the storage case is in the most pulled out state. Note that the display position of the line L included in the second guide image AG2 differs depending on the model of the refrigerator 2A in which the photographing device 3 is installed. The display control unit 402 receives the image data of the second guide image AG2 according to the model of the refrigerator 2A from the article management server 5 in advance, and displays the second guide image AG2 indicated by the received image data.

The user P pulls out the drawer 28A so that the line L15 of the second guide image AG2 matches the front edge of the drawer 25A included in the photographic results displayed by the terminal device 4.

Returning to the explanation of the flowchart shown in FIG. 14, the receiving unit 403 determines whether or not it has received an input from the user P indicating that the opening state has been set to a predetermined open state (step SE2). If the reception unit 403 determines that the input indicating that the predetermined open state has been set has not been received (step SE2: NO), the reception unit 403 performs the determination in step SE2 again.

If the reception unit 403 determines that an input indicating that the predetermined open state has been entered has been received (step SE2: YES), the terminal communication control unit 401 transmits registration request information to the photographing device 3 (step SE3). The registration request information is information requesting registration of a predetermined open state, and records the type of the drawer 28A to be registered. In this explanation, the drawer 25A is illustrated, so the registration request information transmitted in step SE3 records the drawer 25A as the type of the drawer 28A to be registered.

As shown in the flowchart FF, the fourth communication control unit 353 receives registration request information from the terminal device 4 (step SF1).

Next, the registration unit 360 generates second registration data 373B (step SF2).
In step SF2, the registration unit 360 recognizes the position of the front edge of the drawer 25A in the most recent photographed image taken by the narrow-angle camera 306. The registration unit 360 performs image analysis such as shape recognition and color difference recognition on the photographed image, and recognizes the position of the front edge of the drawer 25A in the photographed image. Next, the registration unit 360 generates second registration data 373B in which the recognized position is recorded.
Note that the second registration data 373B records that the drawer 28A to be registered is the drawer 25A. The registration unit 360 records the type of registration target indicated by the registration request information in the second registration data 373B. In this explanation, the drawer 25A is illustrated, so the second registration data 373B generated in step SF2 records the drawer 25A as the type of the drawer 28A to be registered.

The registration unit 360 stores the second registration data 373B generated in step SF2 in the second memory 370 (step SF3).

In addition, in the description of FIG. 14, the case where the drawer 25A is registered as the drawer 28A is illustrated, but the drawers 28A other than the drawer 25A are also registered in the same manner. In this case, the second registration data 373B of each drawer 28A will be stored in the second memory 360.

Note that the various registrations described above can be performed even after the photographing device 3 starts to be used. In this case, the registration unit 360 updates the corresponding registration data to the latest data every time registration is performed.

Next, the operation of each part of the article management system 1A after the installation of the photographing device 3 and the settings of the photographing device 3 are performed will be described.

[1-2-3. Operations related to recording]
First, the operation of the photographing device 3 related to recording will be explained.
FIG. 16 is a flowchart FG showing the operation of the photographing device 3.
In the operation shown in FIG. 16, each functional unit of the second processor 350 performs processing using the detected value of each sensor to which the sensor control board 313 is connected.

The person presence determination unit 354 determines whether or not there is a person around the refrigerator 2A (step SG1). When the person presence determining unit 354 determines that there is no person around the refrigerator 2A (step SG1: NO), it performs the determination in step SG1 again.

On the other hand, when the person presence determining unit 354 determines that there is a person around the refrigerator 2A (step SG1: YES), the photographing control unit 356 causes the wide-angle camera 305 and the narrow-angle camera 306 to start photographing (step SG2). .

The opening/closing determination unit 355 determines whether the opening/closing body 29A is in the open state (step SG3). When the opening/closing determination unit 355 determines that the opening/closing body 29A is not in the open state (step SG3: NO), the second processor 350 performs the process of step SG7.

On the other hand, when the opening/closing determination unit 355 determines that the opening/closing body 29A is in the open state (step SG3: YES), the recording control unit 357 starts recording the shooting results of the wide-angle camera 305 and the narrow-angle camera 306 ( Step SG4).

Next, the opening/closing determination unit 355 determines whether the opening/closing body 29A is in the closed state (step SG5). When the opening/closing determination unit 355 determines that the opening/closing body 29A is not in the closed state (step SG5: NO), it performs the determination in step SG5 again.

On the other hand, if it is determined that the opening/closing body 29A is in the closed state (step SG5: YES), the recording control unit 357 ends recording of the photographing result (step SG6). As a result, first recorded data 374 and second recorded data 375 are generated in the second memory 370. Note that the second recorded data 375 records the type of drawer 28A that appears in the moving image shown by the second recorded data 275. The type is specified by the opening/closing determination unit 355 using a method such as a detection value of the drawer ranging sensor 310 or image analysis of a moving image taken by the narrow-angle camera 306. The recording control unit 357 records the type specified by the opening/closing determination unit 355 in the header of the second recording data 275.

The person presence determination unit 354 determines whether or not there is a person around the refrigerator 2A (step SG7). When the person presence determination unit 354 determines that a person exists around the refrigerator 2A (step SG7: YES), the second processor 350 again executes the processes from step SG3 onwards.

On the other hand, when the person presence determination section 354 determines that there is no person around the refrigerator 2A (step SG7: NO), the photographing control section 356 causes the wide-angle camera 305 and the narrow-angle camera 306 to finish photographing (step SG8). ).

[1-2-4. Operations related to uploading to the article management server]
Next, operations related to uploading to the article management server 5 will be explained.
FIG. 17 is a flowchart showing the operation of the article management system 1A.
In FIG. 17, a flowchart FH shows the operation of the photographing device 3, and a flowchart FI shows the operation of the article management server 5.

The generation unit 359 determines whether any one of the first recorded data 374 and the second recorded data 375 is stored in the second memory 370 (step SH1). When the generation unit 359 determines that none of the recorded data is stored (step SH1: NO), it performs the determination in step SH1 again.

On the other hand, when the generation unit 359 determines that the recorded data is stored (step SH1: YES), it determines whether the stored recorded data is the first recorded data 374 or the second recorded data 375 (step SH2). Note that the determination in step SH2 is made based on the type information of the recorded data recorded in the header.

When the generation unit 359 determines that the data is the first recorded data 374 (step SH2: first recorded data), the first selection unit 358A selects the video from the moving image indicated by the first recorded data 374 stored in the second memory 370. , selects the first photographed image (step SH3).

Step SH3 will be explained in detail.
The first photographed image is a photographed image when the left door 21B and the right door 21C are opened at a rotation angle of α°≦θ≦β°, and is a photographed image in which the user P is least reflected. . α° is, for example, 60°. β° is, for example, 100°.

The first selection unit 358A converts the detected value of the left door ranging sensor 308, which is associated with the first recorded data 374, into a rotation angle of the left door 21B, and which is associated with the first recorded data 374. The detected value of the right door ranging sensor 309 is converted into the rotation angle of the right door 21C.
The first selection unit 358A refers to the first registered data 373A and selects the detected value of the left door ranging sensor 308 associated with the first recorded data 374, for example, as follows: Convert to angle. The first selection unit 358A acquires the detection value of the left door distance measurement sensor 308 corresponding to 0° and the detection value of the left door distance measurement sensor 308 corresponding to 90° from the first registration data 373A. . Next, the first selection unit 358A determines to which rotation angle the detection value of the left door ranging sensor 308 associated with the first recorded data 374 corresponds, based on the acquired two types of detection values. Recognize what is there. This recognition method involves, for example, determining the degree of change in the detected value of the left door distance measurement sensor 308 with respect to the rotation angle of the left door 21B from the detected value of 0° and the detected value of 90°, and according to the determined degree of change. This is a method of recognizing the rotation angle. That is, using at least two detected values, the degree of change in the detected value of the left door ranging sensor 308 with respect to the rotation angle of the left door 21B is determined, and the rotation angle is recognized according to the determined degree of change. The distance measurement sensor may have three or more detected values. The first selection unit 358A then converts the detected value of the left door distance measurement sensor 308 associated with the first recorded data 374 into the recognized rotation angle. The first selection unit 358A similarly converts the detection value of the right door distance measurement sensor 309 into the rotation angle of the right door 21C.

Next, the first selection unit 358A refers to the converted rotation angle and selects a captured image when the left door 21B and the right door 21C are opened at a rotation angle of α°≦θ≦β° as the first recorded data. Select from 374.
Next, the first selection unit 358A selects the captured image in which the user P is least reflected from the selected captured images as the first captured image. The first selection unit 358A recognizes the user P appearing in the photographed image using a method such as pattern matching, and selects the photographed image in which the recognized image of the user P has the smallest area as the photographed image in which the user P is least reflected. .

The first selection unit 358A determines whether the first captured image has been selected (step SH4). If the first selection unit 358A determines that the first captured image could not be selected (step SH4: NO), the second processor 350 performs the process of step SH11.

On the other hand, when the first selection unit 358A determines that the first captured image has been selected (step SH4: YES), the generation unit 359 generates first upload data (step SH5). The first upload data includes the image data of the first photographed image selected by the first selection unit 358A and the photographing device ID 372 stored in the second memory 370.

Next, the third communication control unit 352 transmits the first upload data generated by the generation unit 359 to the article management server 5 (step SH6).

Returning to the explanation of step SH2, if the generation unit 359 determines that the second recorded data 375 is the second recorded data (step SH2: second recorded data), the second selection unit 358B selects the second recorded data stored in the second memory 370. A second captured image is selected from the moving images indicated by the data 375 (step SH7).

Step SH7 will be explained in detail.
The second photographed image is a photographed image in which the drawer 28A whose degree of withdrawal is γ% or more is captured when the degree of withdrawal in the closed state is 0% and the degree of withdrawal in the predetermined open state is 100%. Note that the withdrawal degree refers to the degree of withdrawal. Furthermore, the second photographed image is the photographed image in which the user P is least reflected. γ% is, for example, 70%.

First, the second selection unit 358B reads from the second memory 370 the second registration data 373B corresponding to the type of the drawer 28A recorded in the second recorded data 375 to be processed. Then, the second selection unit 358B recognizes the position of the front edge of the drawer 28A recorded in the read second registered data 373B as the position when the degree of draw-out is 100%, and also takes a photograph using the narrow-angle camera 306. The position of the upper end of the captured image is recognized as the position of the front edge of the drawer 28A when the degree of drawing is 0%. Note that this recognition is based on the premise that the direction in which the drawer 28A is pulled out is the direction from the upper end to the lower end of the photographed image in the photographed image by the narrow-angle camera 306. Then, the second selection unit 358B selects a photographed image in which the front edge of the drawer 28A is located at a position of γ% or more based on the two recognized positions. Next, the second selection unit 358B selects the captured image in which the user P is least reflected from the selected captured images as the second captured image.

The second selection unit 358B determines whether the second captured image has been selected (step SH8). If the second selection unit 358B determines that the second captured image could not be selected (step SH8: NO), the second processor 350 performs the process of step SH11.

On the other hand, if the second selection unit 358B determines that the second captured image has been selected (step SH8: YES), the generation unit 359 generates second upload data (step SH9). The second upload data includes the image data of the second photographed image selected by the second selection unit 358B and the photographing device ID 372 stored in the second memory 370.

Next, the third communication control unit 352 transmits the second upload data generated by the generation unit 359 to the article management server 5 (step SH10).

The generation unit 359 erases the recorded data that has been processed after step SH2 from the second memory 370 (step SH11). When the transmission in step SH6 is performed, the generation unit 359 erases the first recorded data 374 from the second memory 370 in step SH11. When the transmission in step SH10 is performed, the generation unit 359 erases the second recorded data 375 from the second memory 370 in step SH11.

As shown in the flowchart FI, the server communication control unit 501 determines whether or not the first upload data or the second upload data has been received from the photographing device 3 (step SI1). When the server communication control unit 501 determines that the first upload data or the second upload data has been received (step SI1: YES), the server processing unit 502 processes the article based on the upload data received by the server communication control unit 501. The management DB 512 is updated (step SI2).

Step SI2 will be explained in detail.
The server processing unit 502 identifies a record including the photographing device ID 372 included in the first upload data from the article management DB 512, and converts the image data of the first photographed image of the specified record into the first photographed image included in the first upload data. Update to image data.
The server processing unit 502 specifies a record including the photographing device ID 372 included in the second upload data from the article management DB 512, and converts the image data of the second photographed image of the identified record into the second photographed image included in the second upload data. Update to image data.

[1-2-5. Operations related to app UI]
Next, the operation of the article management system 1A related to the application UI will be explained.
FIG. 18 is a flowchart showing the operation of the article management system 1A. In FIG. 18, a flowchart FJ shows the operation of the terminal device 4, and a flowchart FK shows the operation of the article management server 5.

As shown in the flowchart FJ, the terminal communication control unit 401 determines whether to request image data stored in the article management DB 512 from the article management server 5 (step SJ1). For example, when the article management application 411 is launched, the terminal communication control unit 401 makes an affirmative determination in step SJ1.

If the terminal communication control unit 401 determines to request the image data stored in the article management DB 512 (step SJ1: YES), it transmits request data requesting the image data to the article management server 5 (step SJ2). . The request data includes the photographing device ID 372 stored in the terminal memory 41. Note that the terminal device 4 acquires the photographing device ID 372 from the photographing device 3 at the timing of communication connection with the photographing device 3.

As shown in the flowchart FK, the server communication control unit 501 receives request data (step SK1). Next, the server processing unit 502 acquires image data associated with the photographing device ID 372 included in the request data received by the server communication control unit 501 from the article management DB 512 (step SK2). Next, the server communication control unit 501 transmits the image data acquired by the server processing unit 502 to the terminal device 4 as a response to the request data (step SK3).

Referring to flowchart FJ, terminal communication control unit 401 receives image data from article management server 5 (step SJ3). Next, the display control unit 402 causes the touch panel 43 to display an application UI including the image indicated by the image data received in step SJ3 (step SJ4).

[1-3. Effects, etc.]
As described above, the photographing device 3 uses the wide-angle camera 305 to photograph the refrigerator compartment 21 of the refrigerator 2A and the wide-angle camera 305 based on the rotation angle of the rotary door 21A that opens and closes the opening of the refrigerator compartment 21. It includes a first selection unit 358A that selects a first photographed image showing the refrigerator compartment 21 with the door 21A open from the photographed moving images.
According to this, since the first photographed image is selected based on the rotation angle of the door 21A, it is possible to select the first photographed image in which the refrigerator compartment 21 is appropriately captured from the moving image photographed by the wide-angle camera 305. Therefore, from the moving images photographed by the camera 307, a photographed image that appropriately depicts the internal situation of the refrigerator 2A can be selected.

The photographing device 3 includes a main body 301, which has a distance sensor 312 that measures the distance to the door 21A, and is installed on the top surface 20A of the box 20 of the refrigerator 2A. The first selection unit 358A selects the first photographed image based on the rotation angle of the door 21A corresponding to the detection value of the distance measurement sensor 312.
According to this, it is possible to select a captured image that appropriately captures the internal situation of the refrigerator 2A using the distance measuring sensor 312. Moreover, since the main body 301 installed on the upper surface 20A of the box body 20 has the distance measurement sensor 312, the distance measurement sensor 312 can stably and accurately measure the distance to the door 21A. Therefore, a photographed image that appropriately captures the internal situation of the refrigerator 2A can be selected with high accuracy using the distance measurement sensor 312.

The distance measuring sensor 312 is installed on the main body 301 so that the sensing direction is directed toward the rotation axis of the door 21A when the main body 301 is installed on the box 20.
According to this, the distance measuring sensor 312 can appropriately measure the distance to the door 21A according to the rotation of the door 21A. Therefore, it is possible to select a photographed image that appropriately captures the internal situation of the refrigerator 2A using the distance measurement sensor 312 with higher accuracy.

The photographing device 3 includes a main body 301 installed on the upper surface 20A of the box 20 of the refrigerator 2A, and a photographing member 302 extending from the main body 301 to the front of the box 20 and having a wide-angle camera 305.
According to this, since the wide-angle camera 305 can photograph from above the refrigerator 2A, the inside of the refrigerator compartment 21 and the inner surface of the door 21A in the open state can be included in the photographed image photographed by the wide-angle camera 305. Therefore, it is possible to select a captured image that more appropriately captures the internal situation of the refrigerator 2A.

The photographing device 3 includes a narrow-angle camera 306 that photographs the drawer 28A that opens and closes the opening of the drawer chamber 28 of the refrigerator 2A. The refrigerator compartment 21 is provided above the drawer compartment 28. A recess 205 extending in the height direction of the refrigerator 2A is formed on the front surface of the door 21A in the closed state. The wide-angle camera 305 and the narrow-angle camera 306 are installed in line with the photographing member 302 in the width direction of the photographing device 3 at an interval shorter than the length of the recess 205 in the width direction of the refrigerator 2A.
According to this, the opening of the drawer 28A can be photographed even when the door 21A is closed. Furthermore, since the photographing device 3 can be designed so that the photographing member 302 of the photographing device 3 does not protrude in front of the refrigerator 2A as much as possible, the photographing device 3 that is installed in the refrigerator 2A forming the recess 205 can be made smaller. be able to.

The photographing device 3 uses a narrow-angle camera 306 to photograph the drawer 28A opening and closing the opening of the drawer chamber 28 of the refrigerator 2A, and based on the degree to which the drawer 28A is pulled out, from the moving image photographed by the narrow-angle camera 306. A second selection section 358B is provided that selects a second photographed image in which the drawer 28A in an open state is captured.
According to this, since the second photographed image is selected based on the degree to which the drawer 28A is pulled out, it is possible to select the second photographed image in which the drawer 28A is properly captured from the moving image photographed by the narrow-angle camera 306. Therefore, from the moving images photographed by the camera 307, a photographed image that appropriately depicts the internal situation of the refrigerator 2A can be selected.

The photographing device 3 includes a main body 301 installed on the upper surface 20A of the box 20 of the refrigerator 2A, and a photographing member 302 extending from the main body 301 to the front of the box 20 and having a narrow-angle camera 306.
According to this, since the narrow-angle camera 306 can photograph from above the refrigerator 2A, the opening of the drawer 28A can be included in the photographed image photographed by the narrow-angle camera 306. Therefore, it is possible to select a captured image that more appropriately captures the internal situation of the refrigerator 2A.

The photographing device 3 includes a wide-angle camera 305 that photographs the refrigerator compartment 21 of the refrigerator 2A. The refrigerator compartment 21 is provided above the drawer compartment 28. A recess 205 extending in the height direction of the refrigerator 2A is formed on the front surface of the door 21A in the closed state. The wide-angle camera 305 and the narrow-angle camera 306 are installed in line with the photographing member 302 in the width direction of the photographing device 3 at an interval shorter than the length of the recess 205 in the width direction of the refrigerator 2A.
According to this, the opening of the drawer 28A can be photographed even when the door 21A is closed. Furthermore, since the photographing device 3 can be designed so that the photographing member 302 of the photographing device 3 does not protrude in front of the refrigerator 2A as much as possible, the photographing device 3 that is installed in the refrigerator 2A forming the recess 205 can be made smaller. be able to.

The information processing system 7A includes a photographing device 3 and a terminal device 4 that communicates with the photographing device 3. The photographing device 3 includes a camera 307 that photographs the box 20 in which the storage chamber 27 is formed, a refrigerator 2A having an opening/closing body 29A that opens and closes the opening of the storage chamber 27, and registers a predetermined open state of the opening/closing body 29A. and a selection unit 358 that selects a captured image when the opening/closing body 29A is in the open state from the images captured by the camera 307 based on the predetermined open state registered by the registration unit 360. Be prepared.
According to this, in order to select a photographed image when the opening/closing body 29A is in the open state based on a registered predetermined open state, the photographed image is selected using a criterion corresponding to the refrigerator 2A photographed by the photographing device 3. can be selected. Therefore, regardless of the model of the refrigerator 2A photographed by the photographing device 3, it is possible to select from the images photographed by the camera 307 a photographed image that appropriately depicts the internal situation of the refrigerator 2A.

The terminal device 4 includes a display control unit 402 that displays a guide image for guiding the state of the opening/closing body 29A to a predetermined open state, and a reception unit 403 that receives an input indicating that the state of the opening/closing body 29A has been changed to the predetermined open state. , and the registration unit 360 registers the current state of the opening/closing body 29A as a predetermined open state when the reception unit 403 receives the input.
According to this, the user P can easily set the state of the opening/closing body 29A to the predetermined open state by visually recognizing the guide image displayed by the terminal device 4. Further, the registration unit 360 can accurately register the predetermined open state of the opening/closing body 29A. Therefore, it is possible to select a captured image that more appropriately captures the internal situation of the refrigerator 2A while improving the user's convenience in registering the predetermined open state.

The terminal device 4 includes a terminal communication control unit 401 that receives a moving image photographed by the camera 307 from the photographing device 3. The display control unit 402 displays the captured image indicated by the image data received by the terminal communication control unit 401 and the guide image AG in a superimposed manner.
According to this, since the user P can intuitively set the opening/closing body 29A to the predetermined open state, it is possible to further improve the convenience for the user P in registering the predetermined open state.

The opening/closing body 29A is a rotary door 21A. The display control unit 402 displays a first guide image AG1 that is a guide image that guides the state of the door 21A to a predetermined open state. The first guide image AG1 includes a line L corresponding to the edge of the door pocket 211 of the door 21A in a predetermined open state.
According to this, since a guideline for guiding the door 21A to a predetermined open state is displayed, the user P can more intuitively and accurately set the door 21A to the predetermined open state.

The opening/closing body 29A is a drawer 28A. The display control unit 402 displays a second guide image AG2 that is a guide image that guides the state of the drawer 28A to a predetermined open state. The second guide image AG2 includes a line L corresponding to the front edge of the drawer 28A in a predetermined open state.
According to this, since a guideline for guiding the drawer 28A to a predetermined open state is displayed, the user P can more intuitively and accurately set the drawer 28A to the predetermined open state.

The display position of the line L in the guide image differs depending on the model of the refrigerator 2A photographed by the camera 307.
According to this, the user P can more intuitively and accurately set the opening/closing body 29A to a predetermined open state, regardless of the model of the refrigerator 2A.

The photographing device 3 includes a camera 307 that photographs the box 20 in which the storage chamber 27 is formed, a refrigerator 2A having an opening/closing body 29A that opens and closes the opening of the storage chamber 27, and registers a predetermined open state of the opening/closing body 29A. and a selection unit 358 that selects a captured image when the opening/closing body 29A is in the open state from the images captured by the camera 307 based on the predetermined open state registered by the registration unit 360. Be prepared.
According to this, effects similar to those of the information processing system 7A described above can be achieved.

(Embodiment 2)
Next, a second embodiment will be described. In the description of the second embodiment, the same components as those of each part of the article management system 1A of the first embodiment are given the same reference numerals, and detailed explanations are omitted as appropriate.

[2-1. composition]
[2-1-1. Configuration of goods management system]
FIG. 19 is a diagram showing the configuration of article management system 1B in the second embodiment.
Compared to the article management system 1A in the first embodiment, the article management system 1B in the present embodiment does not include the photographing device 3 and the refrigerator 2A, but instead includes the refrigerator 2B. Furthermore, compared to the information processing system 7A in the first embodiment, the information processing system 7B in this embodiment includes a refrigerator 2B instead of the photographing device 3. Note that the refrigerator 2B of this embodiment may be a refrigerator in which the recess 205 is formed in the door 21A, or may be a refrigerator in which the recess 205 is not formed.
The chilled compartment 26 formed in the refrigerator compartment 21 of the refrigerator 2B corresponds to the "second storage compartment" of the present disclosure in this embodiment.

[2-1-2. Refrigerator configuration]
FIG. 20 is a plan view of the refrigerator 2B viewed from the right. FIG. 21 is a block diagram showing the configuration of the refrigerator 2B. In FIG. 20, the same X-axis, Y-axis, and Z-axis as in FIG. 2 are illustrated.

As shown in FIG. 20, the refrigerator 2B includes a wide-angle camera 305 and a narrow-angle camera 306. These two cameras are connected to the camera control board 314 via a harness placed between the outer box 201 and the inner box 202. The wide-angle camera 305 and the narrow-angle camera 306 of the second embodiment are provided at the front end of the upper surface 21D in the refrigerator compartment 21, and take pictures of the lower part from the upper surface 21D. Note that in FIG. 20, the wide-angle camera 305 and the narrow-angle camera 306 are shown lined up in the front and back direction of the refrigerator 2B, but this is because the wide-angle camera 305 and the narrow-angle camera 306 are installed inside the refrigerator compartment 21. This is merely shown for convenience, and the installation target is not limited to the embodiment shown in FIG. 20. For example, the wide-angle camera and the narrow-angle camera 306 may be arranged in the left-right direction of the refrigerator 2B.

The photographing range of the wide-angle camera 305 of the second embodiment includes, for example, the following range when viewed from the front of the refrigerator 2B. This range includes the door pockets 211 of the left door 21B and right door 21C and the opening of the refrigerator compartment 21 when the left door 21B and right door 21C are in the open state. Furthermore, the photographing range of the wide-angle camera 305 of the second embodiment includes the following range when viewed from the right side of the refrigerator 2B. This range includes the front end of the open door 21A to the front end of the shelf 212 in the front-back direction of the refrigerator 2B, and also includes the opening of the refrigerator compartment 21 in the vertical direction.

The photographing range of the narrow-angle camera 306 of the second embodiment includes, for example, the following range when viewed from the front of the refrigerator 2B. The range includes the drawer 26A when viewed from the front of the refrigerator 2B. Further, the photographing range of the narrow-angle camera 306 of the second embodiment includes, for example, the following range when viewed from the right side of the refrigerator 2B. This range includes the drawer 26A in the most pulled out state in the front-rear direction of the refrigerator 2B.

As shown in FIGS. 20 and 21, the refrigerator 2B includes a left door distance sensor 308, a right door distance sensor 309, and a human sensor 311. These sensors are connected to a sensor control board 313.
The distance sensor 312 is installed on the top surface 20A of the box body 20. The installation manner of the left door distance measurement sensor 308 and the right door distance measurement sensor 309 is the same as in the first embodiment. Note that in FIG. 20, the left door distance sensor 308 and the right door distance sensor 309 are shown aligned in the front-rear direction of the refrigerator 2B, but this is because the distance sensor 312 is installed on the top surface 20A. This is merely shown for convenience, and the installation mode of the left door distance measurement sensor 308 and the right door distance measurement sensor 309 is not limited to that shown in FIG.
The human sensor 311 is installed at the upper front of the door 21A.

As shown in FIGS. 20 and 21, the refrigerator 2B includes a heater 222. The heater 222 heats the lenses of the wide-angle camera 305 and the narrow-angle camera 306.
The heater 222 corresponds to the "heating section" of the present disclosure.

As shown in FIG. 21, the refrigerator 2B includes a sensor control board 313 and a camera control board 314. The sensor control board 313 and the camera control board 314 are housed in a housing space of a recess formed in the top surface 20A.
A third communication module 315, a fourth communication module 316, a wide-angle camera 305, a narrow-angle camera 306, and a heater 222 included in the refrigerator 2B are connected to the camera control board 314.

The second memory 370 of the second embodiment includes first registration data 373A, third registration data 373C, and fourth registration data 373D.
The fourth registration data 373D is data in which a predetermined open state of the drawer 26A is registered. In this embodiment, the predetermined open state of the drawer 26A is the state when the drawer 26A is pulled out the most. The fourth registration data 373D records the position of the front edge of the drawer 26A when the drawer 26A is in a predetermined open state.

The second processor 350 of the second embodiment functions as a heating control section 361 in addition to each functional section of the second processor 350 of the first embodiment.

The photographing control unit 356 of the second embodiment causes the wide-angle camera 305 and the narrow-angle camera 306 to start photographing when the person presence determination unit 354 determines that there is a person around the refrigerator 2B. Furthermore, after the wide-angle camera 305 and the narrow-angle camera 306 start shooting, if the person presence determination unit 354 determines that there is no person around the refrigerator 2B, the photographing control unit 356 of the second embodiment controls the wide-angle camera 305 and the narrow-angle camera 306 to The narrow-angle camera 306 is made to finish photographing. Note that the photographing control unit 356 may turn on the first LED 317 when the wide-angle camera 305 and the narrow-angle camera 306 start photographing. Further, the photographing control unit 356 may turn off the first LED 317 when the wide-angle camera 305 and the narrow-angle camera 306 finish photographing.

When the opening/closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is open, the recording control unit 357 of the second embodiment records the shooting result of the wide-angle camera 305 and controls the narrow-angle camera. Recording of the photographing results in step 306 is started. Specifically, the recording control unit 357 stores third recording data 377 indicating a moving image captured by the wide-angle camera 305 and third recording data 377 indicating a moving image captured by the narrow-angle camera 306 in the second memory 370. 4 recording data 378 is started. When the open/close determination unit 355 determines that the door 21A is in the closed state, the recording control unit 357 of the second embodiment ends the recording of the shooting results of the wide-angle camera 305 and the recording of the shooting results of the narrow-angle camera 306. . Note that the recording control unit 357 may turn on the second LED 318 when the opening/closing determination unit 355 determines that at least one of the left door 21B and the right door 21C is in an open state. Moreover, when the open/close determination unit 355 determines that the door 21A is in the closed state, the second LED 318 may be turned off. Note that, as in the first embodiment, the recording control unit 357 of the second embodiment adds the detected value of the left door distance sensor 308 and the right door distance sensor 308 to the shooting results of the wide-angle camera 305 and the narrow-angle camera 306 during recording. and the detection value of the distance measuring sensor 309. Further, during recording, the recording control unit may associate the withdrawal degree with the photographing results of the wide-angle camera 305 during recording and the narrow-angle camera 306 during recording.

The first selection unit 358A of the second embodiment selects the first captured image from the moving images indicated by the third recorded data 377. The second selection unit 358B selects a second captured image from the moving images indicated by the fourth recorded data 378.

The registration unit 360 in the second embodiment registers information for connecting to the network NW (for example, SSID and password) by generating third registration data 373C and storing it in the second memory 370. In this embodiment, the first registration data 373A and the fourth registration data 373D are stored in the second memory 370 before the photographing device 3 starts to be used. Note that in this embodiment as well, the registration unit 360 can perform various types of registration even after starting to use the refrigerator 2B. In this case, the registration unit 360 updates the corresponding registration data to the latest data every time registration is performed.

The heating control unit 361 causes the heater 222 to heat up. The heating control unit 361 causes the heater 222 to heat for a predetermined period when the person presence determination unit 354 determines that a person exists around the refrigerator 2B. Further, the heating control unit 361 causes the heater 222 to heat up when the opening/closing determination unit 355 determines that the door 21A is in the open state. The heating control unit 361 ends the heating of the heater 222 when the opening/closing determination unit 355 determines that the door 21A is in the closed state.

[2-1-3. Article management server configuration]
The article management DB 512 included in the article management server 5 of the second embodiment has one record for each photographing device 3. This one record includes the photographing device ID 372, image data of the first photographed image, and image data of the second photographed image. In the second embodiment, the image data of the second captured image included in the record is different from the image data of the first embodiment. The second captured image in the second embodiment will be described later.

[2-2. motion]
Next, the operation of each part of the article management system 1B will be explained.

[2-2-1. Refrigerator settings]
For example, after purchasing the refrigerator 2B, the user P configures the refrigerator 2B before starting to use the refrigerator 2B. In the settings of the refrigerator 2B, at least information for connecting to the network NW is registered.

[2-2-1-1. [Registering information for connecting to local network]
The information processing system 7B of the second embodiment registers information for connecting to a local network, similarly to the information processing system 7A of the first embodiment. That is, the refrigerator 2B receives the SSID and password received by the terminal device 4 from the terminal device 4, and the registration unit 360 of the refrigerator 2B generates third registration data 373C and stores it in the second memory 370.

[2-2-2. Operations related to recording]
First, the operation of the refrigerator 2B related to recording will be explained.
Refrigerator 2B according to the second embodiment operates in the same way as the photographing device 3 according to the first embodiment. That is, the person presence determination unit 354 of the second embodiment performs the processes of steps SG1 and SG7. The opening/closing determination unit 355 of the second embodiment performs steps SG3 and SG5. The photographing control unit 356 of the second embodiment performs the processing of steps SG2 and SG8. Furthermore, the recording control unit 357 of the second embodiment performs the processing of steps SG4 and SG6.

[2-2-3. Operations related to uploading to the article management server]
Next, operations related to uploading to the article management server 5 will be explained.
FIG. 22 is a flowchart showing the operation of the article management system 1B. In FIG. 22, a flowchart FL shows the operation of the refrigerator 2B, and a flowchart FM shows the operation of the article management server 5.

The generation unit 359 determines whether the recording file 376 is stored in the second memory 370 (step SL1). When the generation unit 359 determines that the recording file 376 is not stored (step SL1: NO), it performs the determination in step SL1 again.

On the other hand, when the generation unit 359 determines that the recorded file 376 has been stored (step SL1: YES), the first selection unit 358A selects the third recorded data 377 included in the recorded file 376 stored in the second memory 370. The first photographed image is selected from the following (step SL2). The first captured image of this embodiment is the same captured image as the first captured image described in Embodiment 1. The first selection unit 358A of the second embodiment selects the first captured image similarly to the first embodiment.

The first selection unit 358A determines whether the first captured image has been selected (step SL3). If the first selection unit 358A determines that the first captured image could not be selected (step SL3: NO), the second processor 350 performs the process of step SL11.

On the other hand, when the first selection unit 358A determines that the first captured image has been selected (step SL3: YES), the second selection unit 358B selects the fourth recording included in the recording file 376 stored in the second memory 370. A second captured image is selected from the data 378 (step SL4).

The second photographed image of the present embodiment is a photographed image when the drawer 26A is in the following state, and is a photographed image in which the user P is least reflected. The above-mentioned next state is a state in which the degree of withdrawal is γ% or more when the degree of withdrawal in the closed state is 0% and the degree of withdrawal in the predetermined open state is 100%. γ% is, for example, 70%.

The second selection unit 358B recognizes the position of the front edge of the drawer 26A recorded in the fourth registration data 373D as the position when the degree of draw-out is 100%, and also recognizes the position of the front edge of the drawer 26A recorded in the fourth registration data 373D, and The position of the upper end of the image is recognized as the position of the front edge of the drawer 26A when the degree of drawing is 0%. Then, the second selection unit 358B selects a captured image in which the front edge of the drawer 26A is located at a position of γ% or more based on the two recognized positions. Next, the second selection unit 358B selects the captured image in which the user P is least reflected from the selected captured images as the second captured image.

The second selection unit 358B determines whether the second captured image has been selected (step SL5). When the second selection unit 358B determines that the second captured image has been selected (step SL5: YES), the generation unit 359 generates third upload data (step SL6). The third upload data includes the image data of the first photographed image selected in step SL1, the image data of the second photographed image selected in step SL4, and the photographing device ID 372 stored in the second memory 370.

Next, the third communication control unit 352 transmits the third upload data generated by the generation unit 359 to the article management server 5 (step SL7).

Returning to the explanation of step SL5, if the second selection unit 358B determines that the second captured image could not be selected (step SL5: NO), the generation unit 359 generates fourth upload data (step SL8). The fourth upload data includes the image data of the first photographed image selected in step SL1 and the photographing device ID 372 stored in the second memory 370.

Next, the third communication control unit 352 transmits the fourth upload data generated by the generation unit 359 to the article management server 5 (step SL9).

The generation unit 359 deletes the recorded file 376 that has been processed from step SL2 onwards from the second memory 370 (step SL10).

As shown in the flowchart FM, the server communication control unit 501 determines whether or not the third upload data or the fourth upload data has been received from the photographing device 3 (step SM1). If the server communication control unit 501 determines that upload data has not been received (step SM1: NO), it performs the determination in step SM1 again.

On the other hand, if the server communication control unit 501 determines that the third upload data or the fourth upload data has been received (step SM1: YES), the server processing unit 502 performs a process based on the upload data received by the server communication control unit 501. and updates the article management DB 512 (step SM2).

Step SM2 will be explained in detail.
The server processing unit 502 specifies a record including the photographing device ID 372 included in the third upload data from the article management DB 512, and converts the image data of the first photographed image and the image data of the second photographed image of the specified record into the second photographed image. 3 Update the image data of the first photographed image and the image data of the second photographed image included in the upload data.
The server processing unit 502 specifies a record including the imaging device ID 372 included in the fourth upload data from the article management DB 512, and converts the image data of the first captured image of the identified record into the first captured image included in the fourth upload data. Update to image data.

[2-2-4. Operations related to app UI]
Next, the operation of the article management system 1B related to the application UI will be explained.
The article management system 1B of the second embodiment operates in the same way as the article management system 1A of the first embodiment. The terminal device 4 of this embodiment displays an image of the refrigerator compartment 21 and an image of the drawer 26A.

[2-3. Effects, etc.]
As described above, the refrigerator 2B has the refrigerator compartment 21. The refrigerator 2B uses a wide-angle camera 305 that photographs the refrigerator compartment 21 and a rotating angle of the rotary door 21A that opens and closes the opening of the refrigerator compartment 21 to determine whether the door 21A is open based on the moving image photographed by the wide-angle camera 305. 358 A of 1st selection parts which select the 1st photographed image in which the refrigerating room 21 of the state is reflected are provided.
According to this, the same effects as in the first embodiment can be achieved.

The refrigerator 2B includes a box 20 in which a refrigerator compartment 21 is formed, and a distance sensor 312 that is installed on the top surface 20A of the box 20 and measures the distance to the door 21A. The first selection unit 358A selects the first photographed image based on the rotation angle of the door 21A corresponding to the detection value of the distance measurement sensor 312.
According to this, the same effects as those of the first embodiment described above can be achieved.

The distance measurement sensor 312 is installed with the sensing direction facing the rotation axis side of the door 21A.
According to this, the same effects as those of the first embodiment described above can be achieved.

The chilled compartment 26 is formed inside the refrigerator compartment 21 . The wide-angle camera 305 is installed on the inside top surface of the refrigerator compartment 21 .
According to this, since the wide-angle camera 305 can photograph the refrigerator compartment 21 from above, the inside of the refrigerator compartment 21 and the inner surface of the door 21A in the open state can be included in the photographed image photographed by the wide-angle camera 305. Therefore, it is possible to select a captured image that more appropriately captures the internal situation of the refrigerator 2B.

The refrigerator compartment 21 is cooled by cold air. A heater 222 is provided to heat the lens of the wide-angle camera 305.
According to this, it is possible to prevent the lens of the wide-angle camera 305 from fogging up when the door 21A is in the open state, so it is possible to prevent the fogging of the lens from being reflected in the photographed image taken by the wide-angle camera 305. Therefore, it is possible to select a captured image that more appropriately captures the internal situation of the refrigerator 2B.

Refrigerator 2B has a chilled compartment 26. The refrigerator 2B uses a narrow-angle camera 306 to photograph the drawer 26A that opens and closes the opening of the chilled compartment 26, and a video image taken by the narrow-angle camera 306 based on the degree to which the drawer 26A is pulled out. 26A is included.
According to this, the same effects as in the first embodiment can be achieved.

The chilled compartment 26 is formed inside the refrigerator compartment 21 . The narrow-angle camera 306 is installed on the inside top surface of the refrigerator compartment 21 .
According to this, since the narrow-angle camera 306 can take a picture from above the refrigerator compartment 21, the opening of the drawer 26A can be included in the photographed image taken by the narrow-angle camera 306. Therefore, it is possible to select a captured image that more appropriately captures the internal situation of the refrigerator 2B.

The refrigerator compartment 21 is cooled by cold air. A heater 222 is provided to heat the lens of the narrow-angle camera 306.
According to this, it is possible to suppress the lens of the narrow-angle camera 306 from fogging when the door 21A is in the open state, and therefore it is possible to suppress the fogging of the lens from being reflected in the image taken by the narrow-angle camera 306. . Therefore, it is possible to select a captured image that more appropriately captures the internal situation of the refrigerator 2B.

(Other embodiments)
As mentioned above, the first and second embodiments have been described as examples disclosed in this application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments in which changes, replacements, additions, omissions, etc. are made. Furthermore, it is also possible to create a new embodiment by combining the components described in the first and second embodiments. Therefore, other embodiments will be illustrated below.

In Embodiment 1 and Embodiment 2 described above, the double door door 21A is illustrated. However, the door 21A is not limited to the double door door 21A. The left door 21B and the right door 21C may have different widths.

In the first and second embodiments described above, the chilled compartment 26 formed in the refrigerator compartment 21 and having an internal temperature lower than that of the refrigerator compartment 21 is exemplified. However, the drawer formed in the refrigerator compartment 21 is not limited to the chilled compartment 26. That is, a storage case such as a vegetable case may be formed in the refrigerator compartment 21.

In the first and second embodiments described above, the first LED 317 and the second LED 318 are illustrated. However, the first LED 317 and the second LED 318 may be configured as one LED. Thereby, it is possible to notify the user P that photography has started using one LED, and to illuminate the storage room 27. Furthermore, the number of parts can be reduced.

In the first and second embodiments described above, the wide-angle camera 305 and the narrow-angle camera 306 are illustrated. However, the wide-angle camera 305 and the narrow-angle camera 306 may be configured as one camera. In this case, the photographing range of the camera may be changed based on the open/close state of the open/close body 29A. Further, the camera may be switched to wide-angle or narrow-angle based on the open/close state of the open/close body 29A. Thereby, the number of parts can be reduced.

In other embodiments, the refrigerator 2A may include a communication module connectable to a local network. In this configuration, the communication module is installed to overlap the first hinge member 203A or the second hinge member 203B. Further, in the case of this configuration, a member capable of shielding radio waves may be provided between the position of the first hinge member 203A or the second hinge member 203B and the position of the installed photographing device 3. For example, the communication module of the photographing device 3 may be surrounded by metal. Further, the side surface of the photographing device 3 may be made of metal. Further, in the case of this configuration, the terminal device 4 may transmit the SSID and password passed to the photographing device 3 to the communication module of the refrigerator 2A. The main body control board of the refrigerator 2A receives the SSID and password via the communication module of the refrigerator 2A, and uses the received SSID and password to connect the refrigerator 2A to the network NW. Note that if the main body control board of the refrigerator 2A stores the SSID and password, the SSID and password may be transmitted to the photographing device 3. The photographing device 3 is connected to the network NW using the received SSID and password. Note that the main body control board of the refrigerator 2A is arranged in a housing space of a recess formed on the upper surface of the refrigerator 2A.
Moreover, in other embodiments, the refrigerator 2B may include a communication module connectable to a local network. In this configuration, the communication module is installed to overlap the first hinge member 203A or the second hinge member 203B. Furthermore, in the case of this configuration, the third upload data and the fourth upload data may be transmitted to the article management server 5 by the communication module.

In other embodiments, input of the SSID and password of the local network constructed by the communication device 6 may be accepted via an application, and the photographing device 3 may be connected to the local network.

In another embodiment, the refrigerator model information may be registered in advance via an app. The model information includes, for example, the number of refrigerator compartments, the number of door pockets, the number of doors, and the number of storage cases in the vegetable compartment and freezer compartment.

In other embodiments, the heating temperature of the heater 222 may be configured to vary depending on the temperature and humidity outside the refrigerator 2B. In this case, the refrigerator 2B is provided with a sensor that measures temperature and humidity, and the heating control unit 361 recognizes the temperature and humidity outside the refrigerator 2B using the sensor.

In the embodiment described above, the distance measuring sensor is used as an example of the sensor used to detect the opening/closing of the door 21A and the drawer 28A. However, the sensor used to detect the opening/closing of the door 21A may be a Hall IC, an angle sensor, a camera, a temperature sensor, or other sensors. Further, the sensor used to detect the opening/closing of the drawer 28A may be a Hall IC, an acceleration sensor, a mechanical switch, an angle sensor, a camera, a temperature sensor, or other sensors.

In the embodiment described above, the left door 21B and the right door 21C are opened at 90 degrees as an example of the predetermined open state of the door 21A. However, the predetermined open state of the door 21A is not limited to the open state at 90°, and may be opened at any angle such as 60° or 100°, for example. Moreover, the left door 21B and the right door 21C may be fully opened as the predetermined open state of the door 21A. The fully open state is, for example, a state where the door 21B and the right door 21C are opened at 150 degrees. In this case, the first guide image AG1 may not be shown. The same applies to the predetermined open state of the drawer 28A, and the predetermined open state of the drawer 28A is not limited to the fully drawn out state.

In other embodiments, the second processor 350 may function as a detection unit in addition to each functional unit of the second processor 350 in the first embodiment or the second embodiment. During the period from when the recording control unit 357 starts recording the camera 307 to when it ends, the opening/closing angle of at least one of the left door 21B and the right door 21C becomes an angle (for example, 45 degrees or more) that allows food to be taken out. In this case, the detection unit may cut out images of articles being taken in and taken out of the refrigerator compartment 21 from the moving image. Furthermore, if a marker is attached to the article, the detection unit may detect the article with the marker attached.

In other embodiments, in addition to each functional unit of the second processor 350 in the first embodiment or the second embodiment, the second processor 350 may function as an entry/exit determination unit. In this case, depending on the opening/closing angles of the left door 21B and the right door 21C, the entry/exit determination unit changes the boundary line used for determining entry/exit, and determines entry/exit of the article. Furthermore, depending on the degree to which the drawer 28A is pulled out, the storage/unloading determination section may change the boundary line used for determining storage/unloading, and determine whether an article is stored/unloaded.

In other embodiments, the first photographed image may be corrected and uploaded to the article management server 5. An example of image correction is correction of viewpoint conversion. The correction value for the image correction may be a value depending on the rotation angle of the door 21A, or may be a fixed value.

In the embodiment described above, the case where the first captured image and the second captured image are used for display on the terminal device 4 was exemplified. However, the first captured image and the second captured image may be used for other purposes, such as detecting articles stored in the refrigerators 2A and 2B.

In another embodiment, a configuration may be adopted in which a captured image showing the storage case provided in the drawer 28A is selected from the moving images captured by the narrow-angle camera 306 and uploaded to the article management server 5. In this case, the predetermined open state of the storage case is registered by the registration unit 360. Note that when registering the state of the accommodation case, the terminal device 4 displays the second guide image AG2. Since the second guide image AG2 includes the line L17, the user P can easily set the state of the storage case to the predetermined open state.

In the embodiment described above, the refrigerators 2A and 2B are exemplified as the "storage" of the present disclosure. However, the "storage" of the present disclosure is not limited to the refrigerators 2A and 2B, and may be other storages such as a freezer or a normal temperature storage.

In other embodiments, the chambers formed in the box bodies 20 of the refrigerators 2A and 2B may include accommodation chambers other than the five types described above, or may not include at least one of the drawer chambers 28. . In other embodiments, the number of doors provided at the front opening of the refrigerator compartment 21 may be one. In this case, the distance sensor 312 is provided on the side surface of the main body 301 that is closer to the hinge that pivotally supports the door. Thereby, the opening/closing angle of the door can be detected. Further, the distance measuring sensor 312 is provided on the upper surface 20A of the box body 20 so as to face the hinge that pivotally supports the door. Thereby, the opening/closing angle of the door can be detected.

The first processor 320, the second processor 350, the server processor 50, and the terminal processor 40 may be composed of a single processor or may be composed of a plurality of processors. These processors may be hardware programmed to implement corresponding functional units. That is, these processors may be configured with, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The configurations of the photographing device 3, refrigerator 2B, terminal device 4, and article management server 5 shown in FIGS. 9, 10, and 21 are examples, and the specific implementation form is not particularly limited. In other words, it is not necessarily necessary to implement hardware corresponding to each part individually, and it is also possible to have a configuration in which the functions of each part are realized by a single processor executing a program. Furthermore, in the embodiments described above, some of the functions implemented by software may be implemented by hardware, or some functions implemented by hardware may be implemented by software.

The step units of the operations shown in FIGS. 11, 12, 14, 16, 17, 18, and 22 are divided according to the main processing content to facilitate understanding of the operations. The operation is not limited by how the processing units are divided or by their names. Depending on the processing content, the process may be divided into more steps. Furthermore, the process may be divided so that one step unit includes more processes. Furthermore, the order of the steps may be changed as appropriate within the scope of the spirit of the present disclosure.

Note that the above-described embodiments are for illustrating the technology of the present disclosure, and therefore various changes, substitutions, additions, omissions, etc. can be made within the scope of the claims or equivalents thereof.

As described above, the information processing system and the photographing device according to the present invention can be used in applications that use photographed images showing the internal situation of a storage warehouse.

1A, 1B Goods management system 2A, 2B Refrigerator (storage)
3 Photographing device 4 Terminal device 5 Article management server 7A, 7B Information processing system 20 Box 20A Top surface 21 Refrigeration room (first storage room)
21A Door 26 Chilled room (second storage room)
26A drawer 27 storage room 28 drawer room (second storage room)
28A drawer 29A opening/closing body 205 recess 211 door pocket (accommodating member)
222 Heater (heating part)
301 Main body 302 Photographing member 305 Wide-angle camera (first camera)
306 Narrow angle camera (second camera)
307 Camera 358 Selection section 358A First selection section 358B Second selection section 360 Registration section 401 Terminal communication control section (communication control section)
402 Display control unit AG1 First guide image AG2 Second guide image L line

Claims (7)

comprising a photographing device and a terminal device that communicates with the photographing device,
The photographing device is
A camera that photographs a storage having a box body in which a storage chamber is formed and an opening/closing body that opens and closes an opening of the storage chamber;
a registration unit that registers a predetermined open state of the opening/closing body;
a selection unit that selects a photographed image when the opening/closing body is in the open state from among images photographed by the camera, based on the predetermined open state registered by the registration unit;
Information processing system. The terminal device is
a display control unit that displays a guide image that guides the state of the opening/closing body to the predetermined open state;
a reception unit that receives an input that the state of the opening/closing body is set to the predetermined open state;
The registration unit registers the current state of the opening/closing body as the predetermined open state when the reception unit receives the input.
The information processing system according to claim 1. The terminal device includes a communication control unit that receives a moving image photographed by the camera from the photographing device,
The display control unit superimposes and displays the guide image and the photographic result of the camera received by the communication control unit.
The information processing system according to claim 2. The opening/closing body is a revolving door,
The display control unit displays a first guide image that is the guide image that guides the state of the door to the predetermined open state,
The first guide image includes a line corresponding to an edge of a door pocket of the door in the predetermined open state.
The information processing system according to claim 3. The opening/closing body is a drawer,
The display control unit displays a second guide image that is the guide image that guides the state of the drawer to the predetermined open state;
The second guide image includes a line corresponding to a front edge of the drawer in the predetermined open state.
The information processing system according to claim 3. The display position of the line differs in the guide image depending on the model of the storage warehouse photographed by the camera.
The information processing system according to claim 4 or 5. A camera that photographs a storage having a box body in which a storage chamber is formed and an opening/closing body that opens and closes an opening of the storage chamber;
a registration unit that registers a predetermined open state of the opening/closing body;
a selection unit that selects a photographed image when the opening/closing body is in the open state from among images photographed by the camera, based on the predetermined open state registered by the registration unit;
Photography equipment.

Images