WO2023095536A1 - Detection device, control method for detection device, and program - Google Patents

Abstract

The present invention enables a user of a storage to understand that a detection device has detected an article being stored in a storage chamber or an article being taken out from the storage chamber.　This detection device is provided at the top of a storage, and comprises: a camera for capturing an image of a storage chamber from the front upper side of the storage; a detection unit for detecting, on the basis of the image capturing result of the camera, an article being stored in the storage chamber, or an article being taken out from the storage chamber; and a notification unit that, upon detection, by the detection unit, of an article being stored in the storage chamber or an article being taken out from the storage chamber, gives a notification to a user of the storage that the article being stored in the storage chamber or the article being taken out from the storage chamber has been detected.

Description

DETECTION DEVICE, CONTROL METHOD FOR DETECTION DEVICE, AND PROGRAM

The present disclosure relates to a detection device, a detection device control method, and a program.

Patent Document 1 discloses a technique for providing a camera on the top of a refrigerator, capturing an image when an object passing through the opening of the refrigerator is detected, and managing food stored in the refrigerator based on the image capturing result.

JP 2019-070476 A

The present disclosure provides a detection device, a control method for the detection device, and a program that enable the user of the storage to recognize that the detection device has detected an article to be stored in the storage room or an article to be delivered to the storage room.

A detection device according to the present disclosure is a detection device provided at the top of a storage, comprising a camera that captures an image of the storage from above the front of the storage, and an article that is stored in the storage based on the image captured by the camera. Alternatively, a detection unit that detects an article to be delivered from the storage room, and an article to be stored in the storage room when the detection unit detects an article to be delivered to the storage room or an article to be delivered from the storage room. Alternatively, a notification unit that notifies a user of the storage that an article to be delivered from the storage is detected.

Further, a method for controlling a detection device according to the present disclosure is a method for controlling a detection device provided in an upper part of a storage, in which a camera captures an image of the storage from above the front of the storage, and based on the results of the imaging by the camera, , an article entering the storage room or an article leaving the storage room is detected, and an article entering the storage room or an article leaving the storage room is detected; A user of said storage is notified of the detection of an item being stored in or leaving said storage.

Further, the program according to the present disclosure includes a photographing control unit that causes a processor of a detection device provided in the upper part of the storage to photograph the storage chamber from above the front of the storage, and based on the photographing result of the camera, the a detection unit for detecting an article entering a storage room or an article leaving the storage room; It functions as a notification unit that notifies the user of the storage that an article to be stored in the storage room or an article to be delivered from the storage room has been detected.
This specification includes all the contents of Japanese Patent Application No. 2021-193593 filed on November 29, 2021.

The detection device, the control method of the detection device, and the program according to the present disclosure allow the storage user to know that the detection device has detected an article to be stored in the storage room or an article to be delivered to the storage room.

FIG. 1 is a diagram showing the configuration of an article management system FIG. 2 is a front view of the detection device as seen from the front side FIG. 3 is a plan view of the detection device viewed from the right side FIG. 4 is a block diagram showing the configuration of the detection device FIG. 5 is a block diagram showing the configuration of a terminal device and an article management server; FIG. 6 is a diagram showing an example of a record stored in an article management DB; FIG. 7 is a flow chart showing the operation of the detection device FIG. 8 is an explanatory diagram for explaining the specific area FIG. 9 is an explanatory diagram for explaining the extraction range FIG. 10 is an explanatory diagram for explaining entering/leaving judgment FIG. 11 is an explanatory diagram for explaining entering/leaving judgment FIG. 12 is a flow chart showing the operation of the article management system FIG. 13 is a flow chart showing the operation of the article management system FIG. 14 is a diagram showing an example of an input screen;

(Knowledge, etc. on which this disclosure is based)
At the time when the inventors came up with the present disclosure, there was a technology for detecting articles entering and leaving storage such as refrigerators. However, the inventors of the present invention have discovered a problem in the prior art that the user of the storage cannot know whether or not the detection device has detected an article entering or leaving the storage. has come to constitute the subject matter of the disclosure.
Therefore, the present disclosure provides a detection device, a control method for the detection device, and a program that enable a user of the storage to recognize that the detection device has detected an article entering or leaving the storage. .

Hereinafter, embodiments will be described in detail with reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially the same configurations may be omitted.
It should be noted that the accompanying drawings and the following description are provided to allow those skilled in the art to fully understand the present disclosure and are not intended to limit the claimed subject matter thereby.

(Embodiment 1)
Embodiment 1 will be described.
[1. composition]
[1-1. Configuration of goods management system]
FIG. 1 is a diagram showing the configuration of an article management system 1000. As shown in FIG.
The article management system 1000 is a system that manages the article name, which is the name of the article BP, and the storage start date and time, which is the date and time when the storage of the article BP in the refrigerator compartment 11 is started, for the article BP stored in the refrigerator compartment 11 .
The refrigerator compartment 11 corresponds to the "storage compartment" of the present disclosure.

In the present embodiment, the goods BP include food and containers containing food. In addition, in the present embodiment, the article BP managed by the article management system 1000 is marked with a marker MK. The marker MK may be printed on the article BP or attached to the article BP. The marker MK functions as a mark for setting the article BP as an object of management in the article management system 1000 . The marker MK may be a two-dimensional code such as a QR code (registered trademark), a bar code, or a character string, and the specific mode is not limited.

An article management system 1000 includes a refrigerator 1 . A refrigerator 1 has a main box body 10 with an open front surface. A refrigerator compartment 11 , an ice making compartment 12 , a fresh freezing compartment 13 , a freezing compartment 14 and a vegetable compartment 15 are formed in the main box body 10 . Inside the refrigerator compartment 11, a chilled compartment 11D is provided. The chilled compartment 11</b>D is an accommodation compartment having an internal temperature lower than that of the refrigerator compartment 11 . The internal temperature of the chilled compartment 11D is set, for example, between 0.degree. C. and 3.degree. The front opening of the refrigerator compartment 11 is provided with a rotary left door 11A and a rotary right door 11B. The ice making compartment 12, the fresh freezing compartment 13, the freezing compartment 14, and the vegetable compartment 15 are provided with drawers 12A, 13A, 14A, and 15A, respectively, for storing articles BP.
In the following description, when the left door 11A and the right door 11B are not distinguished from each other, they are referred to as "doors" and denoted by "11C". Moreover, in the following description, when the drawers 12A, 13A, 14A, and 15A are not distinguished, they are referred to as "drawers" and denoted by "16A".
The refrigerator 1 corresponds to the "storage" of the present disclosure.

The article management system 1000 includes a detection device 2. The detection device 2 detects articles BP stored in the refrigerator 1 . Further, the detection device 2 detects the article BP that is delivered from the refrigerator 1 . The detection device 2 is provided on the upper surface 10A of the main housing 10 of the refrigerator 1 .

An article management system 1000 includes a terminal device 3 . The terminal device 3 is, for example, a smart phone or a tablet terminal. The terminal device 3 is installed with an application program related to management of the articles BP stored in the refrigerator compartment 11, and communicates with the article management server 4 by the function of this application program.
In the following description, this application program is referred to as an "item management application" and denoted by "311".

In FIG. 1, the user P who is at home is indicated by a solid line, and the user P who is out of the home H is indicated by a dotted line. When the terminal device 3 is used by a user P who is at home, the terminal device 3 communicates with the article management server 4 via the communication device 5 or not via the communication device 5 by the function of the article management application 311 . Further, when the terminal device 3 is used by the user P who has left home H and cannot establish a communication connection with the communication device 5 , the function of the product management application 311 enables the terminal device 3 to access the article management server without the communication device 5 . Communicate with 4.

The communication device 5 connects to the communication network NW and communicates with the article management server 4 via the communication network NW. The communication network NW is a network composed of a public line network, a leased line, other communication lines, and various types of communication equipment, and is not limited to specific aspects. For example, the communication network NW may be a wide area network. The communication network NW may be configured to include at least one of a wireless communication circuit and a wired communication circuit. The communication device 5 functions as an interface device for connecting the detection device 2 and the terminal device 3 to the communication network NW.

The item management system 1000 includes an item management server 4. The article management server 4 is a server device that manages articles BP stored in the refrigerator compartment 11 . The article management server 4 connects to the communication network NW. In each figure, the article management server 4 is represented by one block, but this does not necessarily mean that the article management server 4 is composed of a single server device. For example, the article management server 4 may be configured including a plurality of server devices with different processing contents.

[1-2. Configuration of detection device]
The detection device 2 will be described with reference to FIGS. 2 to 4. FIG.
2-4, the X, Y, and Z axes are illustrated. The X-axis, Y-axis, and Z-axis are orthogonal to each other. The Z-axis indicates the vertical direction. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the horizontal direction. The Y-axis indicates the front-rear direction. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the forward direction. The positive direction of the Z-axis indicates the upward direction.

FIG. 2 is a front view of the detection device 2 provided in the refrigerator 1 as seen from the front. In the refrigerator 1 shown in FIG. 2, the left door 11A and the right door 11B are open. FIG. 3 is a plan view of the detector 2 provided in the refrigerator 1 as viewed from the right side. Refrigerator 1 shown in FIG. 3 has drawer 15A in an open state, that is, in a drawn-out state.

The detection device 2 includes a refrigerator camera 27 and a drawer camera 28 .
The refrigerator camera 27 is a camera with a wider angle than the drawer camera 28 . Refrigerating-compartment camera 27 photographs refrigerator 1 from above the front of refrigerator 1 in a state where detector 2 is installed in refrigerator 1 . The photographed result of the refrigerator camera 27 is a moving image.
The refrigerator camera 27 corresponds to the "camera" of the present disclosure.

For example, the photographing range of the refrigerator camera 27 includes the range A1 shown in FIG. 2 when viewed from the front of the refrigerator 1. The range A1 is a range including the door pocket 111 of the door 11C and the opening of the refrigerator compartment 11 when the left door 11A and the right door 11B are open. In addition, the shooting range of the refrigerator compartment camera 27 includes the range A3 shown in FIG. 3 when viewed from the right side of the refrigerator 1 . Range A3 is a range that includes the open door 11C to the front end of the shelf 112 in the front-rear direction, and a range that includes the opening of the refrigerator compartment 11 in the vertical direction.

The drawer camera 28 photographs the refrigerator 1 from above the front of the refrigerator 1 . The photographed result of the drawer camera 28 is a moving image.
For example, the photographing range of the drawer camera 28 includes the range A2 shown in FIG. 2 when viewed from the front side of the refrigerator 1 . Range A2 is a range including drawer 16A when viewed from the front side of refrigerator 1 . Also, for example, the photographing range of the drawer camera 28 includes the range A4 shown in FIG. 3 when viewed from the right side of the refrigerator 1 . Range A4 is a range including drawer 16A in the most pulled out state in the front-rear direction.

The detection device 2 includes a left door ranging sensor 23 , a right door ranging sensor 24 , and a drawer ranging sensor 25 .
The left door ranging sensor 23 is a ranging sensor. The left door distance measuring sensor 23 detects the distance between the left door 11A and the left door distance measuring sensor 23 at a predetermined cycle, and outputs a detection value indicating the detected distance to the detector processor 20 .
The right door ranging sensor 24 is a ranging sensor. The right door distance measuring sensor 24 detects the distance between the right door 11B and the right door distance measuring sensor 24 at a predetermined cycle, and outputs a detection value indicating the detected distance to the detector processor 20 .
The drawer ranging sensor 25 is a ranging sensor. The drawer distance measuring sensor 25 detects the distance between the drawer 16A and the drawer distance measuring sensor 25 at a predetermined cycle, and outputs a detection value indicating the detected distance to the detector processor 20 .

The left door ranging sensor 23, right door ranging sensor 24, and drawer ranging sensor 25 may be optical ranging sensors or ultrasonic ranging sensors.

The detection device 2 includes a human sensor 26. The human sensor 26 outputs a detection value to the detection device processor 20 at predetermined intervals. The human sensor 26 may be an infrared sensor, an ultrasonic sensor, or a visible light sensor.

Note that the arrangement order and installation position of each camera and each sensor in FIGS. The arrangement order and installation position shown in FIGS. 2 and 3 are not limited.

FIG. 4 is a block diagram showing the configuration of the detection device 2. As shown in FIG.
The detection device 2 includes a detection device processor 20 that is a processor such as a CPU (Central Processing Unit) or MPC (Micro-processing unit), and a detection device memory 21 that is a memory for storing programs and data.
The detector processor 20 corresponds to the "processor" of this disclosure.

The detecting device processor 20 reads out and executes a control program 211 stored in the detecting device memory 21, thereby performing a detecting device communication control unit 201, a human presence determination unit 202, an open/close determination unit 203, a photographing control unit 204, and a recording control. It functions as a unit 205 , a detection unit 206 , an entry/exit determination unit 207 , a notification unit 208 , a recording unit 209 , and a generation unit 210 .
The person presence determination unit 202 corresponds to the “first determination unit” of the present disclosure. The open/close determination unit 203 corresponds to the “second determination unit” of the present disclosure. The entering/leaving determination unit 207 corresponds to the “third determination unit” of the present disclosure. The control program 211 corresponds to the "program" of the present disclosure.

The detection device memory 21 stores programs executed by the detection device processor 20 and data processed by the detection device processor 20 . The detector memory 21 stores a control program 211 executed by the detector processor 20, a detector ID (Identification) 212, audio data 213, and various other data. Also, the detector memory 21 stores the recorded data 214 until it is erased. The detector memory 21 has a non-volatile storage area. The detector memory 21 may also comprise a volatile storage area and constitute a work area for the detector processor 20 .

The detection device ID 212 is identification information of the detection device 2 . Examples of the detection device ID 212 include a serial number and a manufacturing number.
The audio data 213 is audio data output from the speaker 29 by the notification unit 208 .
The recorded data 214 of the present embodiment is data indicating the photographing result of the refrigerator camera 27 .

The detector processor 20 includes a detector communication unit 22, a left door distance sensor 23, a right door distance sensor 24, a drawer distance sensor 25, a motion sensor 26, a refrigerator camera 27, and a drawer camera. 28, and a speaker 29 are connected. Devices other than these devices may be connected to the detector processor 20 .

The detection device communication unit 22 is a communication interface having a communication circuit, connector, etc. conforming to a predetermined communication standard, and communicates with the article management server 4 under the control of the detection device processor 20 . In this embodiment, the communication standard used by the detecting device communication unit 22 is a wireless communication standard, but may be a wired communication standard.

As described above, the detection device processor 20 includes the detection device communication control unit 201, the person presence determination unit 202, the opening/closing determination unit 203, the photographing control unit 204, the recording control unit 205, the detection unit 206, the entering/leaving determination unit 207, the notification It functions as a unit 208 , a recording unit 209 and a generation unit 210 .

The detection device communication control unit 201 communicates with the article management server 4 via the detection device communication unit 22 .

The person presence determination unit 202 determines whether or not a person exists around the refrigerator 1 based on the detection value output by the human sensor 26 . For example, when the human sensor 26 outputs a detection value equal to or greater than a predetermined value, the human presence determining unit 202 determines that a person exists around the refrigerator 1, and the human sensor 26 outputs a detection value lower than the predetermined value. If so, it is determined that there is no person around the refrigerator 1 . When the person presence determination unit 202 determines that a person exists around the refrigerator 1 , the person presence determination unit 202 outputs first presence information indicating that a person exists to the imaging control unit 204 . Further, when the person presence determination unit 202 determines that no person exists around the refrigerator 1 , the person presence determination unit 202 outputs second presence information indicating that no person exists to the imaging control unit 204 .

The open/close determination unit 203 determines whether the left door 11A is open or closed based on the detection value of the left door distance measuring sensor 23 . The open/close determination unit 203 determines that the left door 11A is open when the left door distance measuring sensor 23 outputs a detection value that is larger than the detection value corresponding to the closed state of the left door 11A. The open/close determination unit 203 determines that the left door 11A is closed when the left door range sensor 23 outputs a detection value corresponding to the closed state of the left door 11A.
The open/close determination unit 203 determines whether the right door 11B is open or closed based on the detection value of the right door distance measuring sensor 24 . The open/close determination unit 203 determines that the right door 11B is open when the right door distance measuring sensor 24 outputs a detection value that is larger than the detection value corresponding to the closed state of the right door 11B. The open/close determination unit 203 determines that the right door 11B is closed when the right door range sensor 24 outputs a detection value corresponding to the closed state of the right door 11B.
The open/close determination unit 203 determines whether the drawer 16A is open or closed according to the distance indicated by the detection value of the drawer distance measuring sensor 25 .

When the open/close determination unit 203 determines that at least one of the left door 11A and the right door 11B is open, it outputs first door open/close information indicating that the door 11C is open to the recording control unit 205. do. When the open/close determination unit 203 determines that the left door 11A and the right door 11B are closed, the second door open/close information indicating that the left door 11A and the right door 11B are closed is sent to the recording control unit 205. Output.
When the open/close determination unit 203 determines that one of the drawers 16A is open, it outputs first drawer open/close information indicating that one of the drawers 16A is open to the recording control unit 205 . When the open/close determination unit 203 determines that all the drawers 16A are closed, it outputs second drawer open/close information indicating that all the drawers 16A are closed to the recording control unit 205 .

When the person presence determining unit 202 outputs the first presence information, the shooting control unit 204 causes the refrigerator camera 27 and the drawer camera 28 to start shooting. When the person presence determination unit 202 outputs the second existence information after the refrigerator camera 27 and the drawer camera 28 start shooting, the shooting control unit 204 causes the refrigerator camera 27 and the drawer camera 28 to finish shooting. Let

When the opening/closing determination unit 203 outputs the first door opening/closing information, the recording control unit 205 starts recording the photographed result of the refrigerator camera 27 . That is, when the opening/closing determination unit 203 outputs the first door opening/closing information, the recording control unit 205 starts recording the recording data 214 indicating the photographing result of the refrigerator camera 27 in the detection device memory 21 . When recording is started, the recording control unit 205 outputs recording start information indicating that recording has started to the notification unit 208 . When the opening/closing determination unit 203 outputs the second door opening/closing information, the recording control unit 205 ends the recording of the imaging result of the refrigerator camera 27 . That is, when the opening/closing determination unit 203 outputs the second door opening/closing information, the recording control unit 205 ends the recording of the recording data 214 to the detection device memory 21 . In addition, when the opening/closing determination unit 203 outputs the first drawer opening/closing information, the recording control unit 205 starts recording the photographed result of the drawer camera 28 . When the opening/closing determination unit 203 outputs the second drawer opening/closing information, the recording control unit 205 ends the recording of the photographed result of the drawer camera 28 that has started.

The detection unit 206 detects an article BP entering the refrigerator compartment 11 or an article BP leaving the refrigerator compartment 11 based on the recorded data 214 whose recording to the detection device memory 21 has started. The details of the detection method of the detection unit 206 will be described later. When the detection unit 206 detects an article BP to be stored in the refrigerator compartment 11 or an article BP to be delivered from the refrigerator compartment 11, the detection unit 206 outputs article detection information indicating that the article BP to be stored in and out of the refrigerator compartment 11 is detected. Output to notification unit 208 . More specifically, the detecting unit 209 determines that the article BP has been detected when the article image BPG can be cut out from the moving image indicated by the recorded data 214, and outputs article detection information to the notification unit 208. FIG. Note that the article image BPG is an image of the article BP.

The entry/exit determination unit 207 performs entry/exit determination. The entry/exit determination is a process of determining whether the article BP detected by the detection unit 206 is the article BP that has been stored in the refrigerating compartment 11 or the article BP that has been taken out of the refrigerating compartment 11 . The details of the determination method of the entering/leaving determination unit 207 will be described later. After performing the warehousing/dispatching determination, the warehousing/dispatching determination unit 207 outputs determination completion information indicating that the warehousing/dispatching determination has been completed to the notification unit 208 .

The notification unit 208 makes various notifications by causing the speaker 29 to output the audio indicated by the audio data 213 stored in the detection device memory 21 . The notification unit 208 notifies the user when the recording control unit 205 outputs the recording start information, when the detection unit 206 outputs the article detection information, and when the entering/leaving determination unit 207 outputs determination completion information. Notify P.

The recording unit 209 records various types of information in association with the recording data 214 of the imaging results for which recording has been started. Various information recorded by the storage unit 209 will be described later.

The generation unit 210 generates upload data. Upload data is data to be transmitted to the article management server 4 . The generation unit 210 outputs the generated upload data to the detection device communication control unit 201 .

[1-3. Configuration of article management server]
FIG. 5 is a block diagram showing the configuration of the terminal device 3 and the article management server 4. As shown in FIG.
First, the configuration of the article management server 4 will be described.
The article management server 4 includes a server processor 40 such as a CPU or MPC, a server memory 41 that stores programs and data, and a server communication unit 42 .

The server processor 40 functions as a server communication control unit 401 and a server processing unit 402 by reading and executing a control program 411 stored in the server memory 41 .

The server memory 41 is a memory that stores programs executed by the server processor 40 and data processed by the server processor 40 . The server memory 41 stores a control program 411 executed by the server processor 40, an article management DB (database) 412, and various other data. The server memory 41 has a nonvolatile storage area. The server memory 41 may also include a volatile storage area and constitute a work area for the server processor 40 .

FIG. 6 is a diagram showing an example of the record R stored in the article management DB 412. As shown in FIG.
One record R stored in the article management DB 412 has an account ID 4121, communication information 4122, detection device ID 212, and one or more sub-records SR. Sub-record SR is a record stored within record R. One sub-record SR has marker ID 4123 , article name information 4124 , accommodation area information 4125 , accommodation start date and time information 4126 , and article image data 4127 .

The account ID 4121 is identification information that identifies the account assigned to the user P who uses the article management application 311.

The communication information 4122 is information for communicating with the terminal device 3 used by the user P. Communication information 4122 is, for example, address information.

The marker ID 4123 is identification information of the marker MK attached to the item BP.

The product name information 4124 is information indicating the product name.

The accommodation area information 4125 is information indicating the accommodation area. The accommodation area indicated by the accommodation area information 4125 is the area in the refrigerator compartment 11 . The accommodation area information 4125 of this embodiment indicates a number assigned to each accommodation area as the accommodation area.

The storage start date/time information 4126 is information indicating the storage start date/time when storage in the refrigerator compartment 11 is started.

The article image data 4127 is the image data of the article image BPG.

Returning to the description of FIG. 5, the server communication unit 42 is a communication interface equipped with a communication circuit or the like conforming to a predetermined communication standard, and communicates with devices connected to the communication network NW according to the predetermined communication standard. In this embodiment, the server communication unit 42 communicates with the detection device 2 and the terminal device 3 .

As described above, the server processor 40 functions as a server communication control unit 401 and a server processing unit 402.

The server communication control unit 401 communicates with the detection device 2 and the terminal device 3 via the server communication unit 42 .

The server processing unit 402 performs various information processing with the detection device 2 and the terminal device 3 as clients.

[1-4. Configuration of terminal device]
Next, a functional configuration of the terminal device 3 will be described.
The terminal device 3 includes a terminal processor 30 such as a CPU or MPC, a terminal memory 31 that stores programs and data, a terminal communication unit 32 , and a touch panel 33 .

The terminal processor 30 functions as an application execution unit 300 by reading out and executing an article management application 311 stored in the terminal memory 31 .

The terminal memory 31 stores programs executed by the terminal processor 30 and data processed by the terminal processor 30 . The terminal memory 31 stores an article management application 311 executed by the terminal processor 30 and various other data. The terminal memory 31 has a nonvolatile storage area. The terminal memory 31 may also include a volatile storage area and constitute a work area for the terminal processor 30 .

The terminal communication unit 32 is a communication interface having communication circuits, connectors, and the like conforming to a predetermined communication standard, and communicates with the article management server 4 under the control of the terminal processor 30 . In this embodiment, the communication standard used by the terminal communication unit 32 is exemplified by the wireless communication standard, but may be the wired communication standard.

The touch panel 33 includes a display panel such as a liquid crystal display panel, and a touch sensor provided over the display panel or integrally therewith.

As described above, the terminal processor 30 functions as the application executing section 300 .
Application execution unit 300 communicates with article management server 4 via terminal communication unit 32 .
In addition, the application execution unit 300 causes the touch panel 33 to display a user interface related to management of articles housed in the refrigerator 1 . In the following description, this user interface is referred to as "application UI" and denoted by "330". By displaying an application UI 330 on the touch panel 33, the application execution unit 300 provides the user P with various information related to the goods BP stored in the refrigerator 1, and allows the user to make various inputs related to the goods BP stored in the refrigerator 1. received from P.

[2. motion]
Next, the operation of each part of the article management system 1000 will be described.
[2-1. Operation Related to Article Detection]
First, an operation related to detection of articles BP entering and leaving the refrigerator compartment 11 will be described.
FIG. 7 is a flowchart FA showing the operation of the detection device 2. As shown in FIG.

The person presence determination unit 202 determines whether or not a person exists around the refrigerator 1 (step SA1).

When the person presence determination unit 202 determines that a person exists around the refrigerator 1 (step SA1: YES), the photographing control unit 204 causes the refrigerator camera 27 and the drawer camera 28 to start photographing (step SA2). ).

Next, the open/close determination unit 203 determines whether or not at least one of the doors 11A and 11B is open (step SA3).

When the open/close determination unit 203 determines that the doors 11A and 11B are not open (step SA3: NO), the detection device processor 20 performs step SA16.

When the open/close determination unit 203 determines that at least one of the door 11A and the door 11B is open (step SA3: YES), the recording control unit 205 starts recording the result of photography by the refrigerator camera 27. (Step SA4).

Next, the notification unit 208 notifies the user P that the detection of the article BP has started through the speaker 29 (step SA5).

Next, the detection unit 206 determines whether or not the marker MK has been detected within the specific area TA based on the recorded data 214 indicating the photographing results for which recording was started in step SA4 (step SA6).

The specific area TA will be described with reference to FIG.
FIG. 8 is an explanatory diagram for explaining the specific area TA. FIG. 8 shows the photographing result of the refrigerator 1 photographed by the refrigerator compartment camera 27 when the doors 11A and 11B are open.

The specific area TA is an area within the front area MA formed in front of the chilled compartment 11D on the bottom surface inside the refrigerating compartment 11 . Note that the specific area TA may be the same area as the front area MA.

The detection unit 206 analyzes the moving image indicated by the recording data 214 of the imaging results whose recording was started in step SA4, and determines whether or not the marker MK has been detected within the specific area TA. The detection unit 206 recognizes the marker MK appearing in the moving image based on feature amounts such as shape and color, for example. Note that FIG. 8 shows a case where the marker MK is not positioned within the specific area TA. Therefore, at the position of the marker MK in FIG. 8, the detection unit 206 determines that the marker MK is not detected within the specific area TA.

Returning to the description of the flowchart in FIG. 7, when the detecting unit 206 determines that the marker MK is not detected within the specific area TA (step SA6: NO), the process proceeds to step SA14.

On the other hand, when the detection unit 206 determines that the marker MK has been detected within the specific area TA (step SA6: YES), it determines whether the detected marker MK has stopped for a predetermined time within the specific area TA (step SA7).

Step SA7 will be described in detail.
For example, the detection unit 206 analyzes the moving image indicated by the recorded data 214 of the imaging results whose recording was started in step SA4, and determines the center position of the marker MK detected within the specific area TA and the area of the marker MK. , when there is no difference between a plurality of consecutive frames, it is determined that the detected marker MK has stopped for a predetermined time within the specific area TA. The number of frames to be compared in determining stillness for a predetermined time is determined in advance. For example, if the marker MK is rectangular, the center position and area of the marker MK are determined based on the positions of the four corners of the marker MK in the image area of the moving image indicated by the recorded data 214 .

When the detection unit 206 determines that the detected marker MK is not stationary within the specific area TA for a predetermined period of time (step SA7: NO), the process proceeds to step SA6.

On the other hand, when the detection unit 206 determines that the detected marker MK has been stationary for a predetermined period of time within the specific area TA (step SA7: YES), the detection unit 206 reads the marker ID 4123 from the marker MK that has been determined to be stationary for a predetermined period of time (step SA8). .

Next, the detection unit 206 cuts out the article image BPG of the article BP attached with the marker MK whose marker ID 4123 has been read out from the moving image indicated by the recorded data 214 of the photographing result of which recording was started in step SA4 (step SA9). . At step SA9, the detection unit 206 cuts out the article image BPG from, for example, the frame in which the marker MK is located within the specific area TA.

The cutout range KH of the article image BPG will be described with reference to FIG.
FIG. 9 is an explanatory diagram for explaining the cutout range KH. The image shown in FIG. 9 is an image of the refrigerator 1 captured by the refrigerator compartment camera 27 when the doors 11A and 11B are open. In the image shown in FIG. 9, the user P positions the article BP within the specific area TA.

As shown in FIG. 9, the cutout range KH is within the specific area TA. FIG. 9 illustrates a case where the cutout range KH is narrower than the specific area TA, but the cutout range KH may have the same shape as the specific area TA.

The cutout range KH may be the range corresponding to the marker ID 4123 read in step SA8. Also, the cutout range KH may be a range estimated by the image difference of the article BP located within the specific area TA. When the cutout range KH is the range estimated by the image difference, the detection unit 206, for example, recognizes the article BP from a plurality of frames in which the marker MK is positioned within the specific area TA, and extracts the recognized article BP within the specific area TA. is recognized, and the recognized movement range is defined as a clipping range KH.

Returning to the description of the flowchart of FIG. 7, when the detection unit 206 cuts out the article image BPG, the notification unit 208 detects that the article BP entering the refrigerator compartment 11 or the article BP leaving the refrigerator compartment 11 is detected. The speaker 29 notifies the user P (step SA10).

Next, the warehousing/leaving determination unit 207 performs warehousing/leaving determination based on the recorded data 214 of the imaging results of which recording was started in step SA4 (step SA11).

The entry/exit determination in step SA11 is a process of determining whether the article BP indicated by the article image BPG cut out in step SA9 is the article BP that has been stored in the refrigerator 1 or the article BP that has been taken out of the refrigerator 1. be.

The entering/leaving determination unit 207 performs image analysis on the recorded data 214 and sets a first area R1, a second area R2, a third area R3, and a fourth area R4 in the moving image indicated by the recorded data 214. The first region R<b>1 is a region corresponding to the refrigerator compartment 11 . The second area R2 is an area corresponding to the left door 11A. The third region R3 is a region corresponding to the right door 11B. A fourth region R4 is a region corresponding to the outside of the refrigerator 1 .

FIG. 10 is an explanatory diagram for explaining entering/leaving determination, and is a diagram showing boundary lines among the first area R1, the second area R2, the third area R3, and the fourth area R4.
In the first region R1 corresponding to the refrigerator compartment 11, the range of the first region R1 in the recorded image 215 is always constant even if the opening degrees of the left door 11A and the right door 11B that are opened and closed by the user are different. Therefore, the entering/leaving determination unit 207 sets the preset range of the moving image indicated by the recorded data 214 to the first region R1. Boundary lines 501 shown in FIG. 10 correspond to the boundaries between the first region R1, the second region R2, the third region R3 and the fourth region R4.

Also, the second area R2 corresponding to the left door 11A and the third area R3 corresponding to the right door 11B are changed depending on the degree of opening of the left door 11A or the right door 11B opened and closed by the user. For this reason, the entering/leaving determination unit 207 detects, for example, the edges of the door pockets 111 provided on the left door 11A and the right door 11B, and based on the detected edges of the door pocket 111, the second region R2 and the third region R2 are detected. Set the range of region R3. Marks for identifying the door pockets 111 are placed at the ends of the door pockets 111 of the left door 11A and the right door 11B. You may set the range of 2nd area|region R2 and 3rd area|region R3 based on. In addition, the entering/leaving determination unit 207 sets a region other than the set first region R1, second region R2, and third region R3 as a fourth region R4 outside the refrigerator compartment 11 . A boundary line 502 shown in FIG. 10 indicates the boundary of the second region R2, and a boundary line 503 indicates the boundary of the third region R3.

The entering/leaving determination unit 207 detects the trajectory of the moving marker MK by analyzing the moving image indicated by the recorded data 214 . The warehousing/dispatching determination unit 207 determines in which of the first region R1, the second region R2, and the third region R3 the trajectory of the detected marker MK ends, and determines in which region the article BP enters the warehouse. determine whether the In addition, the entering/leaving determination unit 207 determines that the article BP has been taken out of the refrigerator compartment 11 when the track of the detected marker MK ends in the fourth region R4.

In the warehousing/leaving determination, the warehousing/leaving determination unit 207 may superimpose the warehousing/leaving determination image HG on the moving image indicated by the recording data 214 of the shooting results whose recording was started in step SA4.

FIG. 11 is a diagram for explaining the entering/leaving determination image HG.
The entering/leaving determination image HG shown in FIG. 11 shows a case where the article BP with the marker MK is superimposed on the moving image of the user P entering the refrigerator compartment 11 .

The entering/leaving determination image HG has a boundary line L1.
The boundary line L1 is a line that extends from the left end to the right end of the entering/leaving determination image HG in the figure, substantially in the center of the entering/leaving determination image HG. Boundary line L<b>1 is located in front of refrigerator 1 from the opening of refrigerator compartment 11 captured by camera 27 for refrigerator compartment.

The image area of the entering/leaving determination image HG is divided into a first area HA1 and a second area HA2 by a boundary line L1. The first area HA1 is the front side of the refrigerator 1 from the opening of the refrigerator compartment 11 captured by the refrigerator camera 27 when the moving image captured by the refrigerator camera 27 is superimposed on the entry/exit determination image HG. is the area of The second area HA2 is the rear side of the refrigerator 1 from the opening of the refrigerator compartment 11 captured by the refrigerator camera 27 when the moving image captured by the refrigerator compartment camera 27 is superimposed on the entry/exit determination image HG. is the area of

The entering/leaving determination unit 207 analyzes the moving image on which the entering/leaving determination image HG is superimposed, so that the marker MK, which has been stationary for a predetermined period of time within the specific area TA, is detected within a predetermined period of time after the notification in step SA10. It is determined whether or not it has moved from the specific area TA to the first area HA1, and whether or not it has moved from the specific area TA to the second area HA2. When the entering/leaving determination unit 207 determines that the marker MK that has been stationary for a predetermined time in the specific area TA has moved to the first area HA1, the article BP detected by the detection unit 206 is the article BP that has been delivered from the refrigerator compartment 11. Determine that there is. On the other hand, if the entering/leaving determination unit 207 determines that the marker MK that has been stationary within the specific area TA for a predetermined time has not moved to the first area HA1, that is, has moved from the specific area TA to the second area HA2, the detection unit The article BP detected by 206 is determined to be the article BP stored in the refrigerator compartment 11 .

Returning to the description of the flowchart of FIG. 7, when the entry/exit determination unit 207 performs the entry/exit determination, the notification unit 208 notifies the user P via the speaker 29 that the detected article BP has been determined to enter/exit. (Step SA12).

The recording unit 209 stores the marker ID 4123 read out in step SA8, the article image data 4127 of the article image BPG cut out in step SA9, and the article image data 4127 of the article image BPG extracted in step SA9 for the recorded data 214 of the shooting result whose recording was started in step SA4. The judgment result of the entering/leaving judgment in SA11 is associated and recorded in the detection device memory 21 (step SA13).

Next, the open/close determination unit 203 determines whether or not the open door 11C is closed (step SA14).

When the open/close determination unit 203 determines that the open door 11C is not closed (step SA14: NO), the detection device processor 20 shifts the process to step SA6.

On the other hand, when the open/close determination unit 203 determines that the open door 11C is closed (step SA14: YES), the recording control unit 205 ends the recording of the imaging result of the refrigerator camera 27 (step SA15).

Next, the person presence determination unit 202 determines whether or not there is a person around the refrigerator 1 (step SA16).

When the person presence determination unit 202 determines that a person exists around the refrigerator 1 (step SA16: YES), the detection device processor 20 shifts the process to step SA3.

On the other hand, when the person presence determining unit 202 determines that there is no person around the refrigerator 1 (step SA17: NO), the photographing control unit 204 causes the refrigerator camera 27 and the drawer camera 28 to finish photographing. (Step SA17).

After performing the processing of step SA17, the detection device processor 20 shifts the processing to step SA1, and performs the processing after step SA1 again.

[2-2. Operations related to transmission of upload data]
Next, the operation of the article management system 1000 related to transmission of upload data will be described.
FIG. 12 is a flow chart showing the operation of the article management system 1000. As shown in FIG.
In FIG. 12, a flowchart FB shows the operation of the detection device 2, and a flowchart FC shows the operation of the article management server 4. As shown in FIG.

As shown in the flowchart FB, the generation unit 209 of the detection device 2 determines whether or not a trigger for transmitting the upload data to the article management server 4 has occurred (step SB1).

Examples of triggers for sending upload data include completion of recording of the photographed result of the refrigerator camera 27 and reaching a predetermined time at the current time.

The generation unit 209 identifies one recorded data 214 from the recorded data 214 stored in the detection device memory 21 (step SB2).

Next, the generation unit 209 determines whether the determination result of the warehousing/delivery determination associated with the specified recorded data 214 indicates warehousing or warehousing (step SB3).

When the generation unit 209 determines that the item indicates a warehousing (step SB3: warehousing), it identifies the storage area in which the item BP indicated by the item image data 4127 associated with the identified recorded data 214 is stored (step SB4).

For example, the generation unit 209 identifies the storage area of the refrigerator compartment 11 in which the article is stored by analyzing the moving image indicated by the identified recorded data 214 . Also, for example, a mark for identifying a storage area is attached to the tip of each shelf 112 and each door pocket 111, and the generation unit 209 attaches this mark to the detected article BP in the moving image. The accommodation area is specified based on the positional relationship with the marker MK.

After identifying the accommodation area, the generation unit 209 generates the first upload data (step SB5).

The first upload data generated in step SB6 includes a marker ID 4123 and article image data 4127 associated with the specified recorded data 214, accommodation area information 4125 indicating a number corresponding to the accommodation area specified in step SB6, and detection and a detector ID 212 that the device memory 21 stores. Note that the correspondence between the accommodation areas and the numbers assigned to the accommodation areas is stored in advance in the detection device memory 21 .

The detecting device communication control unit 201 transmits the first upload data generated in step SB5 to the article management server 4 (step SB6).

Next, the generation unit 209 deletes the recorded data 214 specified in step SB2 from the detecting device memory 21 (step SB7).

Returning to the description of step SB3, when the generation unit 209 determines that it indicates delivery (step SB3: delivery), it creates the second upload data (step SB8).

The second upload data generated in step SB8 includes the marker ID 4123 and article image data 4127 associated with the specified recorded data 214, and the detection device ID 212 stored in the detection device memory 21.

The detecting device communication control unit 201 transmits the second upload data generated in step SB8 to the article management server 4 (step SB9).

Next, the generation unit 209 deletes the recorded data 214 specified in step SB2 from the detecting device memory 21 (step SB7).

The generation unit 209 determines whether or not there is recorded data 214 in the detection device memory 21 (step SB10).

When the generation unit 209 determines that the recording data 214 exists in the detection device memory 21 (step SB10: YES), the process returns to step SB2, and the processes from step SB2 onward are performed again.

On the other hand, when the generating unit 209 determines that the recording data 214 does not exist in the detecting device memory 21 (step SB10: NO), it returns the process to step SB1, and performs the processes after step SB1 again.

As shown in flowchart FC, the server communication control unit 401 of the article management server 4 determines whether or not any of the first upload data and the second upload data has been received (step SC1).

When the server communication control unit 401 determines that the upload data has not been received (step SC1: NO), the determination in step SC1 is performed again.

On the other hand, when the server communication control unit 401 determines that upload data has been received (step SC1: YES), the server processing unit 402 updates the article management DB 412 based on the upload data received by the server communication control unit 401. (Step SC2).

Step SC2 will be described in detail.
When the upload data received by the server communication control unit 401 is the first upload data, the server processing unit 402 identifies the record R having the detection device ID 212 included in the first upload data from the article management DB 412 . Next, the server processing unit 402 stores the next sub-record SR in the specified record R. The stored sub-record SR contains marker ID 4123, accommodation area information 4125, and article image data 4127 included in the first upload data, blank article name information 4124, and accommodation start date/time information 4126 indicating the accommodation start date/time. Store in association. The storage start date and time is, for example, the date and time when the first upload data is received.
When the upload data received by the server communication control unit 401 is the second upload data, the server processing unit 402 identifies the record R having the detection device ID 212 and the marker ID 4123 included in the second upload data from the article management DB 412. . Next, the server processing unit 402 deletes the sub-record SR having the marker ID 4123 included in the second upload data from the identified record R.

[2-3. Operations related to product name registration]
Next, the operation of the article management system 1000 related to article name registration will be described.
FIG. 13 is a flow chart showing the operation of the article management system 1000. As shown in FIG.
In FIG. 13, a flowchart FD shows the operation of the article management server 4, and a flowchart FE shows the operation of the terminal device 3. FIG.

As shown in the flowchart FD, the server communication control unit 401 of the article management server 4 determines whether or not the new sub-record SR is stored in the article management DB 412 (step SD1).

When the server communication control unit 401 of the article management server 4 determines that the sub-record SR is stored in the article management DB 412 (step SD1: YES), it transmits the input screen display data to the terminal device 3 (step SD2).
The input screen display data is data for displaying an input screen NG on the application UI 330 . The input screen NG is a screen for accepting input of an article name, which is the name of the article BP.

Step SD2 will be explained.
The server communication control unit 401 generates input screen display data including the marker ID 4123 and article image data 4127 included in the newly stored sub-record SR. Next, the server communication control unit 401 transmits the generated input screen display data to the terminal device 3 based on the communication information 4122 of the record R to which the sub-record SR is added.

As shown in flowchart FE, the application execution unit 300 of the terminal device 3 receives input screen display data from the article management server 4 (step SE1).

Next, the application execution unit 300 causes the touch panel 33 to display the application UI 330 displaying the input screen NG (step SE2).

FIG. 14 is a diagram showing an example of an input screen NG.
The input screen NG displays an article image BPG. The article image BPG displayed by the input screen NG is the article image BPG indicated by the article image data 4127 included in the input screen display data.

The input screen NG displays an input field NL for entering the product name. The user P of the terminal device 3 inputs the product name in the input field NL displayed on the input screen NG. The input screen NG in FIG. 14 illustrates a case where the product name "boiled egg" is input in the input field NL.

The input screen NG displays the confirm button B1. The confirm button B1 is a software button for confirming the content entered in the input field NL.

Returning to the description of the flowchart of FIG. 13, when the user P inputs the name of an article (step SX1) and the user P operates the confirm button B1 (step SC3), the application execution unit 300 sends the update data to the article management server. 4 (step SE3). The update data transmitted at step SE3 includes article name information 4124 indicating the article name input on the input screen NG and marker ID 4123 included in the input screen display data received at step SF1.

As shown in the flowchart FD, the server communication control unit 401 of the article management server 4 receives update data from the terminal device 3 (step SD3).

Next, the server processing unit 402 updates the article management DB 412 based on the update data received in step SD3 (step SD4).

Step SD4 will be described in detail.
Server processing unit 402 identifies sub-record SR having marker ID 4123 included in the update data from article management DB 412 . Next, the server processing unit 402 updates the product name information 4124 of the identified sub-record SR to the product name information 4124 included in the update data.

[3. effects, etc.]
As described above, the detection device 2 is provided on the upper portion of the refrigerator 1 . The detection device 2 includes a refrigerator camera 27 that captures an image of the refrigerator compartment 11 from above the front of the refrigerator 1, and based on the imaging result of the refrigerator compartment camera 27, the articles BP stored in the refrigerator compartment 11 or the The detection unit 206 detects the goods BP to be delivered, and when the detection unit 206 detects the goods BP to be stored in the refrigerating compartment 11 or the goods BP to be delivered from the refrigerating compartment 11, the goods BP to be stored in the refrigerating compartment 11 Alternatively, a notification unit 208 that notifies the user P of the refrigerator 1 that the article BP being delivered from the refrigerator compartment 11 has been detected.

Thereby, the user P of the refrigerator 1 can grasp that the detection device 2 has detected the article BP to be stored in the refrigerator compartment 11 or the article BP to be delivered to the refrigerator compartment 11 .

The detection device 2 includes a person presence determination unit 202 that determines whether or not a person exists around the refrigerator 1 . Refrigerator camera 27 shoots images during a period from when human presence determination unit 202 determines that a person exists around refrigerator 1 to when human presence determination unit 202 determines that there is no person around refrigerator 1. I do. The detection unit 206 detects an article BP entering the refrigerating compartment 11 or an article BP leaving the refrigerating compartment 11 while the refrigerating compartment camera 27 is photographing.

As a result, the detection device 2 detects that the article BP entering the refrigerator compartment 11 or the article BP leaving the refrigerator compartment 11 is detected while the user P of the refrigerator 1 is taking in or out the article BP. , the user P of the refrigerator 1 can be notified. Therefore, the user P of the refrigerator 1 can grasp that the detection device 2 has detected the article BP in real time.

When an article BP standing still for a predetermined time within the specific area TA is captured by the refrigerator camera 27, the detection unit 206 determines whether the article BP is stored in the refrigerator compartment 11 or is taken out from the refrigerator compartment 11. Detected as an article BP.

As a result, it is possible to limit the articles BP to be detected by making the articles BP that remain stationary for a predetermined period of time within the specific area TA the detection targets. Therefore, detection of an article BP other than the article BP that the user P desires to detect can be suppressed, and the article BP that the user P desires to be detected can be accurately detected. Therefore, it is possible to suppress the occurrence of a situation in which an article BP that the user P does not desire to be detected is erroneously detected and the user P of the refrigerator 1 is erroneously notified.

The detection unit 206 detects an article BP marked with a marker MK as an article BP to be stored in the refrigerator compartment 11 or an article BP to be delivered from the refrigerator compartment 11 .

This allows the user P of the refrigerator 1 to know that the detection device 2 has detected the article BP with the marker MK.

The detection device 2 includes an open/close determination unit 203 that determines whether or not the door 11C provided in the refrigerator compartment 11 is opened. When the opening/closing determination unit 203 determines that the door 11C is open, the notification unit 208 notifies the user of the refrigerator 1 that the detection of the article BP entering the refrigerator compartment 11 or the article BP leaving the refrigerator compartment 11 has started. Notify P.

This allows the user P of the refrigerator 1 to know that the detection device 2 has started detecting the article BP. Therefore, the user P of the refrigerator 1 can grasp which timing should be used as a starting point to cause the detection device 2 to detect the article BP.

The detection device 2 includes an entry/exit determination section 207 that determines whether the article BP detected by the detection section 206 is an article BP that has been stored in the refrigerating compartment 11 or an article BP that has been taken out of the refrigerating compartment 11 . The notification unit 208 notifies the user of the refrigerator 1 that the article BP detected by the detection unit 206 has been determined to be in/out when the storage/delivery determination unit 207 has determined that the article BP detected by the detection unit 206 has been stored/retrieved. Notify P.

As a result, the user P of the refrigerator 1 can grasp that the detection device 2 has determined that the detected article BP has entered or left the warehouse.

The detection device 2 has a speaker 29 . Notification unit 208 notifies through speaker 29 .

Thereby, the user P of the refrigerator 1 can easily grasp that the detection device 2 has detected the article BP to be stored in the refrigerator compartment 11 or the article BP to be delivered to the refrigerator compartment 11 .

The detection device 2 provided on the top of the refrigerator 1 is controlled by photographing the refrigerator compartment 11 from above the front of the refrigerator 1 with the refrigerator camera 27 , and based on the photographed results of the refrigerator compartment camera 27 , When an article BP to be stored or an article BP to be taken out from the refrigerating room 11 is detected, and an article BP to be put in the refrigerating room 11 or an article BP to be taken out from the refrigerating room 11 is detected, the article BP to be put in the refrigerating room 11 The user P of the refrigerator 1 is notified of the detection of the article BP being stored or the article BP being delivered from the refrigerator compartment 11 .

As a result, the same effects as those of the detection device 2 described above can be obtained.

The control program 211 includes a photographing control unit 204 that causes the detector processor 20 of the detector 2 provided in the upper part of the refrigerator 1 to photograph the refrigerator compartment camera 27 that photographs the refrigerator compartment 11 from above the front of the refrigerator 1, and the refrigerator compartment. A detection unit 206 that detects an article BP to be stored in the refrigerating room 11 or an article BP to be taken out from the refrigerating room 11 based on the photographing result of the camera 27, When the detecting unit 206 detects the article BP leaving from the refrigerator compartment 11, the notification unit notifying the user P of the refrigerator 1 that the article BP entering the refrigerating compartment 11 or the article BP leaving the refrigerating compartment 11 has been detected. 208.

As a result, the same effects as those of the detection device 2 described above can be obtained.

(Other embodiments)
As described above, the first embodiment has been described as an example disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments with modifications, replacements, additions, omissions, and the like. Also, it is possible to combine the constituent elements described in the first embodiment to form a new embodiment.
Therefore, other embodiments will be exemplified below.

In the embodiment described above, the refrigerator 1 was exemplified as the "storage" of the present disclosure. However, the "storage" of the present disclosure is not limited to the refrigerator 1, and may be other storage such as a freezer or a room temperature storage.

In the embodiment described above, the refrigerating room 11 was exemplified as the "storage room" of the present disclosure. However, the “storage compartments” of the present disclosure are not limited to refrigerator compartment 11 , but may be ice compartment 12 , fresh freeze compartment 13 , freezer compartment 14 , and vegetable compartment 15 . Further, when the “storage” of the present disclosure is a storage other than the refrigerator 1 , the “storage room” of the present disclosure corresponds to the storage room of the storage other than the refrigerator 1 .

In the above embodiment, the article BP is marked with the marker MK, but the article BP managed by the article management system 1000 does not have to be marked with the marker MK.

The embodiment described above is configured to notify the user P when the detection of the article BP is started, when the article BP is detected, and when the entry/exit determination for the detected article BP is completed. However, the notification to the user P may be made at least when the article BP is detected. In other words, the notification when the detection of the article BP is started and the notification when the entry/exit determination for the detected article BP is completed may not be performed. The user P may arbitrarily set whether or not to notify when the detection of the article BP is started. The user P may arbitrarily set the presence/absence of notification when the determination of entering/leaving is completed.

The embodiment described above is configured so that the detection device 2 notifies by voice. However, the notification mode is not limited to voice. For example, the detection device 2 may be provided with a light, and may perform various notifications by turning on and off the light. Further, for example, when the refrigerator 1 has a display, the detection device 2 may make various notifications by causing the refrigerator 1 to display the notification information on the display. In this configuration, the detection device 2 and the refrigerator 1 are connected for communication with each other.

The boundary line L1 of the entry/exit determination image HG described above is not limited to the boundary line L1 shown in FIG. For example, the boundary line L1 of the entering/leaving judgment image HG may be an uneven line such that the area including the specific area TA among the divided image areas includes the door 11C appearing in the superimposed moving image.

For example, the article management system 1000 of another embodiment may manage other items such as the remaining amount of articles BP, the expiration date of articles BP, and the expiration date of articles BP, in addition to the storage start date and time. good.

In the above-described embodiment, the configuration in which the person presence determination unit 202 determines whether or not a person exists around the refrigerator 1 based on the detection value of the human sensor 26 has been exemplified. However, the human presence determination unit 202 may perform determination based on detection values of an illuminance sensor, an electrostatic sensor provided in the grip of the refrigerator 1, or the like, instead of the human sensor 26. FIG. In the case of this electrostatic sensor, the refrigerator 1 and the detection device 2 can communicate, and the refrigerator 1 transmits the detection value of the electrostatic sensor to the detection device 2 .

In another embodiment, the terminal device 3 may store the record R. In this configuration, the application execution unit 300 appropriately updates the contents of the record R stored in the terminal device 3 in the same manner as the article management server 4 described above. As a result, the article management system 1000 does not need to have the article management server 4, and the system configuration is simplified.

In another embodiment, the types of compartments formed in the main box body 10 of the refrigerator 1 include storage compartments other than the refrigerating compartment 11, the ice making compartment 12, the fresh freezing compartment 13, the freezing compartment 14, and the vegetable compartment 15. Alternatively, at least one of the ice making compartment 12, the fresh freezing compartment 13, the freezing compartment 14, and the vegetable compartment 15 may be omitted. In another embodiment, the number of doors provided at the front opening of refrigerator compartment 11 may be one.

For example, the detection device processor 20, the terminal processor 30, and the server processor 40 may be composed of a single processor, or may be composed of multiple processors. Detector processor 20, terminal processor 30, and server processor 40 may be hardware programmed to implement the corresponding functional units. That is, the detection device processor 20, the terminal processor 30, and the server processor 40 are configured by, for example, ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The configurations of the detection device 2, the terminal device 3, and the article management server 4 shown in FIGS. 4 and 5 are examples, and the specific implementation is not particularly limited. In other words, it is not always necessary to mount hardware corresponding to each part individually, and it is of course possible to adopt a configuration in which one processor executes a program to realize the function of each part. Further, part of the functions implemented by software in the above-described embodiments may be implemented by hardware, or part of the functions implemented by hardware may be implemented by software. In addition, the specific detailed configurations of other units of the detection device 2, the terminal device 3, and the article management server 4 can be arbitrarily changed within the scope of the present disclosure.

The operation step units shown in FIGS. 7, 12, and 13 are divided according to the main processing content in order to facilitate understanding of the operation. There are no restrictions on movement. It may be divided into more steps depending on the processing contents. Also, one step unit may be divided to include more processes. Also, the order of the steps may be changed as appropriate within the scope of the present disclosure.

The control program 211 executed by the detection device processor 20 can also be realized by recording the control program 211 on a portable information recording medium. Information recording media include magnetic recording media such as hard disks, optical recording media such as CDs, semiconductor storage devices such as USB (Universal Serial Bus) memories and SSD (Solid State Drives), but other recording media It is also possible to use

It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, replacements, additions, omissions, etc. can be made within the scope of the claims or equivalents thereof.

The detection device, detection device control method, and program of the present disclosure can be used to detect articles entering and leaving the storage.

1 refrigerator (storage)
2 detection device 11 refrigerating room (storage room)
11C door 20 detector processor (processor)
27 Cold room camera (camera)
29 speaker 202 person presence determination unit (first determination unit)
203 open/close determination unit (second determination unit)
204 photographing control unit 206 detection unit 207 entering/leaving determination unit (third determination unit)
208 notification unit 211 control program (program)
BP Article MK Marker TA Specific area

Claims (10)

1. A detection device provided in the upper part of the storage,
   a camera for photographing the storage room from above the front of the storage room;
   a detection unit that detects an article entering the storage room or an article leaving the storage room based on the photographed result of the camera;
   When the detection unit detects an article entering the storage room or an article leaving the storage room, the detection of the article entering the storage room or the article leaving the storage room is detected. a notification unit for notifying a user of the repository;
   detection device.
2. A first determination unit that determines whether or not there is a person in the vicinity of the storage,
   The camera shoots from when the first determination unit determines that a person exists around the storage until the first determination unit determines that there is no person around the storage,
   The detection unit detects an article entering the storage room or an article leaving the storage room while the camera is shooting.
   A detection device according to claim 1 .
3. The detection unit is
   When an article that remains stationary for a predetermined period of time within a specific area is captured by the camera, the article is detected as an article to be stored in the storage room or an article to be delivered from the storage room.
   A detection device according to claim 1 .
4. The detection unit is
   Detecting an article marked with a marker as an article to be stored in the storage room or an article to be delivered from the storage room;
   4. A detection device according to any one of claims 1-3.
5. A second determination unit that determines whether the door provided in the storage room is opened,
   The notification unit
   When the second determination unit determines that the door is opened, notifying the user of the storage that detection of an article entering the storage room or an article leaving the storage room has started;
   5. A detection device according to any one of claims 1-4.
6. A third determination unit that determines whether the article detected by the detection unit is an article that has been stored in the storage room or an article that has been taken out from the storage room,
   The notification unit
   When the third determination unit determines whether the article detected by the detection unit is entering or leaving the warehouse, notifying the user of the storage that the article detected by the detection unit has been determined to be entering or leaving the warehouse;
   6. A detection device according to any one of claims 1-5.
7. equipped with a speaker,
   The notification unit notifies by the speaker,
   7. A detection device according to any one of claims 1-6.
8. the storage is a refrigerator,
   the storage compartment is a cold storage compartment,
   8. A detection device according to any one of claims 1-7.
9. A control method for a detection device provided at the top of a storage, comprising:
   Photographing the storage room from above the front of the storage room with a camera,
   Detecting an article entering the storage room or an article leaving the storage room based on the photographed result of the camera;
   When an article entering the storage room or an article leaving the storage room is detected, the user of the storage is notified that the article entering the storage room or the article leaving the storage room has been detected. Notice,
   A control method for the detection device.
10. The processor of the detection device provided at the top of the storage,
    a photography control unit that causes a camera that photographs the storage room from above the front of the storage room to shoot;
    a detection unit that detects an article entering the storage room or an article leaving the storage room based on the photographed result of the camera;
    When the detection unit detects an article entering the storage room or an article leaving the storage room, the detection of the article entering the storage room or the article leaving the storage room is detected. act as a notifier to notify users of the repository;
    program.

Images