WO2023166916A1 - Article management system, article management method, and program - Google Patents

Abstract

The present disclosure provides an article management system, an article management method, and a program which suppress articles left without being used and increase user convenience.　The article management system according to the present disclosure comprises: an imaging device that captures an image of the interior of a storage compartment in a storage apparatus to generate a captured image and detects, on the basis of the captured image, articles put into the storage compartment, an accommodation area inside the storage compartment where the articles are accommodated, and articles taken out from the accommodation area; and a control device that, on the basis of results of the detection by the imaging device, causes an information processing device to display information on articles that have not been taken out from the accommodation area even after a preset period has elapsed since the day when the articles were put in.

Description

Goods management system, goods management method and program

This disclosure relates to an article management system, an article management method, and a program.

Patent Document 1 discloses a technique for providing a camera on the top of a refrigerator, capturing an image when an object passing through the opening of the refrigerator is detected, and managing food stored in the refrigerator based on the image capturing result.

JP 2019-070476 A

The present disclosure provides an article management system, an article management method, and a program that suppress forgetting to use articles and improve user convenience.

This specification includes all the contents of Japanese Patent Application No. 2022-031875 filed on March 2, 2022.
An article management system according to the present disclosure captures an image of the interior of a storage room to generate a captured image. and an article to be taken out from the accommodation area, and an article that is not taken out of the accommodation area even after a preset period from the date of receipt based on the detection result of the imaging device. and a control device that causes the information processing device to display the information of.

In addition, an article management method according to the present disclosure includes the steps of capturing an image of the inside of a storage compartment of a storehouse, and based on the captured image, an item to be stored in the storage compartment and a storage area in the storage compartment in which the item is stored. and an article to be delivered from the storage area; and information processing of information on an article that has not been delivered from the storage area even after a predetermined period of time has passed since it was stored in the storage chamber. and causing the device to display.

A program according to the present disclosure causes a computer to store an article to be stored in the storage room, a storage area in the storage room in which the article is stored, and an item to be delivered from the storage area based on a captured image of the inside of the storage room. a procedure for detecting an article to be stored, a procedure for displaying on an information processing device information on an article that has not been taken out of the storage area even after a preset period has passed since it was stored in the storage; to run.

The article management system, article management method, and program according to the present disclosure display information on articles that have not been taken out of the storage area even after a preset period has passed since the date of storage, so that it is possible to avoid forgetting to use articles. can be suppressed, and the loss of articles can be reduced. Therefore, user convenience can be improved.

FIG. 1 is a diagram showing an example of the system configuration of an article management system; FIG. 2 is a diagram showing an example of the configuration of an identification information database; FIG. 3 is a diagram showing an example of the configuration of the ingredients management database; FIG. 4 is a diagram showing an example of the configuration of an article management database; FIG. 5 is a diagram showing an example of the configuration of an open state management database; Figure 6 is a front view of the refrigerator FIG. 7 is a block diagram showing the configuration of an imaging device; FIG. 8 is a top view of the refrigerator FIG. 9 is a diagram showing an example of an image captured by a refrigerator camera; FIG. 10 is a diagram showing an example of an image captured by a drawer camera; FIG. 11 is a diagram showing boundaries between the first, second, third, and fourth regions; FIG. 12 is a diagram for explaining the entering/leaving determination image; FIG. 13 is a flowchart showing the operation of the imaging device; FIG. 14 is a flow chart showing the operation of the article management system FIG. 15 is a flowchart showing the operation of the article management server FIG. 16 is a diagram showing an example of an initial screen; FIG. 17 is a diagram showing another example of the initial screen; FIG. 18 is a diagram showing a long-term storage loading/unloading status screen FIG. 19 is a diagram showing a loading/unloading status screen FIG. 20 is a diagram showing an open/close status screen FIG. 21 is a diagram showing a selection screen FIG. 22 is a diagram showing a stored item screen FIG. 23 is a diagram showing a contained item screen FIG. 24 is a flowchart showing the operation of the application execution unit; FIG. 25 is an image captured by the drawer camera, showing boundary lines between the fifth, sixth, seventh, and eighth areas;

(Knowledge, etc. on which this disclosure is based)
At the time when the inventors came up with the present disclosure, there was a technique of capturing an image of a storage such as a refrigerator with a camera and displaying information about the articles stored in the storage from the captured image captured by the camera. However, the conventional technology does not manage the elapsed time after the article is stored in the storage, and the inventors have found a problem that the food material, which is an article, may be forgotten, leading to food loss. In order to solve the problem, the subject matter of the present disclosure has been constructed.
Accordingly, the present disclosure provides an article management system, an article management method, and a program that prevent forgetting to use articles stored in a storage and reduce article loss.

Hereinafter, embodiments will be described in detail with reference to the drawings. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially the same configurations may be omitted.
It should be noted that the accompanying drawings and the following description are provided to allow those skilled in the art to fully understand the present disclosure and are not intended to limit the claimed subject matter thereby.

(Embodiment 1)
[1. System configuration]
FIG. 1 is a diagram showing an example of the system configuration of an article management system 1. As shown in FIG.
The article management system 1 includes an imaging device 3 , an information processing device 6 and an article management server 8 . The imaging device 3 , the information processing device 6 and the article management server 8 are interconnected via a network 10 . The imaging device 3 and the information processing device 6 are connected to the network 10 by wireless connection, but may be connected by wire.

Although one imaging device 3 and one information processing device 6 are shown in FIG. In addition, the number of imaging devices 3 and information processing devices 6 is not limited to the same number. It is also possible to visually recognize with the information processing device 6 .

The imaging device 3 is installed on the top surface of the refrigerator 2. The imaging device 3 includes a plurality of cameras, and when the door and drawers of the refrigerator 2 are opened, it captures an image of the inside of the refrigerator, such as the refrigerator compartment and the vegetable compartment. A captured image is generated by capturing an image of the article. An image captured by the camera may be a moving image or a still image. In addition, the articles to be stored in the refrigerator 2 or the articles to be delivered from the refrigerator 2 include food and food materials, and storage containers containing food or food materials.

The imaging device 3 analyzes the generated captured image to detect the article and the marker ID attached to the article. A marker is attached to the storage container, and the marker includes a marker ID that is identification information of the marker. The imaging device 3 generates upload data including a captured image and a marker ID, and transmits the generated upload data to the article management server 8 .

The article management server 8 manages articles stored in the refrigerator 2 based on the upload data received from the imaging device 3 .
For example, the article management server 8 manages the date and time when the article was put into the refrigerator 2, the elapsed time after the article was put into the refrigerator 2, and the like. In addition, the article management server 8 determines the number of times the door of the refrigerator 2 has been opened and closed, and the accommodation area of the refrigerator 2 in which articles that have been stored for a certain period of time or more have been stored. The article management server 8 transmits these pieces of information to the information processing device 6 and displays them on the touch panel 63 of the information processing device 6 .

The information processing device 6 is a portable computer such as a smartphone or tablet computer. As the information processing device 6, a notebook or desktop personal computer may be used. For example, the information processing device 6 downloads images and information selected by the user's operation from the article management server 8, and generates the application screen 700 based on the downloaded images and information. The information processing device 6 displays the generated application screen 700 on the touch panel 63 .

[2. Configuration of information processing device]
The information processing device 6 includes a first communication module 61 , a touch panel 63 and a first controller 65 .

The first communication module 61 is a communication interface having an interface circuit conforming to wireless communication standards. The first communication module 61 communicates with the article management server 8 under the control of the first control section 65 . In this embodiment, the communication standard used by the first communication module 61 is a wireless communication standard, but may be a wired communication standard.

The touch panel 63 includes a display panel such as a liquid crystal display panel and a touch sensor. Touch panel 63 outputs coordinate information indicating the position of the touched display panel to first control unit 65 .

The first control unit 65 is a computer device that includes a first memory 67 and a first processor 69 .

The first memory 67 includes ROM (Read Only Memory) and RAM (Random Access Memory). Illustration of ROM and RAM is omitted. The ROM stores, as control programs executed by the first processor 69, an article management application program 675, setting information of the information processing device 6, and the like. An application program is hereinafter referred to as an application. The RAM is used as a calculation area for the first processor 69 .

The first processor 69 is an arithmetic processing unit such as a CPU (Central Processing Unit) or an MPU (Micro-Processing Unit). The first processor 69 may be composed of a single processor, or may be composed of a plurality of processors. Also, the first processor 69 may be composed of a part or all of the first memory 67 or a SoC (System-on-a-chip) integrated with other circuits. Also, the first processor 69 may be configured by a combination of a CPU that executes a program and a DSP that executes predetermined arithmetic processing. Furthermore, all the functions of the first processor 69 may be implemented in hardware, or may be configured using a programmable device.

The first processor 69 functions as an application execution unit 695 by executing an article management application 675. Application execution unit 695 communicates with article management server 8 through first communication module 61 and receives data for application screen 700 to be displayed on touch panel 63 .

Also, the application execution unit 695 causes the touch panel 63 to display an application screen 700 as a user interface for managing the items stored in the refrigerator 2 . Application execution unit 695 displays application screen 700 on touch panel 63 to provide the user with various types of information about the items stored in refrigerator 2 and accept various inputs from the user regarding the items stored in refrigerator 2 .

[3. Configuration of article management server]
The article management server 8 has a second communication module 81 and a second control section 83 . The article management server 8 corresponds to a control device.

The second communication module 81 is a communication interface equipped with an interface circuit or the like conforming to a predetermined communication standard. The second communication module 81 is wired to the network 10 and performs data communication with the imaging device 3 and the information processing device 6 .

The second control unit 83 has a second memory 85 and a second processor 87 .

The second memory 85 includes auxiliary storage devices such as ROM, RAM, HDD (Hard Disk Drive) and SSD (Solid State Drive). Illustration of ROM, RAM and HDD is omitted.
ROM stores a control program 851 . The RAM is used as a calculation area for the second processor 87 .

The auxiliary storage stores an identification information database 91, a food management database 93, an item management database 95, and an open state management database 97. Hereinafter, the database is abbreviated as DB.

The second processor 87 is an arithmetic processing device such as a CPU or MPU. The second processor 87 may be composed of a single processor, or may be composed of a plurality of processors. Also, the second processor 87 may be configured by an SoC integrated with part or all of the second memory 85 or other circuits. Also, the second processor 87 may be configured by a combination of a CPU that executes programs and a DSP that executes predetermined arithmetic processing. Furthermore, all the functions of the first processor 69 may be implemented in hardware, or may be configured using a programmable device.
The second processor 87 functions as a processing unit 871 by reading and executing the control program 851 stored in the second memory 85 .

Here, the configurations of the identification information database 91, food management database 93, article management database 95, and open state management database 97 will be described with reference to FIGS.

FIG. 2 is a diagram showing an example of the configuration of the identification information database 91. As shown in FIG.
The identification information database 91 is a database in which identification information is registered.
In the identification information database 91, account information 911, communication information 912, and imaging device ID 913 are registered as one record.

The account information 911 is information identifying an account assigned to the user of the article management application 675 .
The communication information 912 is address information such as an IP address used for communication with the information processing device 6 .
The imaging device ID 913 is an ID for identifying the imaging device 3 installed at the user's home.

FIG. 3 is a diagram showing an example of the configuration of the ingredient management DB 93. As shown in FIG.
The ingredients management DB 93 is a database generated for each account information 911, and includes one record of ingredients name information 931, color information 932, nutrient information 933, storage date/time information 934, ingredients image 935, accommodation area 936, and delivery date/time information 937. registered as.

The food name information 931 is information indicating the name of the food stored in the refrigerator 2 .
The processing unit 871 determines the ingredients captured in the captured image uploaded from the imaging device 3 . The processing unit 871 determines the food based on the color, shape, etc. of the item, and registers the determination result as the food name information 931 .

The color of the ingredient in the ingredient name information 931 is registered in the color information 932 .
When the food is determined from the captured image, the processing unit 871 acquires color information of the determined food. As for the color of the ingredients, a database in which the colors of ingredients are registered in advance may be stored in the second memory 85, and the color information may be obtained by referring to this database, or the color information of the ingredients may be obtained from another server (not shown). may be obtained. The processing unit 871 registers the acquired color as the color information 932 .

The nutrient information 933 registers information on nutrients contained in the ingredients of the ingredient name information 931. When the food is determined from the captured image, the processing unit 871 acquires nutrients contained in the determined food. Regarding the nutrients contained in foods, a database in which the nutrients contained in foods are registered in advance may be stored in the second memory 85, and nutrient information may be obtained by referring to this database, or the ingredients may be obtained from another server (not shown). nutrient information may be obtained. The processing unit 871 registers the acquired nutrients as the nutrient information 933 .

The storage date/time information 934 indicates the date/time when the ingredients of the ingredients name information 931 were stored in the refrigerator 2 . The processing unit 871 registers the date and time when the upload data is received from the imaging device 3 as the warehousing date and time information 934 . Note that the warehousing date/time information 934 may be registered based on the date/time information uploaded from the imaging device 3 .

A captured image uploaded from the imaging device 3 is registered in the food image 935 .
The imaging device 3 captures an image of the inside of the refrigerator 2 to generate a captured image. The imaging device 3 cuts out an image of the range in which the article is imaged from the generated captured image as an article image. The imaging device 3 uploads upload data including the clipped article image to the article management server 8 . The processing unit 871 extracts an article image from the upload data, and registers the article image in the ingredient image 935 when ingredients are detected from the extracted article image.

The accommodation area 936 is information indicating the accommodation area of the refrigerator 2 in which the ingredients are accommodated. The processing unit 871 extracts information indicating the accommodation area from the upload data, and registers the extracted information indicating the accommodation area in the accommodation area 936 . Identification numbers are given to the plurality of storage areas provided in the refrigerator 2 . An identification number is registered in the accommodation area 936 as information indicating the accommodation area.

The delivery date and time information 937 indicates the date and time when the food item of the food name information 931 was delivered from the refrigerator 2 . The processing unit 871 registers the date and time when the upload data is received from the imaging device 3 in the shipping date and time information 937 . Note that the delivery date/time information 937 may be registered based on the date/time information uploaded from the imaging device 3 . In some cases, foodstuffs are taken out of the refrigerator 2 for cooking or the like, and then put back into the refrigerator 2 . Therefore, even if the processing unit 871 determines that the food has been taken out of the refrigerator 2, it does not delete the record of the food material management DB 93 until a certain period of time has passed. When the used ingredient is restocked, the processing unit 871 deletes the record of the shipping date/time information 937 or registers a new warehousing date/time in the warehousing date/time information 934 .

FIG. 4 is a diagram showing an example of the configuration of the article management DB 95. As shown in FIG.
The article management DB 95 is a database generated for each account information 911, in which a marker ID 951, article image 952, warehousing date/time information 953, storage area 954, and warehousing date/time information 955 are registered as one record.

The marker ID 951 is information included in the upload data received from the imaging device 3. The imaging device 3 detects the marker attached to the article from the captured image and acquires the marker ID assigned to the detected marker. The imaging device 3 transmits the acquired marker ID to the article management server 8 as upload data. The processing unit 871 extracts the marker ID from the received upload data and registers the extracted marker ID in the marker ID 951 .

A captured image uploaded from the imaging device 3 is registered as the article image 952 .
As described above, the imaging device 3 uploads the upload data including the cut-out article image to the article management server 8 .
When the upload data includes a marker ID, the processing unit 871 registers the article image extracted from the upload data in the article image 952 .
If the upload data contains a marker ID, an article image that is stored in advance in the article management server 8 and that corresponds to the marker ID may be registered in the article image 952 . In this case, the user takes an image of the article with the marker ID and sends the image of the taken article to the article management server 8 . The processing unit 871 associates the received article image with the marker ID and registers them in the article management DB 95 .

The storage date/time information 953 indicates the date/time when the item with the marker ID was stored in the refrigerator 2 . The processing unit 871 registers the date and time when the upload data is received from the imaging device 3 in the warehousing date and time information 953 .

The storage area 954 is information indicating the storage area of the refrigerator 2 in which the article is stored. The processing unit 871 extracts information indicating the accommodation area from the upload data, and registers the extracted information indicating the accommodation area in the accommodation area 954 .

The delivery date and time information 955 indicates the date and time when the article with the marker ID was delivered from the refrigerator 2 . The processing unit 871 registers the date and time when the upload data is received from the imaging device 3 in the shipping date and time information 955 . An article may be taken out of the refrigerator 2 for cooking or the like, and then put back into the refrigerator 2 . Therefore, even if the processing unit 871 determines that the food has been taken out of the refrigerator 2, it does not delete the record of the food material management DB 93 until a certain period of time has passed. When the article is restocked, the processing unit 871 deletes the record of the shipping date/time information 955 or registers a new warehousing date/time in the warehousing date/time information 953 .

FIG. 5 is a diagram showing an example of the configuration of the open state management DB 97. As shown in FIG.
The open state management DB 97 is a database generated for each account information 911, and includes a column 971 for recording the opening date and time of the refrigerator compartment 21, a column 972 for recording the opening date and time of the vegetable compartment 27, and a column 972 for recording the opening date and time of the freezer compartment 26. It has a column 973 for
The date and time when the left door 22 or the right door 23 of the refrigerator compartment 21 is opened is registered in the column 971 for recording the date and time when the refrigerator compartment 21 is opened.
The date and time when the drawer 27A of the vegetable compartment 27 was pulled out is registered in the column 972 for recording the date and time when the vegetable compartment 27 was opened.
The date and time when the drawer 26A of the freezer compartment 26 was pulled out is registered in the column 973 for recording the date and time when the freezer compartment 26 was opened.
Further, data that has passed a certain period of time after being registered in the open state management DB 97 is deleted from the open state management DB 97 .

[4. Refrigerator configuration]
Here, the configuration of the refrigerator 2 on which the imaging device 3 is mounted will be described.
FIG. 6 is a front view of the refrigerator 2, showing a state in which the left door 22 and the right door 23 of the refrigerator 2 are opened. The X-axis, Y-axis and Z-axis shown in FIG. 6 are three axes orthogonal to each other. In the drawing, the X-axis direction corresponds to the left-right direction, and the positive direction of the X-axis corresponds to the right direction. In the drawing, the Y-axis direction corresponds to the front-rear direction, and the positive direction of the Y-axis corresponds to the front direction. As viewed in the drawing, the Z-axis corresponds to the vertical direction, and the positive direction of the Z-axis corresponds to the upward direction.

The refrigerator 2 has a heat-insulating box body 20 that is a box-shaped member that forms the main body of the refrigerator 2 .
The box body 20 has a configuration in which a space between the outer box and the inner box is filled with a heat insulating material. Illustration of the outer box and the inner box is omitted. The outer box is made of, for example, a metal plate such as iron or a hard resin such as ABS resin. The inner box is made of hard resin such as ABS resin, for example. Hard urethane foam, for example, is adopted as the heat insulating material. Note that the heat insulating material is not limited to this, and may be any material that has heat insulating properties and can be filled in the space between the outer box and the inner box.

The internal space of the refrigerator 2 is partitioned into a plurality of spaces by partition members. In this embodiment, the internal space is partitioned into five spaces, and the five spaces function as a refrigerating compartment 21, an ice making compartment 24, a fresh freezing compartment 25, a freezing compartment 26 and a vegetable compartment 27, respectively. The refrigerating compartment 21, the ice making compartment 24, the fresh freezing compartment 25, the freezing compartment 26, and the vegetable compartment 27 correspond to storage compartments. The ice making compartment 24, the fresh freezing compartment 25, the freezing compartment 26 and the vegetable compartment 27 are provided with drawers 24A, 25A, 26A and 27A, respectively, in which articles can be stored.
The refrigerator compartment 21 is closed by a left door 22 and a right door 23, the ice making compartment 24 is closed by a drawer 24A, the fresh freeze compartment 25 is closed by a drawer 25A, the freezer compartment 26 is closed by a drawer 26A, The vegetable compartment 27 is closed by a drawer 27A.

Refrigerator compartment 21 is provided with shelves 211 , 213 , 215 and 217 , and refrigerator compartment 21 is divided into five spaces by these shelf boards 211 , 213 , 215 and 217 .
The refrigerator compartment 21 partitioned by the shelf plate 211 and located at the top in the Z-axis direction is referred to as a first refrigerator compartment 21A. The refrigerator compartment 21 that is partitioned by the shelf board 211 and the shelf board 213 and located second from the top in the Z-axis direction is referred to as a second refrigerator compartment 21B. The refrigerator compartment 21 partitioned by the shelf board 213 and the shelf board 215 and positioned third from the top in the Z-axis direction is called a third refrigerator compartment 21C. The refrigerating compartment 21 that is partitioned by the shelf board 215 and the shelf board 217 and positioned fourth from the top in the Z-axis direction is called a fourth refrigerating compartment 21D. The refrigerating compartment 21 that is partitioned by the shelf plate 217 and positioned at the bottom in the Z-axis direction is called a chilling compartment 28 . The chilled compartment 28 is a storage compartment having an internal temperature lower than that of the refrigerator compartment 21 . The chilled compartment 28 is provided with a drawer 28A capable of storing articles.

A left door 22 and a right door 23 of a rotary double door are provided on the front face of the refrigerator 2 located in the positive direction of the Y axis.
The left door 22 is provided with four left door pockets. The left door 22 is provided with a first left door pocket 221, a second left door pocket 223, a third left door pocket 225, and a fourth left door pocket 227 in order from the top in the Z-axis direction.
The right door 23 is provided with four right door pockets. The right door 23 is provided with a first right door pocket 231, a second right door pocket 233, a third right door pocket 235, and a fourth right door pocket 237 in order from the top in the Z-axis direction.

[5. Configuration of imaging device]
FIG. 7 is a block diagram showing the configuration of the imaging device 3. As shown in FIG.
The configuration of the imaging device 3 will be described with reference to FIG.
The imaging device 3 includes a sensor board 30 and a camera board 40 .
The sensor board 30 is a signal processing board that processes and converts signals output from various sensors. The sensor board 30 is connected with a left door range sensor 31 , a right door range sensor 32 , a drawer range sensor 33 , and a human sensor 34 . The left door ranging sensor 31, the right door ranging sensor 32, and the drawer ranging sensor 33 correspond to open/close sensors.

The camera board 40 is a signal processing board that processes signals output by the camera module and generates captured images. The camera board 40 is connected with a third communication module 41 , a refrigerator camera 42 and a drawer camera 43 as camera modules, and a light emitting device 44 .

The left door ranging sensor 31 detects the distance to the left door 22 . Left door ranging sensor 31 outputs a signal indicating the detected distance to left door 22 to third control section 430 .
The right door ranging sensor 32 detects the distance to the right door 23 . The right door ranging sensor 32 outputs a signal indicating the detected distance to the right door 23 to the third control unit 430 .

FIG. 8 is a top view of the refrigerator 2 viewed from above.
The left door distance measuring sensor 31 includes a light emitting element that emits detection light and a light receiving element that receives light reflected by the left door 22 from the detection light. The right door distance measuring sensor 32 also includes a light emitting element that emits detection light and a light receiving element that receives the reflected light of the detection light reflected by the right door 23 . Illustration of the light-emitting element and the light-receiving element is omitted. The light-emitting element is provided in the positive direction of the Y-axis direction with respect to the light-receiving element. Thereby, the detection range of the distance measuring sensor can be widened.
The left door distance measuring sensor 31 has a light-emitting element and a light-receiving element installed so as to be inclined at a predetermined angle with respect to the positive direction of the Y-axis. The same applies to the right door distance measuring sensor 32 . When the left door 22 is opened by a predetermined angle or more, the detection light emitted from the light emitting element is reflected inside the left door 22 and above in the Z-axis direction. Further, when the left door 22 is closed, the detection light emitted from the light emitting element is reflected inside the left door 22 and above in the Z-axis direction. For example, the area inside the left door 22 and between the upper edge of the left door 22 and the gasket provided on the left door 22 reflects. It is determined that the left door 22 is open by detecting the light reflected by the upper part of the left door 22 in the Z-axis direction with the light receiving element. For example, when compared with the signal when the left door 22 is closed, if the signal changes by a predetermined value or more, it is determined that the left door 22 is open. Similarly, when the right door 23 is opened at a predetermined angle or more, the detection light emitted from the light emitting element is reflected inside the right door 23 and reflected at the upper part in the Z-axis direction. Further, when the right door 23 is closed, the detection light emitted from the light emitting element is reflected inside the right door 23 and above in the Z-axis direction. For example, the reflection occurs in the area inside the right door 23 and between the upper edge of the right door 23 and the gasket provided on the right door 23 . By detecting the light reflected by the upper part of the right door 23 in the Z-axis direction with the light receiving element, it is determined that the right door 23 is open. For example, when compared with the signal when the right door 23 is closed, if the signal changes by a predetermined value or more, it is determined that the right door 23 is open. The inside of the left door 22 is the side facing the interior of the refrigerator 2 when the left door 22 is closed. The same applies to the inside of the right door 23 .

The drawer distance measuring sensor 33 is a sensor that detects the distance to the drawers 24A, 25A, 26A, 27A, and 28A. The pull-out distance measuring sensor 33 has a light-emitting element that emits detection light and a light-receiving element that receives the reflected light of the detection light so that they face vertically downward, which is the negative direction of the Z-axis. Illustration of the light-emitting element and the light-receiving element is omitted.
When the drawers 24A, 25A, 26A, and 27A are pulled out of the interior of the refrigerator 2, the detection light emitted by the light emitting element is reflected by any one of the drawers 24A, 25A, 26A, and 27A. The light-receiving element receives light reflected by any one of the drawers 24A, 25A, 26A, and 27A. Which of the drawers 24A, 25A, 26A, and 27A has been drawn is determined from the change in the time from when the light-emitting element emits the detection light until when the light-receiving element receives the reflected light. The drawer distance measuring sensor 33 outputs a signal indicating the distance to the drawers 24A, 25A, 26A, and 27A to the third control section 430 .

The human sensor 34 detects the approach of a person. The human sensor 34 may be of infrared, ultrasonic, or visible light type. The human sensor 34 detects the approach of a person and outputs a sensor signal indicating the approach of the person to the third control unit 430 .

A first I/F circuit 310 and a signal processing section 330 are mounted on the sensor substrate 30 . The signal processing unit 330 has a third memory 340 and a third processor 350 . The first I/F circuit 310 and the signal processing section 330 may be configured by SoC (System-on-a-chip).

A first I/F circuit 310 is a communication interface with the camera board 40 . The first I/F circuit 310 includes a cable connection port and an interface circuit compatible with communication standards.

The signal processing unit 330 is a computer device that includes a third memory 340 and a third processor 350 .

The third memory 340 includes ROM and RAM. The ROM stores a control program 345 executed by the third processor 350 . The RAM is used as a calculation area for the third processor 350 .

The third processor 350 is an arithmetic processing unit such as a CPU or MPU. The third processor 350 may be composed of a single processor, or may be composed of multiple processors. Also, the third processor 350 may be configured by an SoC integrated with part or all of the third memory 340 or other circuits. Also, the third processor 350 may be configured by a combination of a CPU that executes programs and a DSP that executes predetermined arithmetic processing. Furthermore, all the functions of the third processor 350 may be implemented in hardware, or may be configured using a programmable device.

The signal processing unit 330 performs processing such as A/D conversion on the signals input from the left door ranging sensor 31, the right door ranging sensor 32, the drawer ranging sensor 33, and the motion sensor 34. and outputs the processed data to the third control unit 430 . The processed data is called sensor data.

The third communication module 41 has a connector, a communication circuit, etc., is connected to the network 10 according to communication standards such as Wi-Fi (registered trademark), and performs mutual data communication with the article management server 8.

The refrigerator camera 42 is a camera equipped with an imaging device such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. Refrigerator camera 42 captures an image under the control of third control section 430 and outputs a captured image generated by the imaging to third control section 430 . The captured image may be a still image or a moving image.

FIG. 9 is a diagram showing an example of an image captured by the refrigerator camera 42. As shown in FIG.
The angle of view of the refrigerator camera 42 is wider than the angle of view of the drawer camera 43. - 特許庁A part of the refrigerator compartment 21, the left door 22, the right door 23, the detection area TA, and the like are imaged in the captured image of the refrigerator compartment camera 42. FIG. The detection area TA is an area within the front area formed in front of the chilled compartment 28 on the bottom surface inside the refrigerating compartment 21 . The detection area TA of the present embodiment is an area smaller than the front area, but may have the same area. The imaging device 3 detects an article entering the refrigerator 2 or an article leaving the refrigerator 2 in the detection area TA.

FIG. 10 is a diagram showing an example of an image captured by the drawer camera 43. As shown in FIG.
The drawer camera 43 is also a camera equipped with an imaging device such as a CCD sensor or a CMOS sensor. The drawer camera 43 also starts imaging under the control of the third control section 430 . The drawer camera 43 also generates a captured image and outputs the generated captured image to the third control section 430 . The captured image may be a still image or a moving image.
FIG. 10 shows an image of the freezer compartment 26 pulled out from the inside of the refrigerator 2 as an example of the image captured by the drawer camera 43 .

The light-emitting device 44 includes a light-emitting element such as an LED (Light Emitting Diode), and emits light or lights under the control of the third control section 430 .

A second I/F circuit 410 and a third control section 430 are mounted on the camera board 40 .
The second I/F circuit 410 and the third control section 430 may be configured by SoC.

A second I/F circuit 410 is a communication interface with the sensor board 30 . The second I/F circuit 410 includes a cable connection port and an interface circuit compatible with communication standards.

The third control unit 430 is a computer device that includes a fourth memory 440 and a fourth processor 450 .

The fourth memory 440 includes ROM and RAM. Illustration of ROM and RAM is omitted. The ROM stores the control program 441 executed by the fourth processor 450, the imaging device ID 443, and the captured image. The imaging device ID 443 is identification information that uniquely identifies the imaging device 3 .

The fourth processor 450 is an arithmetic processing unit such as a CPU or MPU. The fourth processor 450 may be composed of a single processor, or may be composed of multiple processors. Also, the fourth processor 450 may be configured by an SoC integrated with part or all of the fourth memory 440 and other circuits. Further, the fourth processor 450 may be configured by a combination of a CPU that executes programs and a DSP that executes predetermined arithmetic processing. Furthermore, all the functions of the fourth processor 450 may be implemented in hardware, or may be configured using a programmable device.

By executing the control program 441 , the third control unit 430 functions as a person presence determination unit 451 , an opening/closing determination unit 452 , an imaging control unit 453 , an entry/exit determination unit 454 , an acquisition unit 455 and a data generation unit 456 .

The person presence determination unit 451 determines whether or not a person exists around the refrigerator 2 based on the detection value of the human sensor 34 . For example, the human presence determination unit 451 determines that a person exists around the refrigerator 2 when the detection value of the human sensor 34 is equal to or greater than a predetermined value, and determines that a person exists around the refrigerator 2 when it is less than the predetermined value. Decide not to.

The open/close determination unit 452 determines whether or not the left door 22 is open based on sensor data input from the left door ranging sensor 31 . Further, the open/close determination unit 452 determines whether or not the right door 23 is open based on sensor data input from the right door distance measuring sensor 32 .
Based on the sensor data input from the drawer distance measuring sensor 33, the open/close determination unit 452 determines whether any of the drawers 24A, 25A, 26A, 27A, and 28A has been pulled out of the refrigerator 2. determine whether or not

When the person presence determination unit 451 determines that a person exists around the refrigerator 2, the imaging control unit 453 causes the refrigerator camera 42 and the drawer camera 43 to start imaging. After that, the imaging control unit 453 lights or blinks the light emitting device 44 to notify the user of the start of imaging. When the person presence determining unit 451 determines that no person exists around the refrigerator 2 , the image capturing control unit 453 terminates the image capturing of the refrigerator compartment camera 42 and the drawer camera 43 . After that, the imaging control unit 453 turns off the light emitting device 44 .

In addition, the imaging control unit 453 determines that one of the left door 22 and the right door 23 is open, or that one of the drawers 24A, 25A, 26A, 27A, and 28A is open. , the recording of the refrigerator camera 42 and the drawer camera 43 is started. The imaging control unit 453 stores the input captured image in the fourth memory 440 . Further, the imaging control unit 453 determines that both the left door 22 and the right door 23 are closed and that all of the drawers 24A, 25A, 26A, 27A, and 28A are closed. When determined, the recording of the refrigerator camera 42 and the drawer camera 43 is ended.

Note that the imaging control unit 453 may cause the refrigerator camera 42 to start recording when the opening/closing determination unit 452 determines that one of the left door 22 and the right door 23 is open. The imaging control section 453 may store the input captured image in the fourth memory 440 . In addition, when the opening/closing determination unit 452 determines that both the left door 22 and the right door 23 are closed, the imaging control unit 453 may terminate the recording of the refrigerator camera 42 .

In addition, the imaging control unit 453 may cause the drawer camera 43 to start recording when the opening/closing determination unit 452 determines that any one of the drawers 24A, 25A, 26A, 27A, and 28A is open. In addition, the imaging control section 453 may store the input captured image in the fourth memory 440 . When the open/close determination unit 452 determines that all of the drawers 24A, 25A, 26A, 27A, and 28A are closed, the imaging control unit 453 may cause the drawer camera 43 to end recording.

The entry/exit determination unit 454 performs image analysis on the captured image to detect foodstuffs, foodstuffs, and storage containers that are articles. In addition, the entry/exit determination unit 454 determines whether the detected article is stored in the refrigerator 2 or is retrieved from the refrigerator 2 .

The process of detecting an article from a captured image and determining whether it is in or out will be described with reference to FIGS. 11 and 12. FIG. FIG. 11 is a diagram showing boundaries between the first region R1, the second region R2, the third region R3 and the fourth region R4. A case of detecting an article BP marked with a marker MK will be described below.

First, the entering/leaving determination unit 454 sets a first region R1, a second region R2, a third region R3, and a fourth region R4 in the captured image. The first region R1 is a region corresponding to the refrigerator compartment 21 in the captured image. The second region R2 is a region corresponding to the left door 22 in the captured image. The third region R3 is a region corresponding to the right door region 23 in the captured image. The fourth region R4 is a region corresponding to the outside of the refrigerator 2 in the captured image.

In the first region R1 corresponding to the refrigerator compartment 21, the range of the first region R1 in the captured image is always constant even if the opening degrees of the left door 22 and the right door 23 opened and closed by the user are different. Therefore, the entering/leaving determination unit 454 sets the preset range of the captured image to the first region R1. Boundary lines 501 shown in FIG. 11 correspond to boundaries between the first region R1, the second region R2, the third region R3, and the fourth region R4.

Also, the second region R2 corresponding to the left door 22 and the third region R3 corresponding to the right door 23 are changed depending on the degree of opening of the left door 22 or the right door 23 opened and closed by the user. For this reason, the entering/leaving determination unit 454, for example, detects the edges of the first left door pocket 221 to the fourth left door pocket 227 of the left door 22 and the edges of the first right door pocket 231 to the fourth right door pocket 237 of the right door 23. to detect

Based on the detected edges of the first left door pocket 221 to the fourth left door pocket 227 and the detected edges of the first right door pocket 231 to the fourth right door pocket 237, the entering/leaving determination unit 454 determines the second region R2 and the third region R2. Set the range of R3.
Also, marks or the like may be arranged at the ends of the first left door pocket 221 to the fourth left door pocket 227 and the first right door pocket 231 to the fourth right door pocket 237 . The entering/leaving determination unit 454 sets the ranges of the second region R2 and the third region R3 based on the position of the mark detected from the captured image. In addition, the entering/leaving determination unit 454 sets a region other than the set first region R1, second region R2, and third region R3 as a fourth region R4 outside the refrigerator compartment 21 . A boundary line 502 shown in FIG. 11 indicates the boundary of the second region R2, and a boundary line 503 indicates the boundary of the third region R3.

Note that the entering/leaving determination unit 454 may set the range of the second region R2 based on sensor data input from the left door ranging sensor 31. Further, the entering/leaving determination unit 454 may set the range of the third region R3 based on sensor data input from the right door ranging sensor 32 .

The entering/leaving determination unit 454 analyzes the captured image to detect the image of the article BP and the marker MK. The entering/leaving determination unit 454 detects the article BP and the marker MK from the area corresponding to the detection area TA of the captured image. The entering/leaving determination unit 454 detects the article BP and the marker MK, for example, based on changes in brightness and color in the detection area TA.

In addition, the entering/leaving determination unit 454 detects the trajectory of the moving article BP and the marker MK by performing image analysis on the continuously captured images.
The warehousing/dispatching determination unit 454 determines in which of the first region R1, the second region R2, and the third region R3 the trajectory of the detected marker MK ends, and determines in which region the article BP enters the warehouse. determine whether the In addition, the entering/leaving determination unit 454 determines that the article BP has been taken out of the refrigerating compartment 21 when the trajectory of the detected marker MK ends in the fourth region R4.

FIG. 12 is a diagram for explaining the entering/leaving determination image.
The warehousing/dispatching determination unit 454 superimposes the warehousing/dispatching determination image HG on the captured image to determine whether the article BP is warehousing or warehousing.
FIG. 12 shows a case in which an entry/exit determination image HG is superimposed on a captured image of a scene in which an article BP with a marker MK is stored in the refrigerator compartment 21 .

The entering/leaving determination image HG has a boundary line L1.
The boundary line L1 is a line that extends from the left end to the right end of the entering/leaving determination image HG in the figure, substantially in the center of the entering/leaving determination image HG. Boundary line L<b>1 is located in front of refrigerator 2 from the opening of refrigerator compartment 21 captured in the image captured by camera 42 for refrigerator compartment.

The image area of the entering/leaving determination image HG is divided into a first area HA1 and a second area HA2 by a boundary line L1. The first area HA1 corresponds to the front side of the refrigerator 2 from the opening of the refrigerator compartment 21 captured in the image captured by the refrigerator compartment camera 42 when the entering/leaving determination image HG is superimposed on the captured image. The second area HA2 corresponds to the rear side of the refrigerator 2 from the opening of the refrigerator compartment 21 captured in the image captured by the refrigerator compartment camera 42 when the entering/leaving determination image HG is superimposed on the captured image.

The entering/leaving determination unit 454 analyzes the captured image on which the entering/leaving determination image HG is superimposed to determine whether the marker MK detected in the detection area TA has moved to the first area HA1 or moved to the second area HA2. determine whether it did. When the entering/leaving determination unit 454 determines that the marker MK detected within the detection area TA has moved to the first area HA1, it determines that the detected article BP is the article BP discharged from the refrigerator compartment 21 . On the other hand, when the marker MK detected within the detection area TA moves to the second area HA2, the entering/leaving determination unit 454 determines that the detected article BP is the article BP that has entered the refrigerator compartment 21 .

When it is determined that the goods BP have been stored in the warehouse, the entering/leaving determination unit 454 determines the accommodation area in which the goods BP have been stored. The entry/exit determination unit 454 analyzes the captured image to identify the storage area of the refrigerator compartment 21 in which the article BP is stored. Then, the entering/leaving determination unit 454 determines the storage area in which the article BP is stored, based on the moving direction of the marker MK and the layout of a plurality of storage areas provided within the storage.
For example, the tips of the left door 22 or the right door 23 of the shelf boards 211, 213, 215, 217 of the refrigerator 2 and the door pockets are provided with marks for identifying storage areas. The entering/leaving determination unit 454 detects a mark that identifies the storage area from the captured image, and specifies the storage area based on the positional relationship between the detected mark and the marker MK attached to the article BP.

Also, the entering/leaving determination unit 454 may determine whether or not the article BP is the article BP re-stored in the refrigerator 2 . For example, when user P takes food out of the refrigerator 2 for use, uses part of the food taken out, and restocks the remaining food in the refrigerator 2, or stores the food in the storage container that is the article BP. A case is assumed in which the food is used, the food is newly replenished in the storage container, and the food is restocked in the refrigerator 2 .

In the above description, the object to be detected from the captured image is the article BP with the marker MK attached. can do.
The warehousing/delivery determination unit 454 detects foodstuffs and foodstuffs based on the color and shape of the image captured in the captured image, and the detected foodstuffs and foodstuffs are stored in the same manner as in the case of the article BP to which the marker MK is attached. It is determined whether it has been delivered or taken out.

The acquisition unit 455 analyzes the marker MK detected by the entering/leaving determination unit 454 and acquires the marker ID. The marker ID is identification information of the marker MK given to the marker MK. For example, when the marker MK is a two-dimensional code such as a QR code (registered trademark), the obtaining unit 455 obtains the marker ID included in the QR code. Acquisition unit 455 temporarily stores the acquired marker ID in fourth memory 440 .

The data generation unit 456 generates upload data. Upload data is data to be uploaded to the article management server 8 . The data generation unit 456 transmits the generated upload data to the article management server 8 .

The uploaded data includes the image of the article, the storage area information, the imaging device ID, the information indicating whether the article was put in or taken out, the date and time of entry or departure, information on the door or drawer that was opened or pulled out, etc. is included. The information when the goods are stored is called warehousing information, and the information when the goods are taken out is called warehousing information. The article image is an image obtained by cutting out the range in which the article is imaged from the captured image.

Also, if the article is an article with a marker MK attached, the upload data includes an article image, accommodation area information, imaging device ID, date and time of entry or exit, entry information or exit information, opened or withdrawn A marker ID is included in addition to the door or drawer information.

In addition, when at least one of the left door 22 and the right door 23 is opened, or when any one of the drawers 24A, 25A, 26A, and 27A is pulled out, but no goods are stored or retrieved, the upload data , which door or drawer was opened or pulled out, and date and time information.

[6. Operation of Imaging Device]
FIG. 13 is a flowchart showing the operation of the imaging device 3. FIG.
The operation of the imaging device 3 will be described with reference to the flowchart shown in FIG.
Third control unit 430 determines whether or not there is a person around refrigerator 2 based on sensor data from human sensor 34 (step SA1).

When the third control unit 430 determines that there is a person around the refrigerator 2 (step SA1/YES), it causes the refrigerator compartment camera 42 and the drawer camera 43 to start imaging (step SA2).

Next, the third control unit 430 determines whether at least one of the left door 22 and the right door 23 is open, or whether any one of the drawers 24A, 25A, 26A, 27A, and 28A is open. is determined (step SA3).

If the left door 22 and the right door 23 are not open and none of the drawers 24A, 25A, 26A, 27A, and 28A are open (step SA3/NO), the third control unit 430 process.

If at least one of the left door 22 and the right door 23 is open, or if any one of the drawers 24A, 25A, 26A, 27A, and 28A is open (step SA3/YES), the third control unit 430 , store the captured images captured by the refrigerator camera 42 and the drawer camera 43 in the fourth memory 440, and start recording the captured images (step SA4).

Next, the third control unit 430 reads the captured image from the fourth memory 440, and determines whether or not the marker and the article are detected within the detection area TA based on the read captured image (step SA5).

When the third control unit 430 determines that the marker and the article have been detected within the detection area TA (step SA5/YES), it reads the marker ID from the detected marker (step SA7). After that, the third control unit 430 cuts out an article image, which is an image in which the article is imaged, from the captured image (step SA8).

Further, when the marker is not detected in the detection area TA and only the article is detected (step SA6/YES), the third control unit 430 cuts out an article image, which is an image of the article, in the captured image ( Step SA8).

Next, the third control unit 430 performs entering/leaving determination based on the captured image recorded in the fourth memory 440 (step SA9). When the third control unit 430 determines that the article has been stored, the article image of the article that has been put in, the accommodation area where the article has been put in, and the time at which the article has been put in are stored in the fourth memory 440 in association with each other. Let Further, when determining that the article has been delivered, the third control unit 430 causes the fourth memory 440 to store the article image of the delivered article and the delivery time in association with each other.
In addition, the third control unit 430 controls when at least one of the left door 22 and the right door 23 is opened, or when any one of the drawers 24A, 25A, 26A, and 27A is pulled out, but there is no goods entering or leaving the warehouse. If so, the fourth memory 440 stores the information of the door or drawer that was opened or pulled out and the date and time information.

Next, the third control unit 430 determines whether or not the open door and drawer are closed (step SA11). When the third control unit 430 determines that the door and the drawer are not closed (step SA11/NO), the process proceeds to step SA5.

On the other hand, when the third control unit 430 determines that the open door and the drawer are closed (step SA11/YES), the recording of the captured image in the fourth memory 440 is finished (step SA12).

Next, the third control unit 430 determines whether or not there are people around the refrigerator 2 (step SA13). When third control unit 430 determines that a person exists around refrigerator 2 (step SA13/YES), the process proceeds to step SA3.

On the other hand, when the third control unit 430 determines that there is no person around the refrigerator 2 (step SA13/NO), it causes the refrigerator camera 42 and the drawer camera 43 to end imaging (step SA14).

After performing the process of step SA14, the third control unit 430 shifts the process to step SA1 and performs the process from step SA1 onwards again.

[7. Operation of imaging device and operation of article management server]
FIG. 14 is a flow chart showing the operation of the article management system 1. As shown in FIG.
In FIG. 14, a flowchart FB shows the operation of the imaging device 3, and a flowchart FC shows the operation of the goods management server 8. As shown in FIG.

As shown in the flowchart FB, the third control unit 430 determines whether or not a trigger for transmitting the upload data to the article management server 8 has occurred (step SB1). As a trigger for transmitting the upload data, it is determined that an article has entered or left the warehouse, and the determination result is stored in the fourth memory 440 . In addition, the trigger may be the end of the recording of the image captured by the refrigerator camera 42, the current time reaching a predetermined time, or the like.

When a trigger occurs (step SB1/YES), the third control unit 430 acquires upload data from the fourth memory 440 (step SB2).
The upload data includes an article image, storage area information, imaging device ID, warehousing information or warehousing information, warehousing date and time or warehousing date and time, information on opened or pulled out doors or drawers, and the like. Also, if the article is an article with a marker MK attached, the marker ID is included in addition to the above upload data.
In addition, when at least one of the left door 22 and the right door 23 is opened, or when any one of the drawers 24A, 25A, 26A, and 27A is pulled out, but no goods are stored or retrieved, the upload data , which door or drawer was opened or pulled out, and date and time information.

The third control unit 430 transmits the generated upload data to the article management server 8 (step SB3). After that, the third control unit 430 returns to the determination of step SB1.

Next, the operation of the article management server 8 will be explained with reference to the flowchart FC. The second control unit 83 of the article management server 8 determines whether or not the upload data has been received (step SC1).

When the second control unit 83 determines that the upload data has not been received (step SC1/NO), it makes the determination of step SC1 again.

On the other hand, when the second control unit 83 determines that the upload data has been received (step SC1/YES), the second control unit 83 stores the received upload data in the second memory 85 and is updated (step SC2). In addition, the second control unit 83 may update any of the ingredients management DB 93 , the article management DB 95 and the open state management DB 97 .

The second control unit 83 acquires account information 911 associated with the imaging device ID included in the received upload data from the identification information DB 91 .
The second control unit 83 updates any one of the ingredients management DB 93, the goods management DB 95, and the open state management DB 97 corresponding to the acquired account information 911. FIG. In addition, the second control unit 83 may update any of the ingredients management DB 93 , the article management DB 95 and the open state management DB 97 .

The second control unit 83 updates the article management DB 95 when the received upload data contains a marker ID. The second control unit 83 registers the marker ID, article image, warehousing date/time or warehousing date/time, and accommodation area included in the upload data in the article management DB 95 .

Also, if the received upload data does not contain a marker ID, the second control unit 83 identifies the item from the item image and acquires the color information and nutrient information of the identified item. The second control unit 83 registers the food name of the specified food in the food material name information 931 of the food material management DB 93 . The second control unit 83 also registers the acquired color information and nutrient information of the article in the color information 932 and nutrient information 933 of the food material management DB 93 . Further, the second control unit 83 registers the product image included in the upload data in the food image 935, and registers the date and time of arrival or departure and the accommodation area in the date and time of arrival 934 or departure date and time 937 and the accommodation area 936, respectively. .

Also, the second control unit 83 updates the open state management DB 97 based on the information about the opened or pulled out door or drawer included in the upload data.

[8. Operation of article management server]
FIG. 15 is a flow chart showing the operation of the article management server 8. As shown in FIG.
The second control unit 83 of the article management server 8 determines whether or not there is a data acquisition request with account information from the information processing device 6 (step SD1). If there is no data acquisition request (step SD1/NO), the second control unit 83 waits to start processing until a data acquisition request is made.

When the second control unit 83 receives the data acquisition request (step SD1/YES), it determines whether the data registered in the ingredients management DB 93, the goods management DB 95, and the open state management DB 97 corresponding to the received account information have been updated. (Step SD2).

If the data registered in the ingredients management DB 93, the article management DB 95, and the open state management DB 97 are not updated (step SD2/NO), the second control unit 83 notifies the information processing device 6 that there is no update data ( Step SD4). If there is no updated data, the previous data may be transmitted.

Further, when there is an update in the data registered in the ingredients management DB 93, the goods management DB 95, and the open state management DB 97 (step SD2/YES), the second control unit 83 updates the updated data to the ingredients management DB 93, the goods management DB 95. Alternatively, it acquires from the open state management DB 97 and transmits the acquired data to the information processing device 6 as update data (step SD3).

[9. Configuration of application screen 700]
16 and 17 are diagrams showing examples of initial screens 710A and 710B.
Here, the application screen 700 displayed on the touch panel 63 of the information processing device 6 will be described.
16 and 17 show initial screens 710A and 710B displayed immediately after article management application 675 is activated. Initial screens 710A and 710B are included in application screen 700 .

An initial screen 710A shown in FIG. 16 shows an example of displaying a picked-up image of the inside of the refrigerator 2 . An initial screen 710B shown in FIG. 17 shows an example in which an image of an empty refrigerator 2 is displayed with an article image acquired from the article management server 8. FIG. The initial screen 710A and the initial screen 710B can be switched by operating the menu button 711, for example. Hereinafter, the initial screens 710A and 710B are collectively referred to as an initial screen 710. FIG.

When only the left door 22 is opened, the initial screen 710A may display a captured image of an area corresponding to the left door 22 between the left door 22 and the refrigerator compartment 21 . In addition, on the initial screen 710</b>B, the image of the empty refrigerator 2 may be displayed with the article image updated in the area corresponding to the left door 22 among the left door 22 and the refrigerator compartment 21 .
Similarly, when only the right door 23 is opened, the initial screen 710A may display a captured image of a region corresponding to the right door 23 between the right door 23 and the refrigerator compartment 21 . Also, the image of the article updated in the area corresponding to the right door 23 of the right door 23 and the refrigerator compartment 21 may be displayed in the image of the empty refrigerator 2 on the initial screen 710B.

A menu button 711, a refrigerator compartment button 712, a vegetable compartment button 713, a freezer compartment button 714, a recipe search button 715, and the like are displayed on the initial screen 710.

By pressing the menu button 711, a screen to be displayed as the application screen 700 can be selected. The application screen 700 that can be selected by the menu button 711 includes a long-term storage notification screen 730 , a loading/unloading status screen 750 , an opening/closing status screen 770 , and a stored item screen 820 .

A refrigerator compartment button 712, a vegetable compartment button 713, and a freezer compartment button 714 are buttons for switching display targets. When the refrigerator compartment button 712 is selected on the initial screen 710 , the display target is changed to the refrigerator compartment 21 , and when the vegetable compartment button 713 is selected, the display target is changed to the vegetable compartment 27 . Also, when the freezer compartment button 714 is pressed, the display target is changed to the freezer compartment 26 .

A recipe search button 715 is a button for searching for recipes.
By pressing a recipe search button 715, a list of ingredients stored in the refrigerator 2 is displayed. By selecting ingredients that the user wants to use from the listed ingredients, it is possible to search for dishes using the selected ingredients.

FIG. 18 is a diagram showing a long-term storage notification screen 730. As shown in FIG. A long-term storage notification screen 730 is also included in the application screen 700 .
The long-term storage notification screen 730 is a screen that displays information about an article that has not been taken out of the accommodation area even after a preset period of time has passed since it was put into the refrigerator 2 . The long-term storage notification screen 730 shows an example of displaying storage areas as information on articles that will not be delivered. A refrigerator compartment button 731 , a vegetable compartment button 733 , a freezer compartment button 735 and an area display area 740 are displayed on the long-term storage notification screen 730 . A refrigerator compartment button 731, a vegetable compartment button 733, and a freezer compartment button 735 are buttons for switching display objects.

The storage area of the refrigerator 2 is displayed in the area display area 740 . By pressing the refrigerator compartment button 731, the vegetable compartment button 733, or the freezer compartment button 735, the storage area displayed in the area display area 740 can be changed.
In the area display area 740, the accommodation area of the refrigerator 2 where 30 days or more have elapsed since the refrigerator 2 was accommodated is displayed. FIG. 18 shows an example in which 30 days are set as the preset period. The preset period can be arbitrarily changed by a user's operation.
An area display area 740 shown in FIG. 18 shows an example in which the refrigerator compartment 21 is divided into 18 accommodation areas and displayed. The number of divisions of the refrigerator compartment 21 is arbitrary.
The inside of the refrigerator compartment 21 is divided into a left area and a right area in the X-axis direction. Further, the left area is divided into five in the Z-axis direction, and the five divided areas are the first refrigerating compartment 21A, the second refrigerating compartment 21B, the third refrigerating compartment 21C, the fourth refrigerating compartment 21D, and the chilled compartment 28, respectively. handle. Similarly, the right area is divided into five in the Z-axis direction, and the five divided areas are the first refrigerating compartment 21A, the second refrigerating compartment 21B, the third refrigerating compartment 21C, the fourth refrigerating compartment 21D, and the chilled compartment 28, respectively. handle.

The area of the left door 22 is divided into four areas corresponding to the first left door pocket 221 , the second left door pocket 223 , the third left door pocket 225 and the fourth left door pocket 227 .
The area of right door 23 is divided into four areas corresponding to first right door pocket 231 , second right door pocket 233 , third right door pocket 235 and fourth right door pocket 237 .

As shown in FIG. 18, on the long-term storage notification screen 730, storage areas containing items that have been stored in the refrigerator 2 for 30 days or more are displayed. FIG. 18 shows an example of displaying the number of elapsed days, such as 30 days or more, but an image such as a preset figure, symbol, or mark may be displayed.

FIG. 19 is a diagram showing a loading/unloading status screen 750. As shown in FIG. A loading/unloading status screen 750 is also included in the application screen 700 .
The article management server 8 counts the number of articles stored in the refrigerator 2 and the number of articles taken out from the refrigerator 2 for each accommodation area during a preset period based on the article management DB 95 .
The loading/unloading status screen 750 displays information on storage areas in which the total number of stored items and shipped items is equal to or less than a preset value.

A refrigerator compartment button 751 , a vegetable compartment button 752 , a freezer compartment button 753 , period selection buttons 754 and 755 , unit selection buttons 756 and 757 , and a loading/unloading status screen 750 are displayed on the loading/unloading status screen 750 .

A refrigerator compartment button 751, a vegetable compartment button 752, and a freezer compartment button 753 are buttons for switching display objects, like the refrigerator compartment button 712, the vegetable compartment button 713, and the freezer compartment button 714.

Period selection buttons 754 and 755 are buttons for selecting the period for which the loading/unloading status 760 is to be displayed. By selecting the period selection button 754, one month is selected as the period, and by selecting the period selection button 755, one week is selected as the period.

Unit selection buttons 756 and 757 are buttons for selecting display units. A unit selection button 756 selects one day as a display unit, and a unit selection button 757 selects a unit obtained by dividing one day into three units of morning, noon, and evening as a display unit. The user can arbitrarily change the setting of the time zones that separate morning, noon, and evening. For example, 5:00 to 10:00 may be set in the morning, 11:00 to 16:00 may be set in the afternoon, and 16:00 to 24:00 may be set in the evening.

Similar to the long-term storage notification screen 730, the loading/unloading status 760 divides the refrigerating chamber 21 into 18 storage areas, and displays information on loading/unloading of articles in each of the divided storage areas.
For example, assume that the refrigerator button 751 is selected, the period selection button 754 is selected to set one month as the selection period, and the unit selection button 756 is selected to set one day as the display unit. In this case, the ratio of items taken in and out in the past month including the current day is displayed for each of the 18 accommodation areas.
Assuming that the sum of the number of times items have been stored in the refrigerator compartment 21, the left door 22 and the right door 23 and the number of times the item has been taken out from the refrigerator compartment 21, the left door 22 and the right door 23 in the past month is 100%, The ratio of the number of times of entering and exiting is displayed for each of the 18 accommodation areas. In FIG. 19, the ratio of all accommodation areas is displayed, but only accommodation areas whose ratio is equal to or less than a preset value may be displayed.

FIG. 20 is a diagram showing an open/close status screen 770. As shown in FIG. An open/close status screen 770 is also included in the application screen 700 .
On the open/close status screen 770, period selection buttons 771 and 772, unit selection buttons 773 and 774, and an open/close status 780 are displayed.

The period selection buttons 771 and 772 are buttons for selecting the period for which the open/close status 780 is to be displayed. By selecting the period selection button 771, one month is selected as the period, and by selecting the period selection button 772, one week is selected as the period.

Unit selection buttons 773 and 774 are buttons for selecting display units. A unit selection button 773 selects one day as a display unit, and a unit selection button 774 selects a unit obtained by dividing one day into three units of morning, noon, and evening as a display unit.

The open/close state 780 displays a count value of the number of times at least one of the left door 22 and the right door 23 has been opened.
For example, it is assumed that the period selection button 772 is selected, one week is set as the selection period, the unit selection button 774 is selected, and morning, noon, and night are set as the display unit. In this case, the number of opening/closing times of the refrigerator compartment 21, the vegetable compartment 27, and the freezer compartment 26 for one week is displayed for each morning, noon, and night.

The article management server 8 causes the information processing device 6 to perform a notification operation when opening or closing of the door or drawer cannot be detected even after a preset period of time has elapsed. For example, the application execution unit 695 may display the opening/closing status screen 770 in a display mode different from the normal state in which opening/closing of the door or drawer is detected. Also, the application execution unit 695 may change the display color of the open/close status screen 770, or display on the initial screen 710 that opening/closing of the door or drawer is not detected.

FIG. 21 is a diagram showing a selection screen 790, and FIGS. 22 and 23 are diagrams showing items screens 820A and 820B. The application screen 700 also includes a selection screen 790 and contained item screens 820A and 820B.
The selection screen 790 is a screen for selecting a storage area to be displayed on the stored goods screen 820B. On the selection screen 790, display selection buttons 791 and 792, an area selection display 800, and an enter button 810 are displayed.

The display selection buttons 791 and 792 are buttons for selecting the display mode of the article when displaying the image of the article stored in the refrigerator 2 . By selecting the display selection button 791, the articles are displayed in descending order of the number of days that have passed since the storage date. Also, by selecting the display selection button 792, it is possible to select the area in which the article is stored.

The area selection display 800 is a screen for selecting a storage area in which an image of a stored article is to be displayed when the display selection button 792 is selected. When the user selects one of the storage areas and presses the enter button 810, a stored item screen 820B shown in FIG. 23 is displayed.

FIG. 22 is a diagram showing a stored article screen 820A.
The stored article screen 820A is a screen in which all the articles stored in the refrigerator 2 are displayed in descending order of elapsed time from the date of storage.
A delete button 821, a new addition button 823, and an article display area 830 are displayed on the contained article screen 820A. The delete button 821 is a button for deleting the item displayed in the item display area. A new addition button 823 is a button for newly registering an article in the article display area 830 .

An article image 831 , warehousing date and time 833 , and storage area information 835 are displayed in the article display area 830 .
The article image 831 is a captured image of the article. The storage date and time 833 indicates the date and time when the item of the item image 831 was stored in the refrigerator 2 . The accommodation area information 835 indicates the accommodation area of the refrigerator 2 in which the article is accommodated. FIG. 22 shows an example of displaying an image indicating the position of the accommodation area in the refrigerator compartment 21 as the accommodation area information 835 .

FIG. 23 is a diagram showing a stored article screen 820B.
The stored article screen 820B is a screen that displays the articles stored in the storage area selected in the area selection display 800 in descending order of elapsed time from the storage date.
A delete button 821 and a new add button 823 are also displayed on the stored item screen 820B.

An article image 841 , a warehousing date and time 843 , and accommodation area information 845 are also displayed in the accommodation article column 840 .
The article image 841 is a captured image of the article. The storage date and time 843 indicates the date and time when the item of the item image 841 was stored in the refrigerator 2 . The storage area information 845 indicates the storage area of the refrigerator 2 in which the article is stored. In the case of the stored article screen 820B shown in FIG. 23, the same storage area is displayed as the storage area information 845. FIG.
Hereinafter, the stored item screens 820A and 820B are collectively referred to as a stored item screen 820. FIG.

[10. Operation of Information Processing Device]
FIG. 24 is a flow chart showing the operation of the application execution unit 695. As shown in FIG.
The operation of the application execution unit 695 will be described with reference to the flowchart shown in FIG.
When the article management application 675 is selected and starts operating, the application execution unit 695 communicates with the article management server 8 and acquires updated data (step SE1). Next, the application execution unit 695 causes the touch panel 63 to display an initial screen 710 shown in FIG. 16 or 17 (step SE2).

Next, the application execution unit 695 determines whether or not the long-term storage notification screen 730 has been selected (step SE3). When long-term storage notification screen 730 is selected (step SE3/YES), application execution unit 695 causes touch panel 63 to display long-term storage notification screen 730 shown in FIG. 18 (step SE4). Thereafter, application execution unit 695 returns to the determination of step SE3.

Also, when the long-term storage notification screen 730 is not selected (step SE3/NO), the application execution unit 695 determines whether or not the loading/unloading status screen 750 is selected (step SE5). If the loading/unloading status screen 750 is selected (step SE5/YES), the application execution unit 695 causes the touch panel 63 to display the loading/unloading status screen 750 shown in FIG. 19 (step SE6). Thereafter, application execution unit 695 returns to the determination of step SE3.

If the loading/unloading status screen 750 has not been selected (step SE5/NO), the application execution unit 695 determines whether or not the opening/closing status screen 770 has been selected (step SE7). When opening/closing status screen 770 is selected (step SE7/YES), application execution unit 695 causes touch panel 63 to display opening/closing status screen 770 shown in FIG. 20 (step SE8). Thereafter, application execution unit 695 returns to the determination of step SE3.

If the open/close status screen 770 has not been selected (step SE8/NO), the application execution unit 695 determines whether or not the stored goods screen 820 has been selected (step SE9). When the stored item screen 820 is selected (step SE9/YES), the application execution unit 695 displays the selection screen 790 shown in FIG. SE11).

If the display is not for each accommodation area (step SE11/NO), application execution unit 695 displays accommodation item screen 820A shown in FIG. are displayed in descending order (step SE12). Thereafter, application execution unit 695 returns to the determination of step SE3.

Also, if the display is for each accommodation area (step SE11/YES), the application execution unit 695 accepts selection of an accommodation area on the selection screen 790 (step SE13). Then, the application execution unit 695 displays the stored item screen 820B shown in FIG. 23, and displays the items stored in the selected storage area in descending order of elapsed time from the storage date (step SE14).

Further, if the judgment at step SE9 is not the selection of the contained item screen 820B (step SE9/NO), the application execution unit 695 executes the process corresponding to the received operation (step SE10), and returns to the judgment at step SE3. .

[11. effects, etc.]
As described above, the article management system 1 includes the imaging device 3 and the article management server 8 .
The imaging device 3 images the inside of the refrigerator compartment 21 of the refrigerator 2 to generate a captured image.
Based on the captured image captured by the imaging device 3, the item management server 8 stores items to be stored in the refrigerating chamber 21, storage areas in the refrigerating chamber 21 in which the items are stored, items to be delivered from the storage area, to detect
Based on the detection result, the article management server 8 causes the second memory 85 to store the date of entry of the article into the refrigerator compartment 21 and the accommodation area in which the article was accommodated.
The article management server 8 causes the information processing device 6 to display information on articles that have not been taken out of the storage area even after a preset period has passed from the date of storage.

According to this, the information of the article that has not been taken out of the accommodation area even after the lapse of the preset period from the date of storage is displayed on the information processing device 6. Therefore, it is possible to prevent forgetting to use the article. loss can be reduced.

The article management server 8 causes the information processing device 6 to display the storage area information for articles that have not been taken out of the storage area even after a preset period has passed since the date of storage.

According to this, it is possible to prevent forgetting to use the item and reduce the loss of the item. In addition, it is possible to make the user recognize an accommodation area that is not used very often.

The article management server 8 counts the number of articles stored in the refrigerator 2 and the number of articles taken out from the refrigerator 2 for each storage area during a preset period.
The article management server 8 causes the information processing device 6 to display information on storage areas in which the total number of articles that have been stored and the number of articles that have been delivered is equal to or less than a preset value.

According to this, it is possible to make the user aware of the accommodation area that is not used much.

The article management server 8 is an information processing device that stores the storage date of the article stored in the refrigerator 2 and the image showing the storage area in which the article is stored in the order of the elapsed time since the article was stored in the refrigerator 2. to display.

According to this, it is possible to make the user aware of the storage date of the item, the storage area, and the elapsed time, and reduce the loss of the item.

The imaging device 3, the article management server 8, and the refrigerator 2 have a left door ranging sensor 31, a right door ranging sensor 32, and a drawer ranging sensor 33 that detect the open/closed state of at least one of the door and the drawer. Prepare.
If the left door range sensor 31, the right door range sensor 32, and the drawer range sensor 33 cannot detect the opening/closing of the door or the drawer, the article management server 8 , causes the information processing device 6 to perform the notification operation.

As a result, when the opening and closing of the door and the drawer of the refrigerator 2 are not detected, the information processing device 6 is caused to perform the notification operation, thereby preventing forgetting to use the article.

(Other embodiments)
As described above, the first embodiment has been described as an example disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to embodiments with modifications, replacements, additions, omissions, and the like. Also, it is possible to combine the constituent elements described in the first embodiment to form a new embodiment.
Therefore, other embodiments will be exemplified below.

In the embodiment described above, the refrigerator 2 was exemplified as the storage of the present disclosure. However, the storage of the present disclosure is not limited to the refrigerator 2, but may be other storage such as a freezer or a room temperature storage.

In the embodiment described above, the refrigerator compartment 21 was exemplified as the storage compartment of the present disclosure. However, the storage compartments of the present disclosure are not limited to refrigeration compartment 21 , but may be ice compartment 24 , fresh freeze compartment 25 , freezer compartment 26 and vegetable compartment 27 . Also, when the storage of the present disclosure is a storage other than the refrigerator 2 , the storage of the present disclosure corresponds to the storage of the storage other than the refrigerator 2 .

A process of detecting an article from a captured image and determining a storage area will be described with reference to FIG. FIG. 25 is a diagram showing boundary lines between the fifth region R5, the sixth region R6, the seventh region R7, and the eighth region R8. In the following, the vegetable compartment 27 pulled out from the inside of the refrigerator will be described as an example.

The storage/delivery determining unit 454 analyzes the captured image to identify the storage area of the vegetable compartment 27 in which the article is stored. For example, it is determined that the storage area for the articles stored in the fifth region R5 is the fifth region R5. The sixth region R6, the seventh region R7, and the eighth region R8 are similarly determined. That is, the accommodation area is specified from a single image obtained by imaging the entire vegetable compartment 27 . This simplifies the system configuration of the article management system 1 .
Note that the processing unit 871 of the article management server 8 may specify the accommodation area. In this configuration, the captured image of the vegetable compartment 27 is transmitted from the imaging device 3 to the article management server 8, and based on the received captured image, the processing unit 871 identifies the storage area and stores it in the food material management DB or the article management DB. Register an area.
Note that the vegetable compartment 27 may be divided into arbitrary regions such as two regions.

In the above-described embodiment, the human presence determination unit 451 determines whether or not there is a person around the refrigerator 2 based on the detection value of the human sensor 34 . However, the human presence determining unit 451 may perform determination based on detection values of an illuminance sensor, an electrostatic sensor provided in the grip portion of the refrigerator 2, or the like, instead of the human sensor 34. FIG. In the case of this electrostatic sensor, the refrigerator 2 and the imaging device 3 can communicate, and the refrigerator 2 transmits the detection value of the electrostatic sensor to the imaging device 3 .

In another embodiment, the refrigerator camera 42 and the drawer camera 43 may be configured as one camera. In the case of this configuration, the angle of view of the camera may be changed according to the opening/closing state of the door of the refrigerator 2 . As a result, the imaging device 3 does not need to be equipped with a plurality of cameras, thus simplifying the system configuration.

In another embodiment, the information processing device 6 may operate as a control device. In this configuration, the information processing device 6 stores the data of the ingredients management DB 93, the article management DB 95, and the open state management DB 97. FIG. The information processing device 6 appropriately updates the data in the food ingredient management DB 93, the article management DB 95, and the open state management DB 97 in the same manner as the article management server 8 described above. As a result, the article management system 1 does not need to be provided with the article management server 8, and the system configuration is simplified.

In another embodiment, the types of chambers formed in the box body 20 of the refrigerator 2 include storage chambers other than the refrigerating chamber 21, the ice making chamber 24, the fresh freezing chamber 25, the freezing chamber 26, and the vegetable chamber 27. Alternatively, at least one of the ice making compartment 24, the fresh freezing compartment 25, the freezing compartment 26 and the vegetable compartment 27 may be omitted. In another embodiment, the number of doors provided at the front opening of refrigerator compartment 21 may be one.

Also, the first processor 69, the second processor 87 and the fourth processor 450 may be composed of a single processor, or may be composed of a plurality of processors. The first processor 69, second processor 87 and fourth processor 450 may be hardware programmed to implement the corresponding functional units. That is, the first processor 69, the second processor 87 and the fourth processor 450 are configured by, for example, ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The configurations of the imaging device 3, the information processing device 6, and the article management server 8 shown in FIGS. 1 and 5 are examples, and the specific implementation is not particularly limited. In other words, it is not always necessary to mount hardware corresponding to each part individually, and it is of course possible to adopt a configuration in which one processor executes a program to realize the function of each part. Further, part of the functions implemented by software in the above-described embodiments may be implemented by hardware, or part of the functions implemented by hardware may be implemented by software. In addition, the specific detailed configurations of other units of the imaging device 3, the information processing device 6, and the article management server 8 can be arbitrarily changed within the scope of the present disclosure.

The control program 851 executed by the second control unit 83 can also be realized by recording the control program 851 on a portable information recording medium. Information recording media include magnetic recording media such as hard disks, optical recording media such as CDs, semiconductor storage devices such as USB (Universal Serial Bus) memories and SSD (Solid State Drives), but other recording media It is also possible to use

The first processor 69, the second processor 87, the third processor 350 and the fourth processor 450 may be composed of a single processor, or may be composed of a plurality of processors. These processors may be hardware programmed to implement corresponding functional units. That is, these processors may be composed of, for example, ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The configuration of the information processing device 6 and the article management server 8 shown in FIG. 1 and the configuration of the imaging device 3 shown in FIG. 7 are examples, and the specific implementation is not particularly limited. In other words, it is not always necessary to mount hardware corresponding to each part individually, and it is also possible to adopt a configuration in which one processor executes a program to realize the function of each part. Further, part of the functions implemented by software in the above-described embodiments may be implemented by hardware, or part of the functions implemented by hardware may be implemented by software.

The operation step units shown in FIGS. 13, 14, 15, and 24 are divided according to the main processing contents in order to facilitate understanding of the operation. does not limit the operation. It may be divided into more steps depending on the processing contents. Also, one step unit may be divided to include more processes. Also, the order of the steps may be changed as appropriate within the scope of the present disclosure.

It should be noted that the above-described embodiments are intended to illustrate the technology of the present disclosure, and various modifications, replacements, additions, omissions, etc. can be made within the scope of the claims or equivalents thereof.

As described above, the refrigerator and storage according to the present invention can be used for applications that use photographed images showing the conditions inside the storage.

1 Article Management System 2 Refrigerator 3 Imaging Device 6 Information Processing Device 8 Article Management Server 10 Network 20 Box 21 Cold Room 22 Left Door 23 Right Door 30 Sensor Board 31 Left Door Distance Sensor 32 Right Door Distance Sensor 33 Drawer distance measuring sensor 34 motion sensor 35 sensor board 40 camera board 41 third communication module 42 refrigerator camera 43 drawer camera 44 light emitting device 61 first communication module 63 touch panel 65 first control section 67 first memory 69 first Processor 81 Second communication module 83 Second control unit 85 Second memory 87 Second processor 91 Identification information database 93 Ingredient management database 95 Goods management database 97 Open state management database 310 First I/F circuit 330 Signal processing unit 340 Third memory 345 control program 350 third processor 410 second I/F circuit 430 third control unit 440 fourth memory 441 control program 443 imaging device ID
450 Fourth processor 451 Person presence determination unit 452 Open/close determination unit 453 Imaging control unit 454 Delivery determination unit 455 Acquisition unit 456 Data generation unit 459 Fourth processor 675 Article management application 695 Application execution unit 700 Application screen

Claims (7)

1. capturing an image of the inside of a storage chamber of a storage to generate a captured image; based on the captured image, an article to be stored in the storage chamber, an accommodation area in the storage chamber in which the article is accommodated, and an item to be taken out of the accommodation area; an imaging device that detects an article that
   a control device for displaying on an information processing device, based on the detection result of the imaging device, information about an article that has not been taken out of the storage area even after a preset period of time has passed since it was put into the storage room;
   An article management system comprising:
2. 2. The control device according to claim 1, wherein said control device causes said information processing device to display information on a storage area of an article that has not been taken out of said storage area even after a preset period has passed since said article was put into said storage room. Goods management system.
3. The control device counts the number of articles entering the storage and the number of articles leaving the storage for each storage area during a preset period,
   3. The article management system according to claim 1, wherein said information processing device displays information of accommodation areas in which the total number of said articles received and said number of articles taken out is equal to or less than a preset value.
4. The control device stores the storage date of the article stored in the storage and the image showing the storage area in which the article is stored in order of the elapsed time since the article was stored in the storage. 4. The article management system according to any one of claims 1 to 3, which is displayed on a processing device.
5. The imaging device includes an open/close sensor that detects an open/closed state of at least one of a door and a drawer provided in the storage,
   5. The control device according to any one of claims 1 to 4, wherein the control device causes the information processing device to perform a notification operation when the opening/closing of the door or the drawer cannot be detected by the opening/closing sensor even after a preset period of time has elapsed. or the article management system according to item 1.
6. imaging the inside of the storage compartment of the storage;
   a step of detecting an article to be stored in the storage room, a storage area in the storage room in which the article is stored, and an article to be taken out from the storage area based on the captured image;
   a step of causing an information processing device to display information about an article that has not been taken out of the storage area even after a preset period of time has passed since it was put into the storage room;
   An article management method.
7. to the computer,
   A procedure for detecting an article to be stored in the storage chamber, an accommodation area in the storage chamber in which the article is accommodated, and an article to be taken out from the accommodation area, based on an image captured inside the storage chamber of the storage. and,
   a procedure for displaying on an information processing device information about an article that has not been taken out of the storage area even after a preset period has passed since it was put into the storage;
   The program that causes the to run.

Images