JP2018190120A - Article storage device and article management system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an article storage device and an article management system that are capable of reducing a cost, do not require a troublesome setting operation and are capable of performing inventory management of stored reserve articles.SOLUTION: An article storage device 1 has an image acquisition unit 11 acquiring an image inside a housing chamber, an image feature amount extraction unit 12 providing image feature amount information of an article and detection area information, a feature amount detection history storage unit 13 storing history data linking the image feature amount information with the detection area information, a reserve article feature determination unit 14 determining the image feature amount as an image feature amount of a reserve article when image feature amounts similar to each other appear in the same detection area in the image repeatedly from the history data, a reserve article feature management unit 16 storing image feature amounts of the reserve articles, and a reserve article detection unit 15 determining whether the reserve article is stored according to whether the latest extracted image feature amount is similar to the image feature amount of the reserve article.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an article storage device capable of managing inventory of standing articles and an article management system including the article storage device.

  Usually, in a storage (article storage device) such as a refrigerator and a storage shelf, a permanent article that is an article that the user always wants to store is stored. The standing articles (that is, the standing foods) in the refrigerator are, for example, milk, eggs, yogurt, natto and the like. However, there is a case where the supplementary food is not stored in the refrigerator forgetting to replenish. In this case, the user cannot send a habitual diet.

  As a countermeasure, Patent Document 1 detects the stock of food stored in the refrigerator, stores the stock information, detects whether there is a shortage of items corresponding to regular food, and lacks the appropriate items. In this case, a device for outputting a purchase instruction is proposed. In this apparatus, an example is shown in which food is specified by a process of performing a matching operation between the characteristics of the acquired food image (image data) and reference information (that is, a database).

  Further, Patent Document 2 acquires a storage amount of a specific food that is a food (food) stored in a specific position that is a preset position in the refrigerator, and the storage amount of the acquired specific food is preset. The refrigerator which determines whether it is more than the reference | standard storage amount currently determined and alert | reports the information regarding the specific foodstuff (insufficient foodstuff) determined to be less than the reference | standard storage amount is proposed.

JP2013-250698A (paragraphs 0021 and 0022) JP 2014-70740 A (summary)

  However, the reference information necessary for food that can be stored in the refrigerator is a huge amount of data, and the preparation of the database requires a huge cost. Further, in order to perform matching between acquired image features and a huge database, a high-performance arithmetic processing unit is required, which increases the cost of the apparatus.

  Moreover, in patent document 2, since the specific foodstuff stored in the specific position in the storage chamber set beforehand is the object of inventory management, the foodstuff and storage position which can be managed are limited. In order to change the setting of the specific food and the storage position, a setting operation is required, which is very inconvenient for the user.

  The present invention has been made in order to solve the above-described problems, and the object thereof is to reduce the cost, do not require a troublesome setting operation, and perform inventory management of stored articles stored therein. An object storage device and an object management system are provided.

  An article storage device according to an aspect of the present invention includes an image acquisition unit that acquires an image by photographing a storage room that stores an article, and extracts an image feature amount of the article from the image. An image feature amount extraction unit that provides image feature amount information to be displayed and detection area information indicating a detection area in which the image feature amount is detected; and the image feature amount information provided from the image feature amount extraction unit and the detection A feature amount detection history storage unit that stores image feature information obtained by linking area information as history data, and the history data stored in the feature amount detection history storage unit. When image feature amounts determined to be similar to each other using the article determination threshold appear repeatedly in the same detection area in the image, the image feature amounts are always stored in the storage room. A permanent article feature determination unit that determines an image feature quantity of the standing article, a permanent article feature management unit that stores an image feature quantity of the permanent article, and the image feature quantity from the image acquired by the image acquisition unit Depending on whether the latest image feature quantity extracted by the extraction unit is similar to the image feature quantity of the permanent article stored in the permanent article feature management unit, the permanent article is stored in the storage chamber. And a permanent article detection unit for determining whether or not it has been made.

  An article management system according to another aspect of the present invention includes the article storage apparatus and a mobile terminal, wherein the mobile terminal includes a terminal communication unit that communicates with the article storage apparatus, and the article storage apparatus. And a display operation unit for displaying the acquired inventory information.

  According to the present invention, there is an effect that even if a user does not perform a troublesome setting operation, it is possible to automatically find out the standing articles and manage the inventory of the standing articles.

  Further, according to the present invention, since a huge database preparation and matching process are not required, the cost can be reduced.

It is a block diagram which shows roughly the structure of the article | item management system containing the refrigerator and refrigerator which are the article | item storage apparatuses which concern on Embodiment 1 of this invention. (A) And (B) is a flowchart which shows the process which the refrigerator which concerns on Embodiment 1 performs. (A) to (C) is a diagram showing an example of the arrangement of articles in the refrigerator compartment of the refrigerator according to Embodiment 1 and an example of the detection positions of feature points of the articles. (A) to (E) are diagrams showing examples of the arrangement of articles and detection positions of feature points of articles in the refrigerator compartment of the refrigerator according to Embodiment 1, and an example of an image feature amount detection history. It is a flowchart which shows the process which the permanent goods feature management part in the refrigerator which concerns on Embodiment 1 performs. It is a state transition diagram which shows the state transition of the standing goods in the refrigerator which concerns on Embodiment 1. FIG. It is a block diagram which shows roughly the structure of the article | item management system containing the refrigerator and refrigerator which are the article | item storage apparatuses which concern on Embodiment 2 of this invention. (A) And (B) is a flowchart which shows the process which the refrigerator which concerns on Embodiment 2 performs. (A) to (E) are diagrams showing examples of the arrangement of articles and the detection positions of feature points of articles in the refrigerator compartment of the refrigerator according to Embodiment 2, and examples of image feature quantity detection history. It is a block diagram which shows roughly the structure of the article | item management system containing the refrigerator and refrigerator which are the article | item storage apparatuses which concern on Embodiment 3 of this invention. It is a figure which shows an example of the hardware constitutions of the article | item storage apparatus which concerns on the modification of Embodiment 1 to 3 of this invention.

  Hereinafter, an article storage device and an article management system according to an embodiment of the present invention will be described with reference to the accompanying drawings. The following embodiments are merely examples, and various modifications can be made within the scope of the present invention.

<< 1 >> Embodiment 1
<< 1-1 >> Configuration FIG. 1 is a block diagram schematically showing a configuration of an article management system including a refrigerator 1 and a refrigerator 1 according to Embodiment 1 of the present invention. The article management system according to Embodiment 1 includes a refrigerator 1 as an example of an article storage device and a mobile terminal 4. The refrigerator 1 and the portable terminal 4 can communicate with each other by wireless communication or the like. Communication between the refrigerator 1 and the portable terminal 4 may be communication via a communication network such as the Internet. The article storage device may be a storage that does not have a cooling function. The mobile terminal 4 is, for example, a smartphone, a mobile phone, a tablet information terminal, or a personal computer.

  The refrigerator 1 is acquired with an image acquisition unit 11 such as a camera that acquires an image (that is, image data) in the refrigeration room by photographing a refrigeration room that is a storage room (storage room) that stores food as an article. An image feature amount (generally, a local feature amount) of the article is extracted from the captured image, and image feature amount information indicating the image feature amount and detection area information indicating a region where the article from which the image feature amount is detected exist. The image feature quantity extraction unit 12 to be provided, and the image feature information obtained by associating the image feature quantity information provided from the image feature quantity extraction unit 12 with the detection area information are converted into feature quantity detection history (history data). And a feature amount detection history storage unit 13 that stores it as a database.

  The refrigerator 1 determines whether or not the image feature amounts extracted from the history data stored in the feature amount detection history storage unit 13 are similar to each other, and sets a predetermined threshold for determining the same article. When image feature quantities determined to be similar to each other appear repeatedly in the same detection area in the image acquired by the image acquisition unit 11, a predetermined period or a predetermined shooting In the number of times, a permanent article feature determination unit 14 that determines that an image feature amount that repeatedly appears at an appearance ratio that is equal to or higher than a predetermined reference ratio is an image feature quantity of a permanent article that is permanently installed in the refrigerator compartment, and an image of the permanent article And a permanent article feature management unit 16 for storing feature quantities.

  Furthermore, the refrigerator 1 has the latest image feature amount extracted by the image feature amount extraction unit 12 from the image acquired by the image acquisition unit 11 and the past image feature amount stored in the permanent article feature management unit 16. Based on whether or not the image feature quantity of the article is similar, the permanent article detection unit 15 that determines whether or not the permanent article is stored in the refrigerator compartment, and the result of the determination by the permanent article detection unit 15 A stock inventory management unit 17 that manages the inventory of the regular items in the refrigerated room, and a communication unit 18 that notifies the portable terminal 4 as an external device that there are no longer any regular items stored in the refrigerated room. Have. Note that the same article determination threshold is used to determine whether or not the latest image feature quantity is similar to the image feature quantity of the permanent article.

  Note that the refrigerator 1 may include a display unit or an audio output unit that can notify the user that there are no regular items.

  The portable terminal 4 includes a terminal communication unit 41 that communicates with the refrigerator 1, and a display operation unit 42 that displays the content notified from the refrigerator 1 and receives a user operation. The display operation unit 42 is, for example, a touch panel. However, the display operation unit 42 may include a display unit such as a liquid crystal panel and an operation input unit such as an operation button. A user who has the portable terminal 4 can know from the portable terminal 4 that there are no longer any permanent articles in the refrigerator compartment of the refrigerator 1.

<< 1-2 >> Operation FIGS. 2A and 2B are flowcharts illustrating processing performed by the refrigerator 1 which is the article storage device according to Embodiment 1. FIG. FIG. 2A shows the inventory management process, and FIG. 2B shows the update process of the features of the permanent items. The inventory management process in FIG. 2A is mainly executed by the image feature amount extraction unit 12, the permanent item detection unit 15, the permanent item inventory management unit 17, and the communication unit 18. 2B is mainly executed by the image feature amount extraction unit 12, the feature amount detection history storage unit 13, the permanent item feature determination unit 14, and the permanent item feature management unit 16. The

<FIG. 2A, Step S101>
First, the image acquisition unit 11 of the refrigerator 1 acquires an image (image data) by taking an image of the refrigeration room with the door of the refrigeration room being closed as a trigger (step S101). The image acquisition unit 11 is, for example, a wide-angle camera installed on the door of the refrigeration room in a direction in which the refrigeration room is photographed. The opening / closing of the door can be detected by the output of the door open / close detector of the refrigerator compartment. The open / close detector of the refrigerator compartment door may be provided for preventing the door from being forgotten to close. Since the food in and out of the refrigerator compartment is not taken in and out while the door of the refrigerator compartment is closed, it is desirable to acquire the latest image of the refrigerator compartment when the door is closed. At the time of shooting, it is desirable to turn on the interior lamp provided in the refrigerated room or the light provided in the image acquisition unit 11. In addition, since an image acquired by photographing with a wide-angle camera has a distortion, it is desirable to perform a distortion correction process. However, the timing of shooting is not limited to when the door of the refrigerator compartment is closed. For example, when the door is closed after the door has been opened for a certain period of time during which goods can be taken in or out, or the door is closed. You may make it image | photograph after the fixed time progress after being carried out.

<FIG. 2 (A), Step S102>
Subsequently, the image feature amount extraction unit 12 acquires an image feature amount (generally, a local feature amount) from the image in the refrigerator compartment (step S102). As shown in FIG. 2A, after acquiring the image feature amount, the image feature amount extraction unit 12 extracts the image feature amount (step S102), and the permanent item detection unit 15 detects the permanent item (step S121). The stock information update and determination by the stock inventory management unit 17 (steps S122 and S123) and the notification by the communication unit 18 (step S124) are executed. Also, as shown in FIG. 2B, the feature update process of the permanent article is performed in parallel with the extraction of the image feature amount (step S102) by the image feature amount extraction unit 12.

  The image feature amount is, for example, a pixel set converted into a vector for recognition or identification. The image feature value is a local feature point that is obtained from the surroundings of feature points that are particularly resistant to deformations such as image enlargement / reduction and image rotation (points that are not easily affected by enlargement / reduction, rotation, and deformation). It is a feature quantity. Typical local feature quantities include SIFT (Scale-Invariant Feature Transform) or HOG (Histograms of Oriented Gradients). SIFT is an image feature value that is resistant to rotation and scale changes. HOG is an image feature amount that is easy to extract, such as a contour. In the following description, an example using local feature amounts as image feature amounts will be described.

  In the first embodiment, in order to extract the image feature amount of the article, the wall surface in the refrigerator compartment and the region where the storage shelf exists are excluded from the image feature amount extraction target. Therefore, for example, by storing an empty image (background image) in the memory in advance as a base image (base image data) and calculating the difference between the base image and the newly captured image in the refrigerator room, It is possible to distinguish the background area (area where the difference is smaller than the reference value) and food (area where the difference is larger than the reference value).

  FIGS. 3A to 3C are diagrams illustrating an example of arrangement of articles (food) and an example of detection positions of feature points of articles in the refrigerator compartment 1a of the refrigerator 1 according to the first embodiment. In the refrigerator compartment 1a of the refrigerator 1, there are four shelves. The first row from the top contains articles A, milk, and other milk from the left. , Articles B, and C are stored, articles D are stored in the third row from the top, and articles E are stored in the fourth row from the top. In the first embodiment, the case where the two milks stored in the first row are of the same brand will be described. In the example of FIG. 3A, a case where the standing article is a certain brand of milk, which is often stored on the right side of the first stage of the refrigerator 1 will be described. In the first embodiment, the characteristic that the regular articles are often placed in the same place in the refrigerator compartment 1a of the refrigerator 1 is used. Although the place where the standing article is placed in the refrigerator compartment 1a is different for each user (for example, home), the refrigerator 1 according to the first embodiment learns the storage place for the standing article for each user, and stores the standing article. To detect.

  The feature points of the image in FIG. 3A are detected as, for example, a region surrounded by a circle in FIG. What kind of feature point is detected differs depending on the detection method. For example, a point having a feature in the luminance distribution, color distribution, edge distribution, or the like is used as the feature point. By calculating the image feature amount from pixels around the feature point, the characteristic part of the region where the feature point is detected can be parameterized.

<FIG. 2B, Steps S111 to S113>
The processing in steps S111 to S113 in FIG. 2B is performed in parallel with, for example, image feature amount extraction (step S102) by the image feature amount extraction unit 12. First, the feature amount detection history storage unit 13 includes, for example, an image feature amount (local feature amount in the first embodiment), a feature point detection date and time, a feature point detection area in the refrigerator compartment 1a, and a feature. Image feature information obtained by associating images around the points is stored as history data (step S111). Here, the image around the feature point is an image having a size that allows the user to fully recognize what the article is. For example, an image around a feature point may be within a circle of a predetermined radius centered on the feature point.

  For example, when a region obtained by dividing each of the first to fourth storage spaces from the top in the refrigerator compartment 1a into a right half and a left half is used as a detection area, a feature point detection area and an image feature amount The image feature information obtained by associating can be recorded as history data of a permanent article in a format as shown in FIG. In addition, since the image acquisition unit 11 is a camera provided in the refrigerator 1 and its shooting range is constant, information such as which part in the image is a shelf and which part is a detection area is: You may hold | maintain in the feature-value detection log | history memory | storage part 13 previously.

  As shown in FIG. 3C, the image feature amount is expressed by a multi-dimensional vector. In this example, “milk feature 1” and “milk feature 2” indicating the features of two identical brands of milk placed on the right side of the first row (ie, the upper right of the refrigerator compartment 1a) are similar vectors. And have similar characteristics. In addition, the similarity between vectors can be shown by the magnitude | size of the Euclidean distance which shows the linear distance between vectors, for example. “The features are similar” can be determined, for example, by the fact that the Euclidean distance between the two feature vectors to be compared is equal to or less than a threshold value (threshold for determining the same article).

  The detection area that is an item in the left column of FIG. 3C is an example, and a detection area that is more finely divided (eg, divided into three or more) may be used, or the detection areas may be partially separated from each other. They may overlap. Moreover, you may use the detection area divided | segmented more widely (for example, for every step).

  4A to 4E are diagrams showing examples of the arrangement of articles and the detection positions of article feature points in the refrigerator compartment 1a of the refrigerator 1 according to Embodiment 1, and examples of image feature quantity detection histories. It is. The image feature amount is stored in the feature amount detection history storage unit 13 for a certain period. For example, when the arrangement of the article and the detection position of the feature point change in the order of FIGS. 4A, 4B, 4C, and 4D, as shown in FIG. History data indicating in which detection area the feature amount is detected is recorded in the feature amount detection history storage unit 13. The history data shown in FIG. 4E is also referred to as an image feature amount detection history. The milks shown in FIGS. 4A to 4D are all milk of the same brand, and the image feature values “milk features 1 to 5” extracted from the milk are all similar image feature values. The case where it is is demonstrated.

  Subsequently, the permanent item feature determination unit 14 determines which image feature amount is the image feature amount of the permanent item from the image feature amount detection history accumulated in the feature amount detection history storage unit 13 (step S112). . The permanent item feature determination unit 14 determines whether or not the image feature amount of the permanent item is based on whether or not a similar feature appears at a certain ratio or more during a predetermined period.

  In the example of FIG. 4E, the permanent item feature determination unit 14 determines that “milk features 1 to 5” are similar because they are image feature amounts at the same position of the same item. Similar features appear at a certain threshold (for example, the threshold is 75%) or more in a certain period (for example, a period including the past four detections). That is, since “milk characteristics” appear in FIGS. 4A, 4B, and 4D, and “milk characteristics” do not appear in FIG. 4C, the appearance ratio is 3/4 = 75%. It is. For this reason, the representative vector of “milk characteristics 1 to 5” is determined as the image feature amount of the permanent article. The representative vector means an average vector of image feature amounts.

  In the example of FIGS. 4A to 4E, the permanent article feature determination unit 14 also performs “feature of the article B”, “feature of the article H”, and “feature of the article I” a plurality of times within the same detection area. Although appearing, the appearance ratio of a predetermined period (for example, the past four detections) is 50%, and is less than a certain ratio (for example, the threshold is 75% or more). It is not regarded as a feature value.

  Note that “4 times detection” that is a predetermined period and “75% or more” that is a threshold of the appearance ratio are examples of settings, and a longer period may be adopted, and 75% as a threshold. Other ratios may be used. In addition, the appearance ratio may be set using the actual number of days and time instead of setting a predetermined period based on the number of shots. For example, the permanent article feature determination unit 14 can also determine that the characteristic is the characteristic of the permanent article when a similar feature appears in the same detection area for 20 days or more in the past month.

  Subsequently, the permanent item feature management unit 16 updates the database storing the features of the permanent item (step S113). In the database, in addition to the image feature amount of the permanent article, the date and time of the latest detection of the permanent article, the position, the image around the feature point, etc. (hereinafter referred to as “the latest detection information of the permanent article”) are recorded. .

  FIG. 5 is a flowchart illustrating processing performed by the permanent article feature management unit 16 in the refrigerator 1 according to the first embodiment.

  The permanent article feature management unit 16 has the characteristics of the permanent article (new permanent article) determined by the permanent article feature determination unit 14 in step S112 and the characteristics of the permanent article (existing permanent article) determined as a permanent article in the past. Are compared (step S202).

  When the feature of the new permanent article is similar to the feature of the existing permanent article that is already stored, the permanent article feature management unit 16 regards the new permanent article and the existing permanent article as the same article, The image feature quantity of the existing permanent article in the permanent article database is updated to the average value of the image feature quantity of the new permanent article and the existing permanent article (step S203). In this way, even if the way of placing the article in the refrigerator compartment 1a of the refrigerator 1 by the user gradually changes, the permanent article feature management unit 16 follows the change, and the characteristics of the new permanent article Similarity determination with the characteristics of the existing permanent articles can be appropriately performed. The permanent item feature management unit 16 also updates the latest detection information of the permanent item.

  If there is no existing permanent article having a feature similar to that of the new permanent article, the permanent article feature management unit 16 newly registers the characteristic of the new permanent article and the latest detection information (step S204).

  When there are a plurality of new permanent articles, the permanent article feature management unit 16 performs steps S202 to S204 for each of the plurality of new permanent articles.

  With the above processing, the permanent item feature management unit 16 does not specify what the name of the item represented by the image feature amount is, and in the image acquired by the image acquisition unit 11, any of the items in the refrigerator compartment 1 a The image feature amount of the permanent article can be determined from the image feature amount of the article that repeatedly appears at the place.

<FIG. 2A, Step S121>
Next, a process for managing a permanent article will be described. The permanent item detection unit 15 extracts an image feature amount (for example, a local feature amount) from the image in the refrigerator compartment 1a (step S102), and then extracts the image feature amount and the permanent article feature management from the image in the refrigerator compartment 1a. The image feature amount of the permanent article stored in the unit 16 is compared, and if there is a feature similar to the characteristic of the permanent article, it is determined that the permanent article has been detected (step S121).

  In step S121, the permanent article detection unit 15 compares the image feature amount extracted from the image in the refrigerator compartment 1a with the image feature amount of the permanent article in all the detection areas without considering the detection area. May be. If it does in this way, even if it is a case where a regular article is moved to a different detection area in the refrigerator compartment 1a of the refrigerator 1, it can prevent misdetecting that a regular article is lost.

  However, in the first embodiment, the case where the standing articles are placed in the same detection area is described. For this reason, as for the detection area where the permanent articles are frequently placed, the threshold used for the similarity determination is set to a large value so that the similar range of the characteristics of the permanent articles is widened, and the area farther from the detection area. The threshold value used for the similarity determination may be set to a small value so that the similar range of the features of the permanent article becomes narrow.

  With the above processing, the permanent article detection unit 15 can determine whether or not the permanent article is in the refrigerator compartment 1a without specifying what the name of the article represented by the image feature amount is.

<FIG. 2A, Step S122>
Subsequently, the permanent article inventory management unit 17 updates the inventory information of the permanent articles based on the information of the permanent articles in the refrigerator compartment 1a extracted by the permanent article detection unit 15 (step S122).

  The permanent inventory management unit 17 manages the status of the permanent articles in the refrigerator compartment 1a of the refrigerator 1 and the latest detection information. FIG. 6 is a state transition diagram showing transition of the state of the standing articles in the refrigerator 1 according to the first embodiment. The state of the permanent article detected by the permanent article detection unit 15 is “present (state 1)”. Up to now, the status of “present” but not detected this time is set to “pending (status 2)”. A standing article that has not been detected for a certain period of time in the “pending” state has a state of “none (state 3)”. Waiting in the “pending” state for a certain period of time extracts image feature amounts hidden behind the stationary articles and other articles temporarily out of the refrigerator compartment 1a of the refrigerator 1 for temporary use. This is in order to prevent erroneous detection that the stocks that have disappeared are out of stock.

<FIG. 2A, Steps S123 and S124>
In the case where the standing article in the standing article inventory management unit 17 is in the “pending” state until now, the state becomes “none” (YES in step S123), it is determined that the standing article has disappeared, and the communication unit 18 This is notified to the portable terminal 4 (step S124). The notification includes the latest detection information of permanent articles and the image in the refrigerator compartment 1a of the refrigerator 1. The communication path for performing the notification may be, for example, a communication path by wireless communication using a wireless standard such as Wi-Fi (registered trademark) or Bluetooth (registered trademark), or a communication path using a public network. It may be.

  When receiving the notification from the communication unit 18, the terminal communication unit 41 of the portable terminal 4 provides the content to the display operation unit 42, and the display operation unit 42 displays the content of the notification. The detection area detected most recently included in the latest detection information of the permanent article is illustrated in the image in the refrigerator compartment 1a, or the image near the feature point (that is, the permanent article image) detected most recently is displayed to the user. By presenting the user, the user can grasp which permanent article has been lost even if the name of the permanent article is not presented. That is, the user can know the fact that there are no permanent items in the refrigerator compartment 1a.

<< 1-3 >> Effect As described above, in the refrigerator 1 and the article management system according to the first embodiment, the image acquired by the image acquisition unit 11 is repeated at any location in the refrigerator compartment 1a. It is possible to determine the image feature amount of the standing article from the image feature amount of the appearing article, and automatically find and manage the standing article. That is, the refrigerator 1 and the article management system do not specify the name of the article in the refrigerator compartment 1a, but find out (determine) which article among the articles is a permanent article and inventory the permanent articles. Can be managed.

  Therefore, according to the refrigerator 1 and the article management system according to the first embodiment, since it is possible to find a permanent article without preparing feature data of many articles as a database in advance, it is necessary for image analysis using the database. A high-performance processor and a memory with a large storage capacity are unnecessary, and the cost can be reduced. In addition, the refrigerator 1 and the article management system can manage inventory of permanent articles without using high-cost parts such as RFID (Radio Frequency IDentification) tags.

  In addition, according to the refrigerator 1 and the article management system according to the first embodiment, the permanent articles are automatically detected using the characteristic that the permanent articles are often placed in the same place in the refrigerator compartment 1a. be able to. For this reason, there is no restriction | limiting in the place where a permanent article is preserve | saved, and the user does not need to preset the storage place of a permanent article.

  Furthermore, when the comparison target for determining the characteristics of the permanent items is limited to the items in the same detection area, the detection accuracy is improved and the calculation cost is higher than when all the items in the image are set as comparison targets. Can be reduced.

  In addition, when a standing article is determined using an image feature amount, it is a different article. However, if a similar image feature amount is detected by chance, an erroneous determination may be made. However, in the first embodiment, in the image acquired by the image acquisition unit 11, the condition that the image repeatedly appears in the same detection area is set, so that it is possible to more accurately determine whether the item is a regular article. it can.

  Further, according to the refrigerator 1 and the article management system according to the first embodiment, when a permanent article cannot be detected in the refrigerator compartment 1a of the refrigerator 1, a message to that effect is transmitted to the user's portable terminal 4. Even when the user is out of the office, the user can know that there are no items in the refrigerator compartment 1 a of the refrigerator 1.

<< 1-4 >> Modified Example In the first embodiment, the case where the image acquisition unit 11 is a wide-angle camera that acquires an image in the refrigerator compartment 1a of the refrigerator 1 has been described, but a different method may be used. For example, the image acquisition unit 11 may include a plurality of cameras and an image composition device installed side by side, and may generate an image of the refrigerator compartment 1a by combining images acquired by the plurality of cameras. In this way, although the manufacturing cost increases due to an increase in the number of cameras, a high-quality image with little distortion can be acquired.

  Further, in the refrigerator 1 and the article management system according to the first embodiment, the portable terminal 4 is notified that there are no permanent articles, but the user may be notified in other forms. For example, notification may be made by turning on or blinking an LED lamp provided in the refrigerator 1, or by detecting the approach of the user with a proximity sensor and notifying by voice when the user approaches.

  Further, in the refrigerator 1 and the article management system according to the first embodiment, the notification is transmitted from the communication unit 18 when it is determined that the permanent articles are lost, but is repeated for a certain period until the target permanent articles are detected again. You may be notified. In this way, it is possible to prevent the user from forgetting that the user has received a notice that there are no regular items.

  Further, the user who has received the notification may be able to select whether to receive the notification repeatedly by using the portable terminal 4. In this way, it is possible to prevent the user from repeatedly receiving notifications that are determined to be unnecessary.

  Further, in the refrigerator 1 and the article management system according to the first embodiment, the notification is sent from the communication unit 18 when it is determined that there is no regular article, but the inventory information of the permanent article is transmitted at regular intervals. You may do it. In this way, the user can regularly grasp the inventory of the regular items in the refrigerator compartment 1a of the refrigerator 1, and therefore can regularly replenish the regular items.

  Further, in the refrigerator 1 and the article management system according to the first embodiment, notification is performed from the communication unit 18 of the refrigerator 1 to the mobile terminal 4. A similar effect can be obtained by inquiring the stock status. For example, in the portable terminal 4, the display operation unit 42 accepts an input operation of a regular article to be inquired about by the user, and the terminal communication unit 41 transmits the inputted inquiry information about the inventory of the regular article to the refrigerator 1. In the refrigerator 1, the permanent goods inventory management unit 17 transmits an answer to the inquiry about inventory from the communication unit 18. Alternatively, the display operation unit 42 may store a stored article image received in the past, and inquire the refrigerator 1 of the presence or absence of the stored article selected by the user. In this way, it is possible to check the inventory at the timing that the user wants to know while reducing the number of communications.

  In the refrigerator 1 and the article management system according to the first embodiment, the refrigerator 1 has been described as an example. However, if the article storage apparatus stores a specific item in a specific place, the apparatus other than the refrigerator may be used. The present invention is applicable. For example, the present invention can be applied to food shelves that store rice or dry noodles.

  Further, in the refrigerator 1 and the article management system according to the first embodiment, the position of the shelf in the refrigerator compartment 1a of the refrigerator 1 is determined as a known area, but the size of the detection area is defined in the refrigerator compartment 1a. The vertical size of the detection area may be determined by detecting the position of the shelf from the image. As a method for detecting a shelf, for example, a method such as Hough transform for detecting a straight line from an image can be used. By making the detection area changeable according to the position of the shelf, it is possible to handle a plurality of models with different shelf sizes with a single program, or the refrigerator 1 that can change the height of the shelf. There is.

  In addition, although two or more feature points may be detected in the same article, a plurality of images detected from the same article can be obtained by regarding image feature amounts detected in similar positional relationships each time as one article. It is possible to prevent the feature quantity from being treated as a separate permanent article.

  In addition, the permanent article feature management unit 16 may delete the permanent articles that have not been detected by the permanent article detection unit 15 for a certain period from the database. In this way, it is possible to reduce useless processing of repeatedly determining the similarity of the image feature amount to the regular articles that the user can no longer put in the refrigerator 1, and to reduce the calculation load.

  In addition, the permanent item feature determination unit 14 holds in advance a reference image feature amount that is likely to appear as a permanent item, and performs processing only on features that are determined to be somewhat similar to the reference image feature amount. Also good. The reference image feature quantity that is likely to appear as a permanent article is, for example, an image feature quantity of a beverage can, an image feature quantity of a paper pack such as milk, and the like. By reducing the number of objects to be processed by the permanent article feature determination unit 14 in this way, the types of articles that can be handled are reduced. However, it is not desirable to manage the article as a permanent article such as a tableware or a storage plastic container containing the remainder of the dinner. An article can be reliably excluded from a permanent article. In addition, an image feature amount of a specific shape tableware or storage plastic container is stored in advance as a reference image feature amount, and an article having a feature somewhat similar to the reference image feature amount is excluded from the processing target. Similar effects can be obtained.

<< 2 >> Embodiment 2
<< 2-1 >> Configuration The refrigerator 2 and the article management system as the article storage device according to Embodiment 2 each time an image is captured in the same detection area in the refrigerator compartment 2a in the image acquired by the image acquisition unit 11. It is different from the refrigerator 1 and the article management system according to the first embodiment described above in that a plurality of regular articles that appear repeatedly are managed as a permanent article group to which the plurality of permanent articles belong.

  FIG. 7 is a block diagram schematically showing the configuration of the article management system including the refrigerator 2 and the refrigerator 2 according to Embodiment 2 of the present invention. The article management system according to Embodiment 2 includes a refrigerator 2 as an example of an article storage device and a mobile terminal 4. In FIG. 7, the same reference numerals as those shown in FIG. 1 are given to the same or corresponding elements as those shown in FIG. The point that the refrigerator 2 according to the second embodiment includes the permanent article group classification unit 21 and the permanent article detection unit 15 determines whether or not the permanent articles belonging to the permanent article group are lost. Different from the refrigerator 1 according to the first embodiment.

  The stock item group classification unit 21 uses the same product determination threshold value to determine whether or not the image feature amount determined to be similar to the image feature amount of the stock item stored in the stock item feature management unit 16 or predetermined. When the image feature amounts determined to be similar to each other using the same group determination threshold value appear repeatedly in the same detection area in the image acquired by the image acquisition unit 11, the repeated image feature One or more items whose quantity is detected are considered as the same item group. The permanent article detection unit 15 determines whether or not there are no permanent articles belonging to the permanent article group. When the determination result of the permanent item detection unit 15 indicates that there are no permanent items belonging to the permanent item group, the permanent item inventory management unit 22 sends information indicating the determination result from the communication unit 18 to the mobile terminal 4 as an external device. Send to.

<< 2-2 >> Operation FIGS. 8A and 8B are flowcharts illustrating processing performed by the refrigerator 2 according to the second embodiment. 8A shows the inventory management process, and FIG. 8B shows the update process of the feature of the permanent article and the update process of the classification of the permanent article group. In FIGS. 8A and 8B, the same or corresponding processing steps as those shown in FIGS. 2A and 2B are the same as those shown in FIGS. 2A and 2B. A code is attached. The refrigerator 2 according to the second embodiment performs the process of updating the permanent article group (step S301 in FIG. 8B) and the process of determining whether or not the permanent articles belonging to the permanent article group have disappeared (FIG. 8 (A) is different from the processing of the first embodiment shown in FIGS. 2 (A) and 2 (B) in that step S302) is performed. The inventory management process of FIG. 8A is mainly executed by the image feature amount extraction unit 12, the permanent item detection unit 15, the permanent item inventory management unit 22, and the communication unit 18. The update process of the feature of the permanent article and the update process of the classification of the permanent article group in FIG. 8B are mainly performed by the image feature amount extraction unit 12, the feature amount detection history storage unit 13, the permanent item feature determination unit 14, and the permanent item. This is executed by the article feature management unit 16 and the permanent article group classification unit 21.

  As shown in FIG. 8B, in the second embodiment, when the image feature amount extraction unit 12 extracts an image feature amount in step S102, the image feature is stored by the feature amount detection history storage 13, After performing the determination of the feature of the permanent item by the permanent item feature determination unit 14 and the update of the characteristic of the permanent item by the permanent item feature management unit 16 (shown in FIG. 5), the permanent item group classification unit 21 selects the permanent item group. Update (step S301).

  Next, the process of the permanent goods group classification | category part 21 is demonstrated. FIGS. 9A to 9E are diagrams showing examples of the arrangement of articles and detection positions of feature points of articles in the refrigerator compartment 2a of the refrigerator 2 according to the second embodiment, and examples of image feature quantity detection histories. It is. FIGS. 9A to 9E show the case where the arrangement of the articles and the detection positions of the feature points change in the order of FIGS. 9A, 9B, 9C, and 9D. In the second embodiment, unlike the case of the first embodiment shown in FIGS. 4A to 4E, “milk A” and “milk B” are different brands. “Milk A” and “Milk B” are different in brand name but are a food belonging to the same group “milk” for the user, so that both milks are placed in the upper right area in the refrigerator compartment 2a. .

Since the image feature quantities detected from “milk A” and “milk B” are similar in the shape of the milk pack, the detected image feature quantities are similar to some extent (to the extent that they are not regarded as the same article). . However, since the brands are different, the patterns printed on the milk cartons are different from each other, and the image feature amount has a difference that is not regarded as the same article. Therefore, the permanent article group classification unit 21 determines that, in the image acquired by the image acquisition unit 11, when the permanent articles having similar image feature amounts appear in the same region repeatedly, they belong to the same permanent article group. judge. That is, the same article determination threshold value indicating the difference allowed to be determined to be the same article (for example, the Euclidean distance between the two feature vectors to be compared) is determined to be the same group article. It is smaller than the same group determination threshold value indicating an allowable difference (for example, a Euclidean distance between two feature vectors to be compared). That is, it is as follows.
(Threshold for same article determination) <(Threshold for same group determination)
In other words, the degree of similarity between the image feature amounts when it is determined that the two items are the same item is high, and the difference between the image feature amounts of two similar items is not less than or equal to the same item determination threshold value. Don't be. On the other hand, the degree of similarity between image features when two articles are determined to belong to the same group is relatively low, and the difference between the image feature quantities of two articles belonging to the same group is the same for group determination. It is sufficient if it is below the threshold.

  In the image acquired by the image acquisition unit 11, the permanent item group classification unit 21 has an image feature amount difference that is larger than the same item determination threshold among a plurality of permanent items that repeatedly appear in the same detection area. If it is equal to or less than the threshold for use, the plurality of standing articles are regarded as the same group. The permanent article group classification unit 21 stores which permanent articles are determined to be in the same group.

  On the other hand, in the second embodiment, after the inventory information of the permanent articles is updated (step S122), when there are no permanent articles belonging to the permanent article group (YES in step S302), the fact that the permanent articles are no longer present. Notification is made (step S124). The communication unit 18 transmits the latest detection information of the permanent articles with the latest detection date and time among the permanent articles belonging to the permanent article group. Here, the standing articles that are not determined to be in the same group as other standing articles constitute the standing article group by the standing articles alone. In other words, if there are no permanent articles, there are no permanent articles belonging to the permanent article group.

  In the example of FIGS. 9A to 9E, the permanent goods group classification unit 21 detects “milk A” and “milk B” as permanent articles, and in the image acquired by the image acquisition unit 11, Appearing in the same place, they are regarded as the same permanent goods group. Therefore, if both “milk A” and “milk B” are lost, that is, if there is no milk in the refrigerator compartment 2a, the fact is notified.

  What the user of the refrigerator 2 wants to know is that “milk A” is not lost, but milk is lost in the refrigerator compartment 2a. In the second embodiment, the user is notified when “milk A” is lost or when “milk B” is not lost, but when milk that is an article belonging to the group of milk is lost. The user can be notified at a more appropriate timing.

<< 2-3 >> Effect As described above, in the refrigerator 2 and the article management system according to the second embodiment, in the images acquired by the image acquisition unit 11, the same articles are stored in the same area. Since it is regarded as a group and a notification is made that there is no longer a permanent item constituting the permanent item group, the user of the refrigerator 2 can receive a notification that the permanent item has been lost at a more appropriate timing.

  In the refrigerator 2 and the article management system according to the second embodiment, the image acquired by the image acquisition unit 11 is repeatedly placed in the same place even if the name of the article from which the image feature amount is extracted is not known. By using the feature, it can be determined whether or not it is a permanent article belonging to the same group.

  Further, in the refrigerator 2 and the article management system according to the second embodiment, not only the detection position but also the similarity between the image feature quantities of the permanent articles is used for determining whether or not they belong to the same group. For example, it is possible to reduce misjudgments in which regular articles that do not belong to the same group are regarded as the same group simply by being placed in a place close to chance, such as milk and beer.

  In addition, in the refrigerator 2 and the article management system according to the second embodiment, the latest detection information on the latest articles with the latest detection date and time is transmitted from the refrigerator 2 to the portable terminal 4. You may transmit the newest detection information of the some permanent article to which it belongs. In this way, although the amount of data to be transmitted increases, the user can more specifically know what kind of regular items have been lost.

  Except for the above, the second embodiment is the same as the first embodiment.

<< 3 >> Embodiment 3
The refrigerator 3 and the article management system as the article storage device according to the third embodiment is that the refrigerator 1 according to the first and second embodiments is that the information about the permanent articles is acquired from the external article information specifying unit 51. , 2 and the article management system.

  FIG. 10 is a block diagram schematically showing the configuration of the article management system including the refrigerator 3 and the refrigerator according to Embodiment 3 of the present invention. The article management system according to Embodiment 3 includes a refrigerator 3 as an example of an article storage device and a mobile terminal 4, and the refrigerator 3 is configured to be communicable with a Web server 5 that is a server computer. 10, components that are the same as or correspond to the components shown in FIG. 1 are assigned the same reference numerals as those shown in FIG. The refrigerator 3 according to the third embodiment is different from the refrigerator 1 according to the first embodiment in that the refrigerator 3 according to the third embodiment includes a permanent article information acquisition unit 31 that acquires article information of a permanent article from the Web server 5. In addition, it is also possible to provide the permanent article information acquisition unit 31 in the refrigerator 2 according to the second embodiment.

  The Web server 5 according to the third embodiment includes an article information identification unit 51 that identifies article information based on a permanent article image transmitted from the refrigerator 3, and an article information accumulation unit 52 as a storage device that accumulates the identified article information. With.

  In the third embodiment, the image around the feature point of the permanent article for which the permanent article information is not set among the image characteristic quantities of the permanent article stored in the permanent article feature management unit 16 (hereinafter referred to as “normal article image”). ) Along with the detected image feature amount, information on the detection area of the feature point, and the like, and the permanent article information acquisition unit 31 transmits the information to the Web server 5. Compared to the number of times the image is acquired in the refrigerator compartment of the refrigerator 3, the number of times that the standing articles are added is very small. Therefore, this process does not need to be performed every time the image in the refrigerator compartment is acquired. This process may be performed at a frequency of about once a week, for example.

  The article information identification unit 51 of the Web server 5 identifies what the article is (that is, identifies article information) based on the permanent article image transmitted from the permanent article information acquisition unit 31. Here, the article information specifying unit 51 may specify the article information from the correspondence between the article information stored in the article information storage unit 52 and the image feature amount, or viewing the permanent article image with human eyes. An article name may be specified.

  The article information identification unit 51 identifies article information, registers any article information that has not yet been accumulated in the article information accumulation unit 52 in the article information accumulation unit 52, and acquires the identified article information from the refrigerator 3. It transmits to the part 31.

  The refrigerator 3 that has received the specified article information can notify the mobile terminal 4 of a specific article name from the communication unit 18, so that the user can more specifically know the lost article.

  Further, the article information specifying unit 51 may reply to the refrigerator 3 when the received article image is not suitable as a permanent article. Examples of unsuitable items include pots or containers containing leftovers for dinner, and wall surfaces (background regions) of the refrigerator 3 that have been detected by mistake. When receiving a response from the article information specifying unit 51 that the permanent article information acquiring unit 31 is not suitable as a permanent article, the permanent article information acquiring unit 31 is not suitable as a permanent article by deleting the corresponding permanent article from the permanent article feature managing unit 16 It is possible to eliminate the management of the image feature amount as the image feature amount of the permanent article by mistake.

  Further, by storing the item information in the item information storage unit 52, it is possible to create a database of permanent items based on images taken in the refrigerator compartment, and to automatically specify the item information in the future using the database. Can be done.

  Note that the database stored in the Web server in the third embodiment is based on data automatically stored in the article information storage unit 52, and has the following characteristics.

In the refrigerator 3 according to the third embodiment, the database stored in the Web server 5 is
It is not necessary to send an image taken in the refrigerator room to the Web server 5 every time the door of the refrigerator 3 is opened and closed, and send it every time a predetermined period elapses (for example, once a week). Also good. Further, the Web server 5 may simply give an article name to the received permanent article image. For this reason, the cost of database creation is greatly reduced.

  In the third embodiment, the image stored as the database is an image obtained by photographing inside the refrigerator compartment of the refrigerator 3, and the detection area is also known. A database corresponding to the feature amount) can be constructed, and the problem that the image feature amount acquired from the image in the refrigerator compartment does not match the image feature amount stored in the database is unlikely to occur.

  Further, since the database constructed in the third embodiment is limited to permanent articles, it is a small-scale database compared to a database for all articles that may enter the refrigerator compartment of the refrigerator 3. is there. Therefore, in the third embodiment, a similar article search process can be performed at high speed.

  Except for the above, the third embodiment is the same as the first or second embodiment.

<< 4 >> Modifications.
FIG. 11 is a diagram illustrating an example of a hardware configuration of refrigerators 1 to 3 as article storage devices according to modifications of the first to third embodiments. The components 12 to 17 shown in FIG. 1, the components 12 to 16, 21, and 22 shown in FIG. 7, and the components 12 to 17, and 31 shown in FIG. 10 are storage devices that store programs as software. And a processor 61 as an information processing unit that executes a program stored in the memory 62 (for example, by a computer). Note that some of the components shown in FIG. 1 may be realized by the memory 62 shown in FIG. 11 and the processor 61 that executes the program.

  1, 2, 3 Refrigerator (article storage device), 1a, 2a Refrigerated room (storage room), 4 Mobile terminal (external device), 5 Web server (server computer), 11 Image acquisition unit, 12 Image feature amount extraction unit, DESCRIPTION OF SYMBOLS 13 Feature amount detection log | history storage part, 14 Regular article feature determination part, 15 Regular article detection part, 16 Regular article feature management part, 17, 22 Regular article inventory management part, 18 Communication part, 21 Regular article group classification part, 31 Item information acquisition unit, 41 Terminal communication unit, 42 Display operation unit, 51 Item information identification unit, 52 Item information storage unit

Claims (9)

An image acquisition unit for acquiring an image by photographing a storage room for storing articles;
An image feature amount extraction unit that extracts an image feature amount of the article from the image, and provides image feature amount information indicating the image feature amount and detection area information indicating a detection area where the image feature amount is detected;
A feature amount detection history storage unit that stores image feature information obtained by associating the image feature amount information provided from the image feature amount extraction unit and the detection area information as history data;
From the history data stored in the feature quantity detection history storage unit, image feature quantities determined to be similar to each other using a predetermined same article determination threshold are the same detection area in the image. A permanent article feature determination unit that determines the image feature quantity as an image feature quantity of a permanent article that is permanently provided in the storage room, when it appears repeatedly in the storage room;
A permanent article feature management unit for storing an image feature amount of the permanent article;
The latest image feature amount extracted by the image feature amount extraction unit from the image acquired by the image acquisition unit is similar to the image feature amount of the permanent item stored in the permanent item feature management unit. Depending on whether or not there is a permanent article detector that determines whether or not a permanent article is stored in the storage room,
An article storage device comprising:   When the image feature amount determined to be similar appears repeatedly in the same detection area in the image, the image feature amount is determined to be similar in a predetermined period or a predetermined number of photographing times. The article storage device according to claim 1, wherein the determined image feature amount appears in an appearance ratio equal to or higher than a predetermined reference ratio. A communication unit that communicates with an external device;
Information indicating the determination result of the permanent article detection unit, the permanent article inventory management unit that provides the external device with the communication unit,
The article storage device according to claim 1, further comprising:   The stock item inventory management unit, when the determination result of the permanent item detection unit indicates that there is no longer a permanent item, causes the communication unit to transmit information indicating the determination result to the external device. The article storage device according to claim 3. The image feature amount determined to be similar to the image feature amount of the permanent article using the same article determination threshold value, or determined to be similar to each other using a predetermined same group determination threshold value A regular article group classifying unit that regards one or more regular articles from which repeated image feature quantities are detected as the same regular article group when repeated image feature quantities repeatedly appear in the same detection area in the image; In addition,
The stock inventory management unit transmits information indicating the determination result from the communication unit to the external device when the determination result of the permanent item detection unit indicates that there is no longer a permanent item belonging to the permanent item group. The article storage device according to claim 3.   6. The apparatus according to claim 1, further comprising a permanent article information acquisition unit that acquires information on the permanent article from the server computer by transmitting an image of the article detected as a permanent article to the server computer. Item storage device of item 1. The image acquisition unit acquires the image every time the door of the article storage device is opened and closed,
If the image feature quantities similar to each other appear in the same detection area in the image from the history data stored in the feature quantity detection history storage section, the permanent article feature determination section The article storage device according to any one of claims 1 to 6, wherein the feature amount is determined as an image feature amount of a permanent article that is permanently provided in the storage room. The article storage device according to any one of claims 1 to 7,
A mobile device,
Comprising
The portable terminal includes a terminal communication unit that communicates with the article storage device, and a display operation unit that displays inventory information acquired from the article storage device. In the portable terminal, the display operation unit accepts an input operation of a stock item to be inquired about by the user, and the terminal communication unit sends inquiry information about the stock of the inquired stock item to the article storage device. Send
The article management system according to claim 8, wherein the article storage device transmits an answer to the inquiry about the inventory to a previous mobile terminal.

Images