JP6045521B2 - Storage management apparatus and method - Google Patents

Description

  The present invention relates to a storage management apparatus and method, and more particularly to management of articles stored in a storage, and taking in / out of articles from the storage using an image obtained by imaging the storage in the storage. Related to management.

  When storing articles in the storage, it is difficult for the user to memorize all the articles in the storage from when. So, for example, in the case of a refrigerator, you may forget that food is left in the cabinet and the expiry date will expire, or you may purchase (unnecessarily) duplicate items that are already in the cabinet. It can happen.

  Therefore, a method has been devised in which a camera for photographing the interior is installed and the interior management is performed from the photographed image.

  For example, in Patent Document 1, each time the door of the storage is closed, the interior is photographed, the type of the article in the warehouse is determined from the photographed image, and characters, barcodes, It is disclosed to acquire and display article information such as the article name, manufacturer, date of manufacture, and expiration date by recognizing a two-dimensional code. Further, it is also disclosed to display information indicating an image of an article or a position in a warehouse when the article cannot be specified.

  In addition, in Patent Document 2, each time the refrigerator door is closed, the interior is photographed, and the storage of the article is determined by comparing with the storage position at the previous photographing, and when the inventory period exceeds the warning period, An inventory management device that issues a warning is described.

JP 2012-193873 A JP 2010-121784 A

  However, the place where the information such as the expiration date differs depending on the article, and the article is not always placed so that the information can be seen from the camera, and the best taste is obtained from an image taken by a fixed camera as in Patent Document 1. It is difficult to acquire article information such as a time limit. Although it is also described that the user registers the product information, it is not easy to input the information for each product because the user has a lot of trouble.

  In addition, when multiple items are put in and out and the items in the warehouse are moved by a single opening and closing of the door, it is sufficient that the state in the cabinet is completely different between immediately after opening the door and before closing it. It can happen. In such a case, as in Patent Document 2, the method of comparing the images taken every time the door is opened / closed to determine entry / exit is too large to change the storage position so that the entry / exit cannot be determined correctly. Sometimes.

  In addition, when an article is placed and another article is placed in front of it, the former may be hidden behind the latter and may not be recognized.

The warehouse management apparatus of the present invention is
An in-compartment image acquisition unit that repeats the process of acquiring the in-container image obtained by imaging the inside of the storage;
A work determination unit that determines whether or not work for at least one of warehousing, unloading, and movement of articles is performed in the storage;
A pre-operation image storage unit that stores the internal image obtained by imaging the interior of the storage immediately before the start of the operation as the pre-operation image when the operation is started;
When the work is finished, immediately after the work is finished, the inside image obtained by imaging the inside of the storage is set as a post-work image, and from the pre-work image and the post-work image, A difference area extraction unit for extracting a difference area between the two images;
It is provided with the receipt / exit determination part which determines whether goods receipt / exit was performed from the said pre-work image or the said post-work image, and the said difference area.

  According to the present invention, even when a plurality of articles are put in and out and a movement of an article in the warehouse is performed during one opening and closing of the door, it is possible to accurately determine the putting in and out of the article.

It is a block diagram which shows the structure of the storage management apparatus in Embodiment 1 of this invention. It is the external view which looked at the storage provided with the storage management apparatus in Embodiment 1 from the top. It is a state transition diagram of the warehouse management apparatus in Embodiment 1. (A) And (B) is a figure which shows the process of the alignment performed with respect to the shift | offset | difference of the optical axis of a camera. (A)-(D) are the figures which show an example of the difference area extraction of the management apparatus in a warehouse in Embodiment 1, and the determination of warehousing and delivery. (A)-(C) are the figures which show the other example of the difference area extraction of the storage management apparatus in Embodiment 1, and the determination of warehousing and delivery. It is a flowchart which shows the procedure of the process in the door movement state of the storage management apparatus in Embodiment 1. 4 is a flowchart illustrating a processing procedure in a stationary state of the in-house management device according to the first embodiment. (A)-(E) is a figure which shows the process for the acquisition of the image in a store | warehouse | chamber by the store image acquisition part, and the memory | storage of the image before a work. 5 is a flowchart showing a processing procedure in a working state of the in-house management device in the first embodiment. 4 is a flowchart illustrating a procedure of inventory information update processing according to the first embodiment. It is a figure which shows the example of a display with the portable terminal based on the data delivered by the in-box information delivery apparatus in Embodiment 1. FIG. It is a block diagram which shows the structure of the storage management apparatus in Embodiment 2 of this invention. It is a block diagram which shows the structure of the storage management apparatus in Embodiment 3 of this invention. 14 is a flowchart illustrating a procedure of inventory information update processing according to the third embodiment. It is a figure which shows the example of the display in the portable terminal based on the data delivered by the in-box information delivery apparatus in Embodiment 3. FIG. It is a block diagram which shows the structure of the storage management apparatus in Embodiment 4 of this invention. 15 is a flowchart illustrating a procedure of inventory information update processing according to the fourth embodiment. It is a figure which shows the example of the display in the portable terminal based on the data delivered by the in-room information delivery apparatus in Embodiment 4. FIG.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a configuration of a storage management apparatus for storage in Embodiment 1 of the present invention. The storage is a refrigerator, for example, but the present invention is also applicable to other storages.
FIG. 2 shows the storage 10 provided with the storage management device 100 of FIG. FIG. 2 is a view of the state in which the door 14 of the storage 10 is opened as seen from above. The storage 10 includes a main body 12 and a door 14 that can be opened and closed with respect to the main body 12, and the internal management device 100 is installed inside the door 14. In the illustrated example, the door 14 is a so-called right-open door, and is rotatably held on a front edge of the right side wall of the main body 12 by a hinge (not shown).

  The warehouse management apparatus 100 illustrated in FIG. 1 includes an interior image acquisition unit 110, an interior image storage unit 122, a pre-operation image storage unit 124, a difference area extraction unit 130, a write control unit 132, an inventory An information storage unit 134, an image storage unit 126 before door closing, an in-compartment information distribution unit 136, and a control unit 140 are provided.

The internal image acquisition unit 110 acquires and outputs an internal image, and includes a camera 112 and a buffer 114.
The camera 112 images the interior periodically, that is, every 500 milliseconds, for example, every predetermined time, and outputs the interior image obtained by each capturing.

The camera 112 is installed so that when the door 14 of the storage 10 is opened, an image inside the storage can be acquired by imaging the inside of the storage. The camera 112 preferably includes a wide-angle lens so that an image of the entire interior can be acquired.
The camera 112 can be installed in any place other than the door 14 as long as the door is open and the article can be taken in or out when the article is taken in or out of the storage 10. It may be a place. For example, you may install in the ceiling in a warehouse.
When the in-compartment management apparatus 100 includes a camera integrally, the in-compartment management apparatus 100 is also provided at the same position as the camera 112. The storage management device 100 may be built in the side wall of the main body 12 of the storage 10 or the like.

The buffer 114 holds an internal image obtained by photographing with the camera 112 as a new internal image. The image held in the buffer 114 is rewritten every time the camera 112 takes a picture.
The in-compartment image acquisition unit 110 outputs a new in-compartment image held in the buffer 114.

The internal image storage unit 122 stores the new internal image acquired by the internal image acquisition unit 110. The internal image stored in the internal image storage unit 122 is read as the previous internal image when the internal image acquisition unit 110 next acquires a new internal image (when acquiring the next internal image). .
The pre-operation image storage unit 124 stores an image in the warehouse immediately before the operation (stored immediately before) when the operation (operation for warehousing, unloading, moving, etc.) of the user is performed in the warehouse. The internal image obtained by imaging the internal space is stored as a pre-operation image.
When the door 14 is closed, the pre-door closing image storage unit 126 stores an image in the cabinet in a stationary state immediately before closing as an image before closing the door.

  The difference area extraction unit 130 obtains a difference between the in-store image (post-work image) and the pre-work image when the work is performed by the user in the store. A relatively large area is extracted as a difference area.

The control unit 140 determines whether or not the inside of the storage 10 depends on the state of the storage 10, that is, whether the door 14 is closed, whether or not an operation for putting in / out an article, movement, etc. is performed in the storage 10. Control for causing the management apparatus 100 to transition to different operation modes and operate in the respective operation modes is performed.
The control unit 140 includes an open / close determination unit 142, a stationary determination unit 144, an operation mode determination unit 146, and a storage / departure determination unit 148.

The open / close determining unit 142 determines whether the door 14 of the storage 10 is open or closed, that is, whether the door 14 is closed.
The stillness determination unit 144 determines whether or not there is a temporal change, i.e., movement, in the internal image.
The operation mode determination unit 146 determines the operation mode based on the determination result by the open / close determination unit 142 and the determination result by the stillness determination unit 144, and the in-store management device 100 is determined by the operation mode determination unit 146. Operates in operation mode.
The entry / exit determination unit 148 determines whether or not an article has been entered or exited from the storage 10.

FIG. 3 is a state transition diagram showing the transition of the operation mode determined by the operation mode determination unit 146.
When the open / close determination unit 142 determines that the door 14 of the storage management device 100 is closed (when the storage 10 is in the door closed state), the storage management device 100 operates when the door is closed. It is in mode ST1.

  In the door closing operation mode ST1, if the opening / closing determination unit 142 determines that the door 14 is opened, it can be estimated that the door 14 has started to open (the storage 10 has shifted to the door moving state), so the storage management device 100 transitions to the door moving operation mode ST2.

In the door movement operation mode ST2, when the open / close determination unit 142 determines that the door is closed (the storage 10 has shifted to the door closed state), the storage management device 100 transitions to the door close operation mode ST1. .
In the door movement operation mode ST2, when the stationary determination unit 144 determines that the image in the storage has been changed (the storage 10 has shifted to the stationary state), the storage management device 100 enters the stationary operation mode ST3. Transition.

In the stationary operation mode ST3, when the open / close determination unit 142 determines that the door 14 is closed (the storage 10 has shifted to the door closed state), the storage management device 100 transitions to the door closed operation mode ST1. .
In the stationary operation mode ST3, when the stationary determination unit 144 determines that there is a change in the internal image (the storage 10 has shifted to the working state), the internal management device 100 transitions to the operational mode ST4. .

In the operation mode ST4, when the opening / closing determination unit 142 determines that the door 14 is closed (the storage 10 has shifted to the door closed state), the storage management device 100 transitions to the door closing operation mode ST1. .
In the operation time mode ST4, when the stillness determination unit 144 determines that there is no change in the internal image (the storage 10 has moved to the static state), the internal storage management device 100 transitions to the static operation mode ST3. To do.

  The door closing operation mode ST1, the door moving operation mode ST2, the stationary operation mode ST3, and the working operation mode ST4 of the storage management device 100 are respectively the door closed state, the door moving state, and the stationary operation of the storage 10. Since these modes are selected in the state and the working state, these will be referred to as a door closed state, a door moving state, a stationary state, and a working state, similarly to the state of the storage case 10.

The door movement operation mode (door movement state) and the stationary operation mode (stationary state) are the states in which the door 14 is open and the work for taking in and out of goods and movement is not performed in the warehouse. It is an operation mode, and these two states are collectively referred to as “non-working operation mode (non-working state)”, which is denoted by reference numeral ST5.
The stillness determination unit 144 has a role of detecting that the image movement has stopped in the door movement state ST2 and making a transition to the stationary state ST3, and in the stationary state ST3, the image movement has occurred. Is detected, and a transition to the work state ST4 occurs. In terms of having the latter role, it can be said that the stillness determination unit 144 functions as a determination unit (operation determination unit) that determines whether or not work is being performed.

Hereinafter, the operation of each unit of the in-compartment management apparatus 100 will be described in more detail.
The pre-work image storage unit 124 stores the in-store image (output from the camera 112 and buffered by the in-store image acquisition unit 110 immediately before the start of the work when the stationary state ST3 transits to the work state ST4. 114 is stored as a pre-operation image.
The pre-door closing image storage unit 126 stores the pre-working image stored in the pre-working image storage unit 124 as the pre-door closing image when the operation state ST4 transits to the door closed state ST1.

As described above, the open / close determination unit 142 determines whether the door 14 of the storage 10 is opened or closed, that is, whether the door 14 is closed. This determination is acquired by the internal image acquisition unit 110. This is performed based on the image in the store.
When the door 14 is closed, the interior is completely dark, so the average value of the brightness values of the interior image is extremely low. Therefore, it can be determined that the door 14 is closed if the average value of the luminance values of the images in the cabinet is equal to or less than a certain threshold value, and that the door 14 is open otherwise.

  As described above, the stillness determination unit 144 determines whether or not there is a temporal change, i.e., movement, in the internal image. This determination is made based on the internal image read from the internal image storage unit 122 (previous internal image) and the internal image acquired by the internal image acquisition unit 110 (new internal image). Specifically, the stillness determination unit 144 calculates the amount of change of the new internal image with respect to the previous internal image. As an example of the amount of change, for example, the sum of absolute values of pixel value differences for each pixel can be used.

In the door movement state ST2, the stillness determination unit 144 determines that there is no movement if the amount of change is not less than the threshold value Tha continuously for a predetermined time Tka or more, and determines an operation mode according to the determination result. The unit 146 causes the storage management device 100 to transition to the stationary state ST3.
In the stationary state ST3, the stationary determination unit 144 determines that there is a motion if the amount of change is continuously equal to or greater than the threshold Thb for a predetermined time Tkb or longer, and an operation mode determination unit is determined according to the determination result. 146 shifts the internal storage management apparatus 100 to the work state ST4.

In the working state ST4, the stillness determination unit 144 determines that there is no movement if the amount of change is continuously equal to or greater than a predetermined time Tkc and equal to or less than the threshold Thc, and an operation mode determination unit is determined according to the determination result. 146 changes the in-room management device 100 to the stationary state ST3.
The “predetermined time (Tka, Tkb, Tkc)” and “threshold (Tha, Thb, Thc)” in the determination in each of the above states are not necessarily the same as each other, and are optimal in each case. To be decided.

  When the difference area extraction unit 130 transitions from the work state ST4 to the stationary state ST3, the new region image acquired by the store image acquisition unit 110 and stored in the buffer 114 is used as the post-work image. A difference between these is obtained from the post-work image and the pre-work image stored in the pre-work image storage unit 124, and an area where the difference is larger than a predetermined threshold is extracted as a difference area. A difference area outline image representing the area is output, and an image at the same position as the difference area in the pre-work image or the post-work image is output as a difference image.

  The difference area outline image obtained by the above processing has a first gradation value (for example, a value representing white) inside the difference area, and a second gradation value (for example, black) outside the difference area. Value).

  The extraction of the difference area can be performed, for example, by extracting an area where the absolute value of the pixel value difference for each pixel between the pre-work image and the post-work image is equal to or greater than a certain threshold Thd. In this case, it is desirable to perform expansion processing and / or contraction processing of the difference region in order to reduce the influence of noise. Here, the expansion process is a process of replacing the difference value of the target pixel with a value greater than or equal to the threshold The if there is a pixel whose difference value is greater than or equal to a predetermined threshold The around the target pixel. If there is a pixel whose difference value is smaller than a predetermined threshold value Thf in the vicinity of, the difference value of the target pixel is replaced with a value smaller than the threshold value Thf.

  It is also conceivable that the direction of the optical axis of the camera 112 is deviated by touching the door when the user performs an operation for entering, leaving, or moving the article. If this deviation is slight, it is desirable to extract the difference area after aligning the images. Alignment refers to comparing two images and moving the images so that as many articles as possible in each image are at the same position.

  As an example of the alignment method, the feature points are extracted from the images, and the deviation of the feature points of both images is minimized as a whole, for example, the square sum of the deviation amounts is minimized. In addition, there is a method of translating one image. By doing so, the difference area can be accurately extracted even when the optical axis direction of the camera 112 is slightly shifted.

There may be a region that does not overlap between images after alignment, but the difference region is not extracted for that region. For example, the two images may be images Ia and Ib in FIGS. 4A and 4B, and the image after alignment may be as shown in FIG. 4C. In this case, the region Rac of the image Ia and the region Rbc of the image Ib match each other and overlap in the image Ic after alignment, but the region Rad of the image Ia does not have a matching region in the image Ib, There is no overlapping portion in the image Ic after alignment, and the region Rbd of the image Ib has no matching region in the image Ia, and there is no overlapping portion in the image Ic after alignment.
In this case, the difference area is extracted only for the area Rac and the area Rbc, and the areas Rad and Rbd are not targeted for the difference area extraction.

  The entry / exit determination unit 148 determines whether or not an article has been received or delivered every time the work state ST4 transits to the stationary state ST3.

The determination by the entry / exit determination unit 148 is performed for each of the difference areas extracted by the difference area extraction unit, and whether each difference area is due to entry (whether it corresponds to the received article) (Whether it corresponds to the goods that have been delivered) or other reasons (for example, due to the movement of the goods in the warehouse).
The determination as to whether or not the difference area is due to warehousing is performed based on the post-work image and the difference area outline image. The determination as to whether or not the difference area is due to shipping is performed based on the pre-work image and the difference area outline image.

  For example, if there is an image part with a contour similar to the difference area at the same position as the difference area in the pre-work image, the image part is reflected from the position in the storage box corresponding to the position of the difference area. When it can be determined that the article has been lost (removed) and there is no image portion with a contour similar to the difference area at any position in the post-work image, the disappearance is due to the delivery of the article. On the other hand, when there is an image portion with a contour similar to the difference area at another position in the post-work image, it can be determined that the disappearance is due to the movement of the article.

In addition, if there is an image part with a contour similar to the difference area at the same position as the difference area in the post-work image, the image part is reflected at the position in the storage box corresponding to the position of the difference area. If there is no contour image portion similar to the difference area at any position in the pre-work image, it can be determined that the appearance is due to the receipt of the article, On the other hand, when there is an image portion with a contour similar to the difference area at another position in the pre-work image, it can be determined that the appearance is due to the movement of the article.
Here, “similar” to the outline means that the degree of similarity is greater than or equal to a predetermined threshold, and therefore includes the same case.

  In addition, when the area of the detected difference area is extremely small, the difference area may be excluded from the determination target due to noise (not determined as entering or leaving).

  Also, if the number of difference areas is extremely large or the area of the difference area is extremely large, it is determined that the amount of change in the image has increased due to a reason other than the putting in / out action, and the putting in / out operation is performed on the difference area. Treat as if none. Here, as causes other than the taking-in / out operation, for example, the door 14 is largely moved, and many articles in the warehouse are moved.

  FIGS. 5A to 5D show an example of extraction of a difference area and determination of warehousing and leaving using the extracted difference area. In the example of FIGS. 5A to 5D, it is assumed that three articles Pa, Pb, and Pc are stored in advance in the warehouse, and the article Pc is taken out therefrom.

  5A is a pre-work image, and FIG. 5B is a post-work image. 5A includes image portions Aa, Ab, and Ac of articles Pa, Pb, and Pc, but the post-work image of FIG. 5B includes articles Pa, Pb. Image portions Ba and Bb are included, but the image portion of the article Pc does not exist.

  When the difference area is obtained from the pre-work image shown in FIG. 5A and the post-work image shown in FIG. 5B, the difference area is as indicated by a symbol Cc in FIG. By pattern matching between the contour of the difference area Cc and the contours of the image portions Aa, Ab, Ac of the pre-work image and the image portions Ba, Bb of the post-work image, the contour of the difference region Cc of FIG. The image portion having a similar outline and existing at the same position as the difference area Cc is the image portion Ac in the pre-operation image in FIG. 5A, while the image portion Ac in the post-operation image in FIG. It can be seen that there is no image portion having an outline similar to the outline of the difference area Cc at the same position as the difference area Cc. From this, it can be seen that the article Pc shown as the image portion Ac in the pre-work image has disappeared (removed) from the position in the storage corresponding to the position of the difference area Cc. Furthermore, since there is no image portion having an outline similar to the outline of the difference area Cc in the post-work image of FIG. 5B, it can be seen that the above disappearance is due to the departure.

  Similarly, when an image portion similar to the difference area exists in the post-work image and does not exist in the pre-work image, it can be determined that the delivery has been performed.

  When an image part having an outline similar to the difference area is in a place in the pre-work image and in another place in the post-work image, it is determined that the article shown as the image part has moved in the warehouse To do. Hereinafter, this point will be described with reference to FIGS.

  In the example of FIGS. 6A to 6C, it is assumed that three articles Pa, Pb, and Pc are stored in the warehouse in advance, and the article Pc is moved to another location in the warehouse. ing.

  FIG. 6A shows a pre-work image, and FIG. 6B shows a post-work image. The pre-operation image shown in FIG. 6A includes image portions Aa, Ab, and Ac of the articles Pa, Pb, and Pc. On the other hand, the post-work image in FIG. 6B includes image portions Ba, Bb, and Bc of the articles Pa, Pb, and Pc, and the image portions Ba and Bb are in the same positions as the image portions Aa and Ab. The position of the image part Bc is different from the position of the image part Ac.

  When the difference area is obtained from the pre-work image in FIG. 6 (A) and the post-work image in FIG. 6 (B), the difference areas are as indicated by reference characters Cc1 and Cc2 in FIG. 6 (C). By pattern matching between the contours of the difference areas Cc1 and Cc2 and the contours of the image portions Aa, Ab, Ac of the pre-work image and the image portions Ba, Bb, Bc of the post-work image, FIG. It can be seen that the two difference regions Cc1 and Cc2 have similar outlines.

  Further, an image portion having a contour similar to the contour of the difference region Cc1 in FIG. 6C and existing at the same position as the difference region Cc1 is an image portion Ac in the pre-operation image in FIG. On the other hand, in the post-work image of FIG. 6B, it can be seen that there is no image portion having an outline similar to the outline of the difference area Cc1 at the same position as the difference area Cc1. Therefore, the image part Ac of the pre-work image does not exist in the same position in the post-work image, and the article Pc shown as the image part Ac has disappeared from the position in the storage corresponding to the difference area Cc1. Judgment can be made.

  Further, an image portion having a contour similar to the contour of the difference region Cc2 in FIG. 6C and existing at the same position as the difference region Cc2 is an image portion Bc in the post-work image in FIG. 6B. On the other hand, in the pre-work image of FIG. 6A, it can be seen that there is no image portion having an outline similar to the difference area Cc2 at the same position as the difference area Cc2. From this, it is determined that the image part Bc of the post-work image does not exist in the pre-work image, and the article Pc shown as the image part Bc has appeared at a position in the storage box corresponding to the position of the difference area Cc2. can do.

  From the above, the article Pc shown as the image parts Ac and Bc similar to the difference areas Cc1 and Cc2 from the position in the storage box corresponding to the position of the difference area Cc1 to the inside of the storage area corresponding to the position of the difference area Cc2 It is determined that it has moved to the position.

  When the entry / exit determination unit 148 determines that the article has been delivered, the delivery is performed using the image (difference image) Dc in FIG. 5D, which is an image obtained by extracting only the difference area from the pre-operation image, as the delivery article image. This information is supplied to the writing control unit 132 together with information indicating that it has been received and information indicating the date and time of delivery.

  When the entry / exit determination unit 148 determines that the article has been received, an image (difference image) obtained by extracting only the difference area from the post-work image is used as the storage article image, and information indicating that the entry has been performed, This information is supplied to the write controller 132 together with the information shown.

  The writing control unit 132 associates the supplied information with each other, and stores the stock information in the stock information storage unit 134 as stock information about the product shown as a warehousing product image or a warehousing product image.

  When the door determination unit 142 determines that the door is closed, the door pre-close image storage unit 126 stores the pre-operation image stored in the pre-operation image storage unit 124 as the pre-door closing image.

  The in-chamber information distribution unit 136 distributes the inventory information stored in the inventory information storage unit 134 and the image before door closing stored in the image storage unit 126 before door closing to an external terminal (not shown).

  Next, a procedure of processing in the door movement state (door movement operation mode) ST2 of FIG. 3 will be described with reference to FIG.

  First, in step S <b> 101, the internal image acquisition unit 110 acquires an internal image (new internal image) obtained by photographing with the camera 112. This image acquisition is repeated periodically.

  Subsequently, in step S102, when the open / close determination unit 142 determines whether the door 14 is open and determines that the door 14 is closed (NO in S102), the process proceeds to the door closed state ST1 (S103).

  When it is determined in step S102 that the door 14 is open (YES in S102), the process proceeds to step S104, and the stillness determination unit 144 stores the internal image stored in the internal image storage unit 122, that is, the previous internal image. The internal image acquired by the acquisition unit 110 (previous internal image) is read, and the change amount of the new internal image (the internal image acquired by the current internal image acquisition unit 110) with respect to the previous internal image is calculated. To do. The calculated change amount is used for determination in step S106 described later.

  Subsequently, in step S105, the new internal image is stored in the internal image storage unit 122. The internal image stored in the internal image storage unit 122 is read as the previous internal image after the internal image acquisition unit next acquires a new internal image.

Next, in step S106, the stillness determination unit 144 determines whether or not a state in which the amount of change calculated in step 104 is equal to or less than a predetermined threshold Tha continues for a predetermined time Tka.
For this determination, data indicating the amount of change calculated in step S104 may be accumulated and used for a predetermined time Tka or more, and a determination result as to whether or not the amount of change is equal to or less than a threshold value Tha. Alternatively, it may be accumulated and used over a predetermined time Tka.
When the door 14 is moving, the image acquired by the internal image acquisition unit 110 continues to change. When the door 14 is stationary, the change in the interior image is reduced. Therefore, when the amount of change calculated in step S104 is equal to or less than the predetermined threshold value Tha and the state continues continuously for a predetermined time Tka (YES in S106), the opening operation of the door 14 is stopped. It is assumed that the transition is made to the stationary state ST3 (S107).

  When the condition that “the amount of change calculated in step S104 is equal to or smaller than a predetermined threshold Tha and the state continues continuously for a predetermined time Tka” is not satisfied (NO in S106), the process proceeds to step S101. Return.

Next, the processing procedure in the stationary state (operation mode at rest) ST3 will be described with reference to FIG.
The processes in steps S201 to S205 in FIG. 8 are the same as steps S101 to S105 in FIG.

However, when the open / close determination unit 142 determines in step S202 that the door 14 is closed (NO in S202), the process proceeds to step S206, and the pre-operation stored in the pre-operation image storage unit 124 in step S210 described later. After the image is stored in the pre-door closing image storage unit 126 as the pre-door closing image, the process proceeds to the door closing state ST1 (S203).
What is stored in step S206 is an image obtained by photographing at a time point closest to the time point when the door is closed in the stationary state ST3 before the door 14 is closed.

  In step S207 performed after step S205, the stillness determination unit 144 determines whether or not the amount of change calculated in step S204 is equal to or greater than the threshold value (Thb). If the change amount is equal to or greater than the threshold value Thb, the process proceeds to step S208. Then, it is determined whether or not the state equal to or greater than the threshold value continues for a predetermined time Tkb or more.

  If the amount of change calculated in step S204 is equal to or greater than the threshold Thb for a predetermined time Tkb, the user's hand enters the storage 10 and puts the article in or out or moves the article in the storage. Can be estimated. Therefore, the process transits to the work state ST4 (S209).

  As the amount of change calculated in step S204 and used for the determination in step S207, for example, the sum of absolute values of differences in pixel values for each pixel is used. When calculating the sum of absolute values, it is better to give a weight to the calculation of absolute values according to the coordinates in the image. For example, in this embodiment, the camera 112 is installed on the door 14 that opens to the right. In that case, the user's hand should appear from the left side in the image inside the image obtained by the camera 112. . Therefore, as the amount of change used for the determination in step S207 in FIG. 8, if the amount of change is calculated with a greater weight on the left side area of the image in the store, that will be immediately when the user's hand enters the store. Can be detected and a transition can be made to the work state (operation mode during work) ST4. Also, the state can be determined accurately even when taking in and out is performed only in the range shown in the left area of the image inside the cabinet.

  Which direction of the image is to be weighted may be determined in advance according to the opening direction of the door and the installation position of the camera 112, or may be learned from the image acquired by the in-chamber image acquisition unit 110. good. For example, if the optical flow is calculated from the image in the cabinet and the moving direction of the user's hand is estimated, it can be estimated from which the camera 112 is installed so that the user's hand can enter. A weighting method can be determined based on the result.

  In addition, even when the operation of closing the door 14 from the stationary state ST3 is performed, there is a possibility that the transition to the working state ST4 is made based on the determination based on the change amount of the image, but the door 14 is closed by the open / close determination unit 142 as described later. If it is determined that the transition is made to the door closed state ST1 at that time, there is no problem even if the state is temporarily shifted to the work state ST4 as described above. Further, even when the door 14 is moved, there is a possibility of transition to the work state ST4 based on the determination based on the amount of change in the image. In this case as well, in the process in the work state ST4, as described later, the door is closed. There is no problem because the state transitions to the state ST1 (S303 in FIG. 10 described later), and entry / exit determination is not performed.

If the amount of change calculated in step S204 is not equal to or greater than the threshold Thb (NO in S207), the process proceeds to step S210, and the new in-store image (the image acquired as the new in-store image in S201) is used as the pre-operation image. The image is stored in the pre-operation image storage unit 124, and then the process returns to step S201.
If it is determined in step S208 that the state equal to or greater than the threshold Thb has not continued for a predetermined time Tkb or more, the process returns to step S201.

  As a result of performing such processing, the pre-operation image storage unit 124 always stores the latest internal image in the stationary state ST3 (the latest image every time the internal image acquisition unit 110 acquires the internal image). As described later, when a transition is made from the stationary state ST3 to the work state ST4, the image inside the warehouse immediately before the transition is held (that is, always before work). That is, the in-chamber image acquired before a series of image changes is retained).

  This point will be described with reference to FIGS. FIG. 9A shows a time series of new internal images acquired in step S201 for each predetermined time. FIG. 9B shows the determination result in step S207, that is, whether the amount of change calculated in step S203 is equal to or larger than the threshold value or smaller than the threshold value by “large” and “small”. FIG. 9C shows an image stored in the internal image storage unit 122 in step S205. FIG. 9D shows an image stored as a pre-operation image in step S210. FIG. 9E shows “YES” and “NO” whether or not the determination result of step S208, that is, the determination result of step S207 was “large” continuously for a predetermined time Tkb or more.

In the example shown in the drawing, the period in which the image in the warehouse is acquired three times corresponds to “predetermined time Tkb”, and the determination result of “large” in FIG. When the determination is made, as shown in FIG. 9E, the process shifts to the work state ST4 (S209).
In this case, the same image stored in the in-chamber image storage unit 122 in step S205 immediately before the determination result of “large” three times in succession is stored as the pre-operation image storage unit 124. In this state, the process transits to the work state ST4 (S209).

Next, the procedure of the process in the work state (operation mode at work) ST4 will be described with reference to FIG.
The processes in steps S301 to S305 in FIG. 10 are the same as steps S101 to S105 in FIG. 7 or steps S201 to S205 in FIG.

In step S306, the stillness determination unit 144 determines whether or not the amount of change calculated in step S304 is equal to or less than the threshold value Thc for a predetermined time Tkc.
As the amount of change calculated in step S304 and used for the determination in step S306, the sum of absolute values of pixel value differences for each pixel may be used as described in steps S204 and S207. It is even better to give a weight to the absolute value calculation by coordinates.

  When the change amount calculated in step S304 is equal to or less than the threshold value Thc for a predetermined time Tkc (YES in S306), it is estimated that the user's hand is taken out of the storage 10 and the work is finished. it can. Therefore, the inventory information update process (S307) is performed, and then the state transitions to the stationary state ST3 (S308).

FIG. 11 shows details of the stock information update process S307 of FIG.
First, in step S401, the difference area extraction unit 130 extracts a difference area.
That is, the difference area extraction unit 130 sets the new internal image as a post-work image, and the post-work image and the pre-work image stored in the pre-work image storage unit 124 (the pre-work image in step S210 in FIG. 8). The difference area is extracted from the stored image), and a difference area outline image representing the difference area is generated.

  After extracting the difference areas, the entry / exit determination unit 148 performs the processing of steps S402 to S406 for each difference area. That is, any one of the detected difference areas (or one difference area when there is one detected difference area) is selected (S402), and the selected difference area is loaded or unloaded. (S403, S404). For example, it is determined whether or not warehousing has been performed first (S403), and if it is not determined that warehousing has been performed (NO in S403), then it is determined whether or not warehousing has been performed ( S404) is performed.

When it is determined in step S403 that warehousing has been performed (YES in S403), the process proceeds to step S405, the date and time when warehousing is performed, the image of the difference area in the image after warehousing (warehouse article image), and warehousing is performed. Information indicating that the change has occurred is supplied to the write control unit 132, and the write control unit 132 associates these pieces of information with each other to generate inventory information, and stores it in the inventory information storage unit 134.
When it is determined in step S404 that the delivery has been performed (YES in S404), the process proceeds to step S406, the date and time when the delivery was performed, and the image at the position corresponding to the difference area in the pre-shipping image (shipped article image). ), Information indicating that the delivery has been performed is supplied to the write control unit 132, and the write control unit 132 associates these pieces of information with each other to generate inventory information and stores it in the inventory information storage unit 134.

  After step S405 or S406, the process proceeds to step S407. If it is determined in steps S403 and S404 that neither entering nor leaving has been performed, the process proceeds to step S407 (without passing through S405 and S406). In step S407, it is determined whether or not there is an unprocessed difference area (whether there is a difference area that has not been subjected to the processes in steps S403 and S404). If there is an unprocessed difference area, the process returns to step S402.

  When the processes for all the difference areas are completed (NO in S407), the inventory information update process is terminated. Thereafter, the process proceeds to step S308 in FIG. 10, and a transition to the stationary state ST3 is performed.

  As described above, the inventory information indicating what kind of goods have been received or delivered can be stored in the inventory information storage unit 134.

  On the other hand, the image in the store | warehouse | chamber acquired by the process of step S206 in the still state just before closing the door 14 is memorize | stored in the image storage part 126 before door closing as an image before door closing. The pre-door closing image stored in the pre-door closing image storage unit 126 and the inventory information stored in the inventory information storage unit 134 are stored in the portable terminal not shown in FIG. It is desirable to make it viewable.

  As an example of a method for allowing the image before door closing and the inventory information to be viewed on the portable terminal, the in-compartment information distribution unit 136 operates as a web (Web) server, and the in-compartment information distribution unit 136 is accessed by the web browser of the portable terminal. There is a way to do it.

  For example, the in-compartment information distribution unit 136 stores the image before door closing (stored in the pre-door closing image storage unit 126), which is the current in-store image, and information about loading / unloading of articles (stored in the inventory information storage unit 134). HTML file is displayed and displayed. FIG. 12 shows a display example 501 on the portable terminal 500. By doing so, the user can use the web browser of the portable terminal 500 to know the latest image in the warehouse and what kind of goods have entered or exited.

  As described above, in the present embodiment, each time an operation such as loading / unloading or moving of an article is performed in the warehouse, the determination of entry / exit is performed (when the door is open, the (series) of movement of the image is performed). Each time it is finished, entry and exit are determined based on the images before and after the movement), so even if the user puts in and out the article multiple times during one door opening and closing, In and out can be detected.

For example, even if an article is received and another article is received in front of the article, that is, closer to the camera 112 than the article, the determination of entry and exit is performed once. Warehousing can be accurately determined even for an article hidden behind a certain article when the item is closed.
Similarly, even if a certain article is delivered and another article is delivered from the back of the article, that is, from the side farther from the camera 112 than the article, the determination of entry and exit is performed once. It is also possible to accurately detect the delivery of an article hidden in the shadow of the article immediately after opening.

  In addition, when the user moves the article in the warehouse, it can be detected that the article has moved even if the article disappears from one position in the image and the same article appears in another position. Thus, it is possible to prevent erroneous determination that entry or exit has been performed. In addition, when movement is detected in this way, the subsequent difference area is extracted based on the image after movement, so that the entry and exit of the article can be accurately determined.

  Further, by installing the camera 112 on the door 14, it is possible to obtain an image in a wider range in the storage as compared with the case where it is installed on the main body 12 of the storage 10. Thereby, the number of cameras can be reduced.

  Also, if the amount of change is calculated by weighting the direction in which the hand of the in-chamber image enters, the work by the user can be detected quickly.

  In the present embodiment, it is determined whether an operation for loading / unloading or moving an article is performed based on the amount of change in the image. However, a different reference may be used. For example, the determination criterion may be that the skin color region has entered from a certain direction.

  Moreover, in this Embodiment, although one camera which is an image acquisition part in a store | warehouse | chamber is installed with respect to the storage 10, it may install multiple units | sets. For example, when there are a plurality of partitioned spaces in the storage, for example, when the storage 10 is a refrigerator and there are a plurality of partitioned spaces such as a refrigerator room and a vegetable room, You may install a camera.

  Further, in the present embodiment, the open / close determination unit 142 uses the door open / close determination criterion as to whether or not the average value of the pixel values of the interior image is equal to or less than the threshold value, but the determination may be performed by a different method. For example, the door opening / closing may be determined by a sensor that detects that the door of the storage 10 and the main body 12 are in close contact with each other. By doing this, the number of parts increases, but when the door is closed, the operation of the camera 112 of the in-chamber image acquisition unit 110 can be stopped, thereby suppressing the power consumption during operation. Can do.

  Moreover, in this Embodiment, although the outline of the difference area | region was utilized for the determination of warehousing and leaving, you may utilize the similarity with the image of a bottom face or a shelf surface. For example, a feature amount such as the color of the image of the bottom surface or shelf surface is stored in advance, and which of the difference region image in the pre-operation image and the difference region image in the pre-operation image is the bottom surface or shelf surface It is determined whether the image has a feature close to that of the image. When the former is close, there is a high possibility that the bottom surface or the shelf surface is not visible, so it can be determined that the warehousing has been performed. When the latter is close, there is a high possibility that the bottom surface or the shelf surface is visible, so that it can be determined that the shipment has been performed. Whether or not the image has characteristics close to the image of the bottom surface or the shelf surface is determined by, for example, the absolute difference between the pixel values for each pixel, as in the case where the stillness determination unit 144 determines whether or not it is in the working state. This can be done using the sum of values. In this case, an image determined not to be close to the image on the bottom surface or the shelf surface may be stored and displayed as an image of an article that has been received or delivered.

  By doing so, it is not necessary to perform contour extraction and matching processing, so that the processing can be simplified. However, the detection accuracy is lowered when an article is newly received before another article or when another article appears behind the article that has been delivered.

In the above example, the pre-operation image storage unit 124 acquires the in-store image (output from the camera 112 and stored in the buffer 114) acquired by the internal image acquisition unit 110 when transitioning from the stationary state ST3 to the work state ST4. Is stored as a pre-operation image (step S210 in FIG. 8), but instead from the in-compartment image storage unit 122 when transitioning from the stationary state ST3 to the working state ST4. The read previous warehouse image may be stored as a pre-work image.
In short, when transitioning from the stationary state ST3 to the work state ST4, an in-store image obtained by imaging the inside of the storage immediately before the start of the work may be stored as a pre-work image.

Further, in the above example, when the difference area extraction unit 130 transits from the work state ST4 to the stationary state ST3, the new image stored in the buffer 114 is acquired by the image acquisition unit 110 and stored in the buffer 114. A difference between these images is obtained from the post-work image and the pre-work image stored in the pre-work image storage unit 124, but instead, the work state ST4 is changed to the stationary state ST3. At the time of the transition, the previous image stored in the store image storage unit 122 read from the store image storage unit 122 is set as a post-work image, and the post-work image and the pre-work image stored in the pre-work image storage unit 124 are set between these images. It is good also as calculating | requiring the difference of.
In short, when transitioning from the work state ST4 to the stationary state ST3, an in-store image obtained by imaging the inside of the storage immediately after the end of the work may be stored as a post-work image.

In the above example, the pre-door closing image storage unit 126 stores the pre-working image stored in the pre-working image storage unit 124 immediately before the door is closed as the pre-door closing image (step S206 in FIG. 8). Instead, the in-store image acquired by the in-store image acquisition unit 110 immediately before the door is closed, or the in-store image read from the in-store image storage unit 122 immediately before the door is closed (previous store image). May be stored as an image before closing the door.
In short, it is only necessary to store the in-compartment image obtained by imaging the interior of the storage as the in-compartment image before the door closes at the time closest to the time when the door is closed in the stationary state before the door is closed.

Embodiment 2. FIG.
In the first embodiment, whether or not the storage is in the working state is determined by the stillness determination unit 144 based on the presence / absence of a change in the internal image, but this determination may be performed by a different method. As an example, FIG. 13 shows a configuration using a proximity sensor.
The in-house management device 100a shown in FIG. 13 is generally the same as the in-house management device 100 in FIG. 1, but a work determination unit 145 is provided instead of the stationary determination unit 144.

The proximity sensor 150 detects this when the user puts a hand in the storage 10, and outputs a signal indicating the detection result. This signal is transmitted to the work determination unit 145.
When such a signal is input, the work determination unit 145 determines that an operation for putting in / out the article is being performed when proximity is detected by the proximity sensor 150 with the door open. .
When a signal indicating proximity is output from the proximity sensor 150 in the stationary state ST3, the work determination unit 145 determines that work is being performed. In this case, it may be determined that work is being performed when a signal indicating proximity has continued for a predetermined time or more. When it is determined that work is being performed, the operation mode determination unit 146 changes the in-house management device 100a to the work state ST4 accordingly.
When no signal indicating proximity is output from the proximity sensor 150 in the work state ST4, the work determination unit 145 determines that work is not being performed. In this case, it may be determined that the work is not performed when a state in which a signal indicating proximity is not output continues for a predetermined time or more. If it is determined that the work is not being performed, the operation mode determination unit 146 changes the in-house management device 100a to the stationary state ST3 accordingly.

  As the proximity sensor 150, a sensor that detects the presence or absence of an object such as a human hand based on the presence or absence of a reflected wave by emitting electromagnetic waves such as infrared rays or ultrasonic waves can be used.

  If it is determined that the work is being performed based on the output of the proximity sensor instead of image processing, it is not necessary to perform image recognition for stillness determination in the stillness determination unit 144 according to the first embodiment. The computing capacity and power consumption required for the in-compartment management apparatus 100 can be suppressed.

When the proximity sensor is used to determine whether work is being performed, the open / close determination unit 142 is omitted, and if a human hand is detected by the proximity sensor, it is determined that the door is open. It is also good. (In FIG. 13, the open / close determination unit 142 is indicated by a dotted line to indicate that the open / close determination unit 142 can be omitted.)
If the door opening / closing determination is performed based on the output of the proximity sensor, it is not necessary to perform image recognition for determining whether the door is closed by the opening / closing determination unit 142 of the first embodiment. In addition, it is possible to further reduce the computing power and power consumption required for the storage management device 100.

Embodiment 3 FIG.
FIG. 14 is a block diagram showing a configuration of the warehouse management apparatus 100 according to Embodiment 3 of the present invention.
The in-house management device 100b in the present embodiment is generally the same as the in-house management device 100 of FIG. 1, except that the same article determination unit 160 is added, and instead of the write control unit 132 and the inventory information storage unit 134. Are different in that a write control unit 162 and an inventory information storage unit 164 are provided.

The write control unit 162 has the same function as the write control unit 132 of the first embodiment, and also stores a part of the information stored in the inventory information storage unit 164 according to the determination result by the same article determination unit 160. It differs from the write controller 132 of the first embodiment in that it is erased.
The inventory information storage unit 164 stores the same information as the inventory information storage unit 134 of the first embodiment, but as a result of the deletion by the write control unit 162, the information of the articles delivered from the storage is deleted, It differs from the inventory information storage unit 134 of the first embodiment in that only information on articles actually stored in the storage is accumulated.

  The same article determination unit 160 is the same as the newly issued article among the articles stored in the inventory information storage unit 164 when the entry / exit determination unit 148 determines that the article has been delivered. It is determined whether or not there is. The determination of whether two images are of the same article can be made, for example, based on whether the similarity of the shape, color, local feature amount, etc. of the article shown as two images is greater than a certain threshold.

  When the same article determination unit 160 determines that there is the same article as the newly delivered article among the articles stored in the inventory information storage unit 164, the same article determination unit 160 notifies the writing control unit 162 of that fact. In response to this, the writing control unit 162 erases the information about the same article as the newly delivered article stored in the inventory information storage unit 164.

FIG. 15 shows stock information update processing S307b in the present embodiment.
This update process S307b is performed in place of the process S307 of FIG. 11 described in connection with the first embodiment. Except for the update process S307b, the operation of the present embodiment is the same as that of the first embodiment.

  In FIG. 15, steps S401 to S404 are the same as those in FIG. Steps S405b and S406b are the same as steps S405 and S406 except that the information supply destination is different.

If it is determined in step S403 that the warehousing has been performed (YES in S403), the process proceeds to step S405b, and the date and time when the warehousing is performed, the image of the difference area in the image after warehousing (warehousing article image), and the warehousing is performed. Information indicating this is supplied to the writing control unit 162, and the writing control unit 162 associates these pieces of information with each other to generate inventory information, and stores it in the inventory information storage unit 164.
After step S405b, the process proceeds to step S407.

If it is not determined in step S404 that the delivery has been performed (NO), the process proceeds to step S407.
When it is determined in step S404 that the delivery has been performed (YES in S404), the process proceeds to step S406b, the date and time when the delivery was performed, and an image at the position corresponding to the difference area in the pre-shipping image (shipped article image). ), Information indicating that the delivery has been performed is supplied to the write control unit 132, and the write control unit 162 associates these pieces of information with each other to generate stock information and stores it in the stock information storage unit 164.

  After step S406b, the process proceeds to step S411. In step S411, the same article determination unit 160 compares the unloading article image in the pre-shipping image with each of the entering article images of the articles stored in the inventory information storage unit 164 (the entering article image stored in S405b). Then, it is determined whether there are similar articles (articles estimated to be the same).

If it is not determined by the same article determination unit 160 that an article similar to the outgoing article image (an article estimated to be the same) is stored in the inventory information storage unit 164 (NO in S411), step S407 is performed. Proceed to
If it is determined by the same article determination unit 160 that an article similar to the issued article image (an article estimated to be the same) is stored in the inventory information storage unit 164, the article that was in the warehouse has been taken out. The estimation result is transmitted to the write control unit 162. In response to this, the writing control unit 162 deletes the information (such as warehousing information and warehousing date / time) of the article estimated to be delivered from the inventory information storage unit 164 (S412), and proceeds to step S407.

  If the same article determination unit 160 determines that similar articles (articles estimated to be the same) are not stored in the inventory information storage unit 164 (NO in S411), an image that does not exist has been issued. Therefore, it is estimated that there is an error in the determination, nothing is done, and the process proceeds to step S407.

In step S407, it is determined whether or not there is an unprocessed difference area (whether there is a difference area that has not been subjected to the processes of steps S403, S404, S405b, S406b, S411, and S412). If there is, the process returns to step S402.
If there is no unprocessed difference area in step S407 (NO in S407), the inventory information update process is terminated.
Thereafter, the process proceeds to step S308 in FIG. 10, and a transition to the stationary state ST3 is performed.

  By performing (repeating) such a process, the inventory information storage unit 164 accumulates only information related to the articles in the storage, and the information stored in the inventory information storage unit 164 actually stores the information in the storage. You will know what kind of goods you have. FIG. 16 shows a display example 502 on the portable terminal 500 based on the data created by the internal information distribution unit 136. In the example of FIG. 16, there is an article hidden in the shadow of the article in the image in the warehouse, but the user grasps what kind of article has been stored in the warehouse from when the lower display is viewed (stored). be able to.

  As described above, according to the present embodiment, the inventory management is performed by comparing the goods determined to be delivered and the articles received in the past, so the user can determine what kind of articles are in the warehouse from when. I can grasp it.

  In addition, since the determination of warehousing and warehousing and the determination of whether or not they are the same article are not always successful, the information stored in the inventory information storage unit 164 can be corrected by the user using a portable terminal or the like. It is desirable to do.

Embodiment 4 FIG.
FIG. 17 is a block diagram showing the configuration of the warehouse management apparatus according to Embodiment 4 of the present invention.
The in-store management device 100c in the present embodiment is generally the same as the in-store management device 100 of FIG. 1, except that an article recognition unit 170 and an article database 171 are added, and a write control unit 132 and an inventory information storage unit. The difference is that a write control unit 172 and an inventory information storage unit 174 are provided instead of 134.

  The article recognizing unit 170 is an image of a difference area corresponding to an article that has been determined to have been entered or exited by the entry / exit determination unit 148, that is, an article that is shown as the image from the entry article image or the exit article image. Specify the type (name). The “type of article” referred to here means classification based on the type of article, product name, manufacturer name, and the like. The type of the article is specified by identifying the article from, for example, characteristics such as the shape, color, and size of the article, the article or its label, and character information written on the package.

An example of a method for specifying the type of article (article recognition) based on characteristics such as the shape, color, and size of the article will be described. First, local features are extracted for each type of articles that may enter the storage 10 in advance, and information representing the extracted local features is registered in the article database 171 together with information representing the type of the article. . When the article is recognized, the local feature amount of the incoming article image or the outgoing article image is extracted, and the similarity is calculated between the local feature quantity with each article in the article database 171. Among them, the name of the article whose similarity is equal to or higher than a certain threshold and is the highest is taken as the recognition result.
As a result of the article recognition, if there is no article whose similarity is equal to or greater than the threshold value in the article database, the process proceeds as if the article could not be specified.

The writing control unit 172 has the same function as the writing control unit 132 of the first embodiment, the type of the article identified by the article recognition unit 170, the image of the article, the number of articles of the same type as the article, It differs from the inventory information storage unit 134 of the first embodiment in that the goods receipt date and time are associated with each other and generated as inventory information.
The inventory information storage unit 174 has the same function as the inventory information storage unit 134 of the first embodiment, and also stores the inventory information generated by the write control unit 172, and the inventory information storage unit 134 of the first embodiment. Different.

  As the type of the item, a type representing the type specified by the item recognition unit 170 is stored. As the image, an image of the difference area (difference image) extracted by the difference area extraction unit 130 is stored. The storage date and time of each article is stored in the inventory information storage unit 174 by the same process as described for the inventory information storage unit 134 of the first embodiment. As the number of articles of the same type, those counted as described later are stored.

  If the article can be identified by the article recognition unit 170, the writing control unit 172 determines whether or not the same type of article is stored in the inventory information storage unit 174, and stores the inventory information according to the determination result. The information stored in the unit 174 is updated.

  If an article has been received and an article of the same type is already stored in the inventory information storage unit 174, it means that an article of the same type as the article in the store has been added, so the write control unit 172 The number of articles of the corresponding type stored in the inventory information storage unit 174 is increased, and the date and time of receipt is added. Here, the image of the corresponding article (the incoming article image) may be replaced with a new incoming article image.

  If an article has been received and an article of the same type is not stored in the inventory information storage unit 174, it means that an article that was not in the store has been newly received, so the write control unit 172 A new article type and an incoming article image are added to the storage unit 174, the number of articles is set to 1, and the incoming date and time is added.

  If an article has been issued and an article of the same type is already stored in the inventory information storage unit 174, it means that one article that was in the warehouse has been taken out, so the write control unit 172 The number of articles to be reduced is decreased, and the information indicating the date and time of receipt of the delivered article is deleted. Here, when the number of articles before the decrease is one, it indicates that the corresponding articles disappear from the warehouse, so the type of the corresponding article is deleted from the inventory information storage unit 174.

  If an article has been issued and an article of the same type is not stored in the inventory information storage unit 174, an image that does not exist has been issued, and the write control unit 172 estimates that there was an error in the determination. And do nothing.

FIG. 18 shows stock information update processing S307c in the present embodiment.
This update process S307c is performed in place of the process S307 of FIG. 11 described in connection with the first embodiment. Except for the update process S307c, the operation of the present embodiment is the same as that of the first embodiment.

  In FIG. 15, steps S401, S402, S403, and S404 are the same as those in FIG. Steps S405c and S406c are the same as steps S405 and S406 except that the information supply destination is different.

  If it is determined in step S403 that the warehousing has been performed, the process proceeds to step S405c. In step S405c, the date and time when the warehousing is performed, the image of the difference area in the image after warehousing (warehouse article image), and information indicating that the warehousing has been performed are supplied to the writing control unit 172, and the writing control unit 172 Are associated with each other and generated as inventory information and stored in the inventory information storage unit 174. After step S405c, the process proceeds to step S421.

  In step S421, the article recognizing unit 170 tries to specify the article shown as the stored article image from the stored article image (supplied from the difference area extracting unit), and determines whether or not the article has been specified. Do. If the article cannot be specified, the process proceeds to step S407. If it can be identified, the process proceeds to step S422.

  In step S422, the writing control unit 172 determines whether or not an article of the same type as the received article is stored in the inventory information storage unit 174. If there is the same article, the process proceeds to step S423. If there is no same article, the process proceeds to step S424.

  Since the process proceeds to step S423 when an article has been received and an article of the same type has already been stored in the inventory information storage unit 174, it is determined that an article of the same type as the article in the warehouse has been added. it can. Therefore, in step S423, the writing control unit 172 increases the number of articles of the corresponding type stored in the inventory information storage unit 174, and adds the warehousing date and time. Here, the image of the corresponding article (the incoming article image) may be replaced with a new incoming article image.

  The process proceeds to step S424 when the article has been received and the same type of article has not been stored in the inventory information storage unit 174. Therefore, it can be determined that an article not in the store has been newly received. Therefore, in step S424, the writing control unit 172 adds a new article type and a stored article image to the inventory information storage unit 174, sets the number of articles to 1, and adds the storage date and time.

When it is determined in step S404 that the article has not been delivered, the process proceeds to step S407. If it is determined in step S404 that the goods have been delivered, the process proceeds to step S406c.
In step S406c, the date and time when the delivery is performed, the image at the position corresponding to the difference area (the delivery article image) in the pre-shipping image, and information indicating that the delivery has been performed are supplied to the writing control unit 172, and the writing is performed. The control unit 172 associates these pieces of information with each other, generates stock information, and stores the stock information in the stock information storage unit 174.

After step S406c, the process proceeds to step S431. In step S431, the article recognizing unit 170 attempts to specify the delivered article and determines whether or not the article has been specified. If the article cannot be specified, the process proceeds to step S407. If it can be specified, the process proceeds to step S432.
In step S432, the writing control unit 172 determines whether or not the same article as the delivered article is in the warehouse. If there is the same article, the process proceeds to step S433. If there is no same article, the process proceeds to step S407 without doing anything.

  The process proceeds to step S433 when an article has been delivered and an article of the same type has already been stored in the inventory information storage unit 174, so that it can be determined that one article that was in the warehouse has been taken out. Therefore, in step S433, the writing control unit 172 decreases the number of corresponding articles, and deletes information indicating the date and time of receipt of the delivered articles. Here, when the number of articles before the decrease is one, it indicates that the corresponding articles disappear from the warehouse, so the type of the corresponding article is deleted from the inventory information storage unit 174.

  If NO in step S432, that is, if an article has been issued and an article of the same type is not stored in the inventory information storage unit 174, an image that does not exist has been issued, so the write control unit 172 estimates that there was an error in the determination and does nothing.

In step S407, it is determined whether or not there is an unprocessed difference area (whether there is a difference area that has not undergone the processes of steps S403, S404, S405c, S406c, S421 to S424, and S431 to S433). If there is a difference area, the process returns to step S402.
If there is no unprocessed difference area in step S407 (NO in S407), the inventory information update process is terminated.
Thereafter, similarly to the first embodiment, the process proceeds to step S308 in FIG. 10, and a transition to the stationary state ST3 is performed.

By performing such processing, it is possible to store in the inventory information storage unit 174 how many types of articles are in the warehouse.
FIG. 19 shows a display example 503 on the portable terminal 500 based on data created by the internal information distribution unit 136. By using the information stored in the inventory information storage unit 174, as shown in FIG. 19, the user can grasp which items and how many items are in the storage.

  As described above, according to the present embodiment, inventory management is performed by comparing the type of an article determined to have been received or delivered with the type of an article that has been received in the past. It is possible to grasp what kind of articles are present and how long.

  In addition, since the stock information storage unit 174 stores the types of articles existing in the warehouse, the user can search for articles existing in the warehouse using character strings representing the types of articles (not images). It is possible to cooperate with various services such as suggesting to the user a dish that can be created from the type of article existing in the warehouse.

  In the present embodiment, no action is taken when the article type cannot be acquired as a result of the article recognition process. For example, the article image is presented to the user, and the article is presented to the user. The input of the type may be prompted. By doing so, the user's effort is increased, but the inventory information stored in the inventory information storage unit 174 can be made more accurate.

  In addition, since determination of entry and exit and identification of an article are not necessarily successful every time, it is desirable that a user can correct information stored in the inventory information storage unit 174 using a mobile terminal or the like.

  In the first to fourth embodiments, the in-house management apparatus is integrally provided with a camera and all processes are performed in the apparatus installed in the storage 10, but the processes may be performed in different places. For example, the internal management apparatus 100 may perform only image acquisition and internal image transmission to an external server, and image recognition, internal information management, and internal information distribution may be performed by an external server. In this case, the internal management apparatus of this invention is comprised with the apparatus installed in the warehouse, and an external server. By using the external server in this way, it is possible to reduce the computing capacity and storage capacity necessary for the device installed in the storage.

  Although the present invention has been described as an in-house management apparatus, the in-house management method implemented by the above-mentioned in-house management apparatus also forms part of the present invention. Moreover, a part or all of the above-mentioned in-store management device can be realized by software, that is, by a programmed computer. Accordingly, a program for causing a computer to execute the processing in the above-mentioned storage management device or the processing in each step of the above-mentioned storage management method, and a computer-readable recording medium recording such a program are also included in the present invention. Part of

  10 storage, 12 main body, 14 door, 100, 100a, 100b, 100c storage management device, 110 storage image acquisition unit, 112 camera, 114 buffer, 122 storage image storage unit, 124 pre-operation image storage unit, 126 Pre-door closing image storage unit, 130 Difference area extraction unit, 132, 162, 172 Write control unit, 134, 164, 174 Inventory information storage unit, 136 In-box information distribution unit, 140 control unit, 142 Open / close determination unit, 144 Stationary A determination unit, 145 work determination unit, 146 operation mode determination unit, 148 entry / exit determination unit, 150 proximity sensor, 160 identical article determination unit, 170 article recognition unit, 171 article database.

Claims (16)

An in-compartment image acquisition unit that repeats the process of acquiring the in-container image obtained by imaging the inside of the storage;
A work determination unit that determines whether or not work for at least one of warehousing, unloading, and movement of articles is performed in the storage;
A pre-operation image storage unit that stores the internal image obtained by imaging the interior of the storage immediately before the start of the operation as the pre-operation image when the operation is started;
When the work is finished, immediately after the work is finished, the inside image obtained by imaging the inside of the storage is set as a post-work image, and from the pre-work image and the post-work image, A difference area extraction unit for extracting a difference area between the two images;
A warehouse management apparatus comprising: a warehouse entry / exit determination unit that determines whether an article has been entered or exited from the pre-work image or the post-work image and the difference area.   The warehouse management apparatus according to claim 1, wherein the work determination unit determines whether or not work is being performed in the warehouse based on whether or not the image in the warehouse is temporally changed. . Further comprising an internal image storage unit for storing the internal image acquired by the internal image acquisition unit,
The work determination unit reads the internal image stored in the internal image storage unit when the internal image acquisition unit acquires the internal image next time, and the internal image acquisition unit acquires the internal image Determining that the operation is not performed when the amount of change of the image in the warehouse with respect to the image in the warehouse read from the image storage unit is equal to or less than a predetermined threshold value. The warehouse management apparatus according to claim 2.   The work determination unit is configured to obtain the in-store image acquired by the in-store image acquisition unit and the in-store image read from the in-store image storage unit according to a direction in which an article is put in and out of the storage. The in-chamber management device according to claim 3, wherein the amount of change is calculated by assigning a weight to each unit. A proximity sensor for detecting the proximity of the user's hand inserted into the storage;
The warehouse management device according to claim 1, wherein the work determination unit determines whether the work is being performed based on whether the proximity is detected by the proximity sensor.   The entry / exit determination unit, when there is an image part having a contour whose similarity with the contour of the difference region is equal to or higher than a predetermined threshold at the same position as the difference region in the pre-work image, It is determined that the article corresponding to the image portion has disappeared from the position in the storage box corresponding to the position of the difference area of the pre-operation image, and based on the determination result, the article corresponding to the image portion 6. The warehouse management apparatus according to any one of claims 1 to 5, wherein it is determined whether or not delivery has been performed.   The entry / exit determination unit is configured such that when there is no image part in the post-operation image having an outline whose similarity to the outline of the image part corresponding to the article determined to have disappeared is greater than or equal to a threshold value, The warehouse management apparatus according to claim 6, wherein it is determined that the cargo has been delivered.   When the image entry / exit determination unit has an image part having a contour whose similarity is equal to or higher than a threshold value of the contour of the image part corresponding to the article determined to have disappeared, at a different position in the post-work image, 8. The warehouse management apparatus according to claim 6 or 7, wherein the article is determined to have moved in the warehouse.   When the entry / exit determination unit includes an image part having a contour whose similarity with the contour of the difference region is equal to or higher than a predetermined threshold at the same position as the difference region in the post-work image, It is determined that the article corresponding to the image portion has appeared at a position in the storage box corresponding to the position of the difference area of the post-work image, and based on the determination result, the article corresponding to the image portion It determines whether warehousing was performed or not, The management apparatus in a warehouse as described in any one of Claim 1 to 8 characterized by the above-mentioned.   The entry / exit determination unit is configured such that when there is no image part having an outline whose similarity with the outline of the image part corresponding to the article determined to have appeared in the pre-operation image, the article is The warehouse management apparatus according to claim 9, wherein it is determined that the warehouse has been received.   When the image entry / exit determination unit has an image part having an outline whose similarity is equal to or higher than a threshold value of the outline of the image part corresponding to the article determined to have appeared, at a different position in the pre-operation image, The warehouse management apparatus according to claim 9 or 10, wherein it is determined that the article has moved in the warehouse. An inventory information storage unit that stores, as inventory information, information related to an article that has been determined to have been entered or exited by the entry / exit determination unit;
The in-store management device according to any one of claims 1 to 11, further comprising: an in-store information distribution unit that distributes the stock information stored in the stock information storage unit to the outside. An open / close determining unit for determining an open / closed state of the door of the storage;
A pre-door closing image storage unit that stores the internal image stored in the pre-work image storage unit as the pre-door closing image immediately before the door is closed;
The in-chamber management device according to claim 12, wherein the in-chamber information distribution unit distributes the image before door closing in addition to the inventory information stored in the inventory information storage unit. The stock information storage unit stores an image of an article that is determined to have been received by the storage / determination determination unit,
An identical article determination unit that determines whether an image having a similarity with an image of an article determined to have been issued by the entry / exit determination unit is greater than or equal to a predetermined threshold is stored in the inventory information storage unit; In addition,
When it is determined by the same article determination unit that an image whose similarity with the image of the article determined to have been delivered is equal to or greater than a predetermined threshold is stored in the inventory information storage unit, The warehouse management apparatus according to claim 12 or 13, wherein information relating to an article corresponding to the image is deleted from the inventory information storage unit. An article recognizing unit for identifying the type of the article from the image of the difference area corresponding to the article determined to have been entered or exited by the entry / exit determination unit;
The inventory management apparatus according to claim 12 or 13, wherein the inventory information storage unit stores the number of articles whose type is specified by the article recognition unit and the same type of article as the article. An image acquisition step in the warehouse that repeats the process of acquiring the image in the warehouse obtained by imaging the interior of the storage;
An operation determination step for determining whether or not an operation for at least one of warehousing, unloading, and movement of articles is performed in the storage;
A storage step of storing, in the pre-operation image storage unit, the internal image obtained by imaging the interior of the storage immediately before the start of the operation as the pre-operation image when the operation is started;
When the work is finished, immediately after the work is finished, the inside image obtained by imaging the inside of the storage is set as a post-work image, and from the pre-work image and the post-work image, A difference area extraction step for extracting a difference area between the two images;
A storage management method for storage, comprising: a storage / determination determination step for determining whether an article has been stored or discharged from the pre-work image or the post-work image and the difference area.

Images