JPWO2020245965A1 - Abnormal behavior detection device, article storage, abnormal behavior detection method, and abnormal behavior detection program - Google Patents

Abstract

The abnormal behavior detection device (1) is housed in the storage room (102) based on an image obtained by an image pickup device (110) that images the inside of the storage room (102) of the article storage (100). Acquire the classification unit (11) for classifying one or more articles into one or more groups (H1 to Hm) and the number of articles (J1 to Jm) belonging to each of one or more groups (H1 to Hm). The behavior of the subject who uses the article storage (100) includes abnormal behavior based on the result of comparison between the management unit (12) and the number (J1 to Jm) and the predetermined threshold (TH_CNT). It has a determination unit (13) for determining whether or not it is.

Description

The present invention relates to an abnormal behavior detection device, an article storage, an abnormal behavior detection method, and an abnormal behavior detection program.

Proposal of a system for determining whether or not a subject has a disease based on the distance traveled by the subject during a predetermined time and the frequency of use of home appliances used by the subject during that period. There is. See, for example, Patent Document 1.

Japanese Unexamined Patent Publication No. 2003-153868

Oncel Tuzel, Fatih Porikli, Peter Meer, "Region Covariance: A Fast Descriptor for Detection and Classification", European Commission

However, the system proposed in Patent Document 1 is supposed to detect wandering, which is a symptom at a stage where dementia has progressed considerably, and is caused by "forgetfulness", which is a symptom at an early stage of dementia. It is not possible to detect abnormal behavior.

The present invention has been made to solve the above-mentioned problems, and is an abnormal behavior detection device, an article storage, an abnormal behavior detection method, and an abnormal behavior that enable early detection of abnormal behavior of a subject. The purpose is to provide a detection program.

The abnormal behavior detection device according to one aspect of the present invention includes one or more articles housed in the storage room based on an image obtained by an image pickup device that images the inside of the storage room of the article storage. The article storage is based on the result of comparison between the classification unit that classifies into the above groups, the management unit that acquires the number of articles belonging to each of the one or more groups, and the number and a predetermined threshold. It is characterized by having a determination unit for determining whether or not the behavior of the subject who uses the warehouse includes abnormal behavior.

In the method for detecting abnormal behavior according to another aspect of the present invention, one or more articles housed in the storage room are captured based on an image obtained by an imaging device that images the inside of the storage room of the article storage. Based on the results of a step of classifying into one or more groups, a step of acquiring the number of articles belonging to each of the one or more groups, and a comparison between the number and a predetermined threshold value, the article storage It is characterized by having a step of determining whether or not the behavior of the subject who uses the above includes abnormal behavior.

According to the present invention, it is possible to detect an abnormal behavior of a subject who uses the article storage at an early stage.

It is the schematic which shows the refrigerator as the article storage which has the abnormal behavior detection device which concerns on Embodiment 1 of this invention. FIG. 5 is a functional block diagram schematically showing a configuration of an abnormal behavior detection device according to the first embodiment. It is a figure which shows the example of the warehousing / delivery status information of the article storage which concerns on Embodiment 1. FIG. It is a figure which shows the example of the hardware composition of the abnormal behavior detection apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the example of the operation of the abnormal behavior detection apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows other example of operation of the abnormal behavior detection apparatus which concerns on Embodiment 1. FIG. FIG. 5 is a functional block diagram schematically showing a configuration of an abnormal behavior detection device according to a modified example of the first embodiment. It is a figure which shows the example of the warehousing / delivery status information of the article storage which concerns on Embodiment 2 of this invention. It is a figure which shows the example of the time-series change of the warehousing / delivery situation of the article storage which concerns on Embodiment 3 of this invention. It is a functional block diagram which shows schematic structure of the abnormal behavior detection apparatus which concerns on Embodiment 4 of this invention.

Hereinafter, the abnormal behavior detection device, the article storage having the abnormal behavior detection device, the abnormal behavior detection method, and the abnormal behavior detection program according to the embodiment of the present invention will be described with reference to the drawings. The following embodiments are merely examples, and various modifications can be made within the scope of the present invention.

Embodiment 1.
FIG. 1 is a schematic view showing the configuration of an article storage having the abnormal behavior detection device 1 according to the first embodiment. In the first embodiment, the article storage is a refrigerator 100 that stores articles such as food while cooling them. FIG. 1 shows a state in which the door 101 of the refrigerator 100 is opened. FIG. 1 shows a state in which beer, jam, butter, yogurt, and eggs are stored inside (that is, inside) the storage chamber 102 of the refrigerator 100. Beer, jam, butter, yogurt and eggs are specific examples of goods. Further, in FIG. 1, "A", "B", "C", "D", "E", "F", and "G" are trade names. For example, "A beer" indicates the article "beer" of the trade name "A". "B jam" indicates the article "jam" of the trade name "B". The article storage is not limited to the refrigerator as long as it is a storage used by the target person who is the target of detecting abnormal behavior. The article storage may be, for example, a storage for storing food at room temperature.

Inside the door 101 of the refrigerator 100, an image pickup device 110 capable of taking an image of the inside of the refrigerator is provided. Further, the imaging device 110 may be installed so that when the door 101 is opened, a person who has opened the door 101 can be imaged. The image pickup device 110 takes an image of the inside of the refrigerator when the door 101 is closed. The image pickup apparatus 110 may be provided at a position other than the inside of the door 101. Further, the number of image pickup devices 110 is not limited to one, and may be two or more. The abnormal behavior detection device 1 and the image pickup device 110 are connected so as to be able to communicate with each other by wire or wirelessly. The abnormal behavior detection device 1 and the image pickup device 110 may be communicably connected via a router, a server, a network, or the like. When a plurality of image pickup devices 110 are connected to the abnormal behavior detection device 1, the classification unit 11 acquires a plurality of images from the plurality of image pickup devices 110. Alternatively, the classification unit 11 may acquire one composite image formed by performing a process of combining a plurality of images in the vertical direction, the horizontal direction, or both of these directions. Alternatively, the classification unit 11 may obtain one composite image formed by performing image processing on a plurality of images and performing a process of combining the plurality of image-processed images.

The image pickup apparatus 110 is, for example, a camera having an optical lens, an optical filter, an image sensor, and the like. The image pickup apparatus 110 may have a processor that controls the image sensor. The image (that is, image data) output from the image pickup apparatus 110 may be either a color image or a monochrome image. The color image may use any of RGB method, YCbCr method, HSV method, and the like as the color expression method.

Further, the refrigerator 100 includes an open / close detector 111 that detects the opening / closing of the door 101. The abnormal behavior detection device 1 may determine the presence or absence of abnormal behavior by using the opening / closing signal of the door 101 detected by the opening / closing detector 111.

FIG. 2 is a functional block diagram schematically showing the configuration of the abnormal behavior detection device 1 according to the first embodiment. The abnormal behavior detection device 1 is a device capable of implementing the abnormal behavior detection method according to the first embodiment. As shown in FIG. 2, the abnormal behavior detection device 1 has a classification unit 11, a management unit 12, a determination unit 13, and a database unit 14, which is a storage unit that stores a database. The management unit 12 has a storage unit 15 and a time acquisition unit 16 for acquiring time information. The database unit 14 may be provided in another device outside the abnormal behavior detection device 1. Further, the storage unit 15 and the time acquisition unit 16 may be provided in another device outside the management unit 12 or outside the abnormal behavior detection device 1.

The sorting unit 11 detects one or more articles housed in the refrigerator based on the image obtained by the image pickup device 110 that images the inside of the refrigerator 100 as the goods storage, and the detected articles. Are classified into one or more groups H1 to Hm. Groups H1 to Hm mean m groups. m is an integer greater than or equal to 1. "Classifying into one or more groups" means classifying into one or more categories.

The classification unit 11 classifies the articles into groups H1 to Hm, for example, based on the attributes of the articles stored in the refrigerator. Specifically, the classification unit 11 detects the articles included in the image acquired by the image pickup apparatus 110, and classifies the articles into groups H1 to Hm based on the types of the detected articles. In the example of FIG. 1, five types of articles, beer, jam, butter, yogurt, and eggs, are stored in the refrigerator. For example, when the articles are classified based on the attributes, the classification unit 11 classifies the articles into five groups H1 to H5 of beer, jam, butter, yogurt, and eggs.

Further, the classification unit 11 detects the article included in the image acquired by the image pickup apparatus 110, detects the product name of the detected article, and based on the product name of the article stored in the refrigerator, the article May be classified into groups H1 to Hm. In the example of FIG. 1, there are seven types of products in the refrigerator: "A beer", "B jam", "C jam", "D butter", "E yogurt", "F egg", and "G egg". Is housed. For example, the classification unit 11 classifies the articles stored in the refrigerator into seven groups H1 to H7 corresponding to the trade names A to G.

The classification unit 11 can detect an article contained in an image, for example, by using a sliding window method or by using machine learning such as deep learning.

The sliding window method is a method of searching for an article by sliding a sliding window, which is a rectangular area having a predetermined window size (wc, hc), over the entire area of an image. Here, wc indicates the number of pixels in the horizontal direction (that is, the horizontal direction) of the sliding window, and hc indicates the number of pixels in the vertical direction (that is, the vertical direction). In the sliding window method, it is desirable to use sliding windows of a plurality of window sizes (wc, hc). For example, the plurality of window sizes (wc, hc) used are as follows.
(Wc, hc) = (16,16), (32,32), (48,48), (64,64)
The reason for using a plurality of window sizes is that the articles housed in the refrigerator are placed in a position closer to the door 101 or in a position farther from the door 101. This is because they are housed in various states, and the size of the article in the image differs depending on the positional relationship between the image pickup device 110 and the article.

The classification unit 11 calculates the feature amount for each sliding window, calculates the similarity between the calculated feature amount and the reference feature amount which is the feature amount of the article to be detected calculated in advance, and uses this similarity as the feature amount. Detect articles based on. The reference feature amount is information stored in advance in the database unit 14. The database unit 14 may be provided in an external storage device capable of communicating with the abnormal behavior detection device 1, a server on the network, and the like. “Detecting an article” includes, for example, specifying the attribute of the article or specifying the product name of the article. A method for detecting an article using a sliding window is disclosed in, for example, Non-Patent Document 1 described above. The classification unit 11 using this detection method calculates the feature amount in the image based on the brightness, the color, the first derivative of the luminance, the second derivative of the luminance, and the like for each sliding window which is a rectangular region in the image. The distance Dist of the feature amount calculated from the two rectangular regions is obtained. Since the distance Dist is a value indicating the distance of the feature amount calculated from the two rectangular regions, the reciprocal of the distance Dist indicates the similarity R of the feature amount calculated from the two rectangular regions. Therefore, the similarity R is calculated by, for example, the following equation (1).
R = 1 / Dist (1)

In other words, the classification unit 11 obtains the feature amount for each rectangular region Recct which is a sliding window. Here, c = 0, ..., K (k is a positive integer). In the sliding window method, a plurality of rectangular areas Rect0 to Reckk are used as the rectangular areas for which the feature amount is required. Then, the classification unit 11 obtains the similarity R _p of the reference feature amount is a feature amount of the article to be detected which is calculated in advance and these characteristic amounts. Classifying unit 11, the threshold value D _R is larger than the article of similarity R _p is degree of similarity, i.e., detects the article having a feature amount satisfying the formula (2) below, as an article contained in the rectangular area Rectc.
R _{p> D} R (2)
Here, the threshold value D _R of similarity is greater than 0. This process is for ensuring robustness against noise. Finally, the classification unit 11 outputs the article name of the detected article.

The management unit 12 acquires the number of articles J1 to Jm belonging to each of the groups H1 to Hm acquired by the classification unit 11. The management unit 12 stores the warehousing / delivery status information including the acquired number J1 to Jm in the storage unit 15. The storage unit 15 may be provided outside the management unit 12. The storage unit 15 may be provided outside the abnormal behavior detection device 1. The storage unit 15 may be provided in a server that is communicably connected to the communication device of the abnormal behavior detection device 1 (for example, shown in FIG. 4 described later) via a network.

The determination unit 13 determines whether the behavior of the subject using the refrigerator 100 includes abnormal behavior based on the result of comparison between the number of articles J1 to Jm acquired by the management unit 12 and the predetermined threshold value TH_CNT. Judge whether or not. The predetermined threshold value TH_CNT may be defined as, for example, the threshold value TH_CNT1 to TH_CNTm for each of the plurality of groups H1 to Hm.

The determination unit 13 determines that the behavior of the subject includes an abnormal behavior, for example, when the number of articles J1 to Jm exceeds a predetermined threshold value TH_CNT. Further, when the number of articles J1 to Jm is equal to or less than a predetermined threshold value TH_CNT, the determination unit 13 determines that the behavior of the subject does not include the abnormal behavior. Alternatively, when the number of articles J1 to Jm exceeds the predetermined threshold value TH_CNT exceeds the predetermined reference number (also referred to as “first reference number”) TH_REPEAT, the determination unit 13 determines. It may be determined that the behavior of the subject includes abnormal behavior.

FIG. 3 is a diagram showing an example of warehousing / delivery status information f1 of the refrigerator 100 according to the first embodiment. As shown in FIG. 3, the management unit 12 stores the warehousing / delivery status information f1 indicating the warehousing / delivery status of the goods for each group H1 to Hm of the classified goods. For example, the management unit 12 stores the information f1a that identifies the articles housed in the refrigerator and the number information f1b of each article as the warehousing / delivery status information f1. In the example shown in FIG. 3, the information f1a that identifies the article housed in the refrigerator indicates the name of the classified group. For example, the number of articles "A beer" whose product name is "A" and whose article name is "beer" is two. The number of articles "B jam" whose product name is "B" and whose article name is "jam" is seven.

When the number of articles J1 to Jm stored in the refrigerator exceeds a predetermined threshold value TH_CNT, the determination unit 13 determines that the behavior of the subject includes an abnormal behavior, and the behavior of the subject. Outputs information indicating that is containing abnormal behavior. For example, the determination unit 13 can notify the target person or a predetermined related person (for example, the target person's family) of information indicating that the target person's behavior includes abnormal behavior. ..

FIG. 4 is a diagram showing an example of the hardware configuration of the abnormal behavior detection device 1 according to the first embodiment. The abnormal behavior detection device 1 shown in FIG. 4 uses a memory 92 as a storage device for storing a program as software and a processor 91 as an information processing unit for executing a program stored in the memory 92 (for example,). , By computer). The program is installed, for example, via a network or from a recording medium on which information is recorded. The program stored in the memory 92 includes the abnormal behavior detection program according to the first embodiment. Further, the abnormal behavior detection device 1 has a storage 93 which is a storage device. The abnormal behavior detection device 1 in FIG. 2 corresponds to a processor 91 that executes a program. A part of the abnormal behavior detection device 1 shown in FIG. 2 may be realized by the memory 92 shown in FIG. 4 and the processor 91 that executes the program.

Next, the process for the abnormal behavior detection device 1 to determine whether or not the behavior of the target person includes the abnormal behavior will be described. FIG. 5 is a flowchart showing an example of the operation of the abnormal behavior detection device 1.

First, the classification unit 11 determines whether or not the execution condition of the process for determining the presence or absence of abnormal behavior is satisfied (step S10). When it is set to determine the presence or absence of abnormal behavior at regular time intervals, the classification unit 11 satisfies the execution condition when a predetermined time has elapsed from the previous process. Judge that there is. Further, when the door 101 is set to determine the presence or absence of abnormal behavior when the door 101 is closed, the classification unit 11 receives an execution condition when the closing signal of the door 101 is received from the open / close detector 111. Is satisfied.

When the execution condition is satisfied, the classification unit 11 acquires an image from the image pickup device 110 that images the inside of the refrigerator (step S11). The classification unit 11 detects the article based on the acquired image, and outputs the names of the groups H1 to Hm to which the detected article belongs to the management unit 12 (step 12). In the examples shown in FIGS. 1 and 3, the names of groups H1 to Hm are, for example, "D butter", "E egg", and the like.

The management unit 12 acquires the names of the groups H1 to Hm to which the articles belong from the classification unit 11, calculates the number of the same articles J1 to Jm belonging to the same group, and generates the warehousing / delivery status information f1 including these. Store (step S13).

The determination unit 13 acquires the warehousing / delivery status information f1 from the management unit 12 and determines whether or not the behavior of the target person includes an abnormal behavior. When the determination unit 13 determines that the behavior of the target person includes an abnormal behavior, the determination unit 13 outputs information indicating that the behavior of the target person includes the abnormal behavior.

For example, when the determination unit 13 acquires the warehousing / delivery status information f1 shown in FIG. 3 from the management unit 12 and sets four as the threshold value TH_CNT, the determination unit 13 has seven “B jams” as shown in FIG. Since it is in the warehousing state, it is determined that the behavior of the target person includes abnormal behavior (step S14), and information indicating that the behavior of the target person includes abnormal behavior is output (step S15). ..

As described above, if the abnormal behavior detection device 1 is used, the abnormal behavior of the subject can be detected at an early stage before being examined by a specialist, and the behavior of the subject becomes abnormal to the subject or related persons. Can be informed that it contains. As a result, countermeasures can be taken at an early stage.

Further, in the first embodiment, the determination unit 13 can set different threshold values TH_CNT1 to TH_CNTm for each group H1 to Hm as the threshold value TH_CNT. For example, the determination unit 13 can set the threshold value TH_CNT to 4 packs for "F egg" and 6 threshold values TH_CNT for "A beer". By doing so, it is possible to set a threshold value TH_CNT1 to TH_CNTm suitable for each group H1 to Hm.

FIG. 6 is a flowchart showing another example of the operation of the abnormal behavior detection device according to the first embodiment. In FIG. 6, the same processing as that shown in FIG. 5 is assigned the same reference numerals as those shown in FIG. The operation of FIG. 6 differs from the operation of FIG. 5 in that it includes step S14a. In step S14a of FIG. 6, the determination unit 13 determines whether or not the number of times K of the number of articles J1 to Jm belonging to the groups H1 to Hm exceeds the threshold value TH_CNT (or TH_CNT1 to TH_CNTm) exceeds the reference number of times TH_REPEAT. do. When each number of articles J1 to Jm exceeds the threshold value TH_CNT (or TH_CNT1 to TH_CNTm) is expressed by K1 to Km, the number of times K may be the total value of the number of times K1 to Km. Alternatively, the number of times K may be the number of times K1 to Km in which the number of articles J1 to Jm exceeds the threshold value TH_CNT (or TH_CNT1 to TH_CNTm) in any of the groups H1 to Hm.

When the number of times K in which the number of articles J1 to Jm exceeds the threshold value TH_CNT (or TH_CNT1 to TH_CNTm) exceeds the reference number of times TH_REPEAT, the determination unit 13 determines that the behavior of the subject includes abnormal behavior. The determination unit 13 outputs information indicating that the behavior of the subject includes abnormal behavior. If the detection operation shown in FIG. 6 is adopted, the accuracy of the determination can be improved.

Modification example of the first embodiment 1.
A notification unit 17 for transmitting the detection result determined by the determination unit 13 to a pre-registered e-mail address or a pre-associated application via a wireless or wired network can be further provided. By doing so, it is possible to inform related persons such as family members that the behavior of the subject includes abnormal behavior.

FIG. 7 is a functional block diagram schematically showing the configuration of the abnormal behavior detection device 2 according to the first modification of the first embodiment. In FIG. 7, components that are the same as or correspond to the components shown in FIG. 2 are designated by the same reference numerals as those shown in FIG. The abnormal behavior detection device 2 shown in FIG. 7 differs from the abnormal behavior detection device 1 shown in FIG. 2 in that the notification unit 17 is provided.

The determination unit 13 determines whether or not the target person's behavior includes abnormal behavior, and when it is determined that the target person's behavior includes abnormal behavior, the determination unit 13 outputs a signal indicating that to the notification unit 17. do. The notification unit 17 acquires a signal from the determination unit 13 indicating that the behavior of the target person includes an abnormal behavior, and transmits information indicating this to a pre-registered destination. For example, the notification unit 17 transmits notification information including information indicating that the behavior of the target person includes abnormal behavior to the e-mail address or the application associated in advance. The notification information transmitted is, for example, an e-mail message. This message may merely notify that the subject's behavior includes abnormal behavior, or may include the time when the abnormal behavior occurs and the outline of the detected abnormal behavior.

Modification example of the first embodiment 2.
The abnormal behavior detection device of the second modification has the same configuration as that of FIGS. 1 to 7. In the abnormal behavior detection device of the second modification, the classification unit 11 acquires information about an article purchased at a store or via the Internet by, for example, a communication device 94 (shown in FIG. 4), and an image captured by the image pickup device 110. The articles housed in the refrigerator are classified into one or more groups H1 to Hm based on the above. Further, the management unit 12 sets the threshold value TH_CNT (that is, any one of TH_CNT1 to TH_CNTm) for the group to which the most recently purchased article belongs to a small value.

For example, if two packs of "eggs" were purchased one week before, which is not the latest time, the behavior of the subject to purchase an additional two packs of "eggs" one week later today is normal. It is considered to be an action. Therefore, the management unit 12 sets the threshold value TH_CNT (that is, any of TH_CNT1 to TH_CNTm) for the group to which the egg belongs to, for example, 4 packs, which is a normal value.

However, if the subject purchased 3 packs of "eggs" one day before the most recent time, the subject does not need to purchase an additional 3 packs of "eggs" one day later today, or this. Behavior is considered to be unusual. Therefore, the management unit 12 sets the threshold value TH_CNT (that is, any of TH_CNT1 to TH_CNTm) for the group to which the egg belongs to 2 packs, which is a value smaller than the normal threshold value of 4 packs. Then, the management unit 12 increases the threshold value TH_CNT (that is, any one of TH_CNT1 to TH_CNTm) as the date and time elapses. In this way, by changing the threshold value TH_CNT using the purchase history of the article, the accuracy of detecting abnormal behavior can be improved.

Embodiment 2.
Since the abnormal behavior detection device 1 according to the first embodiment does not use the position information of the article housed in the refrigerator, a situation may occur in which the movement of the article in the refrigerator is determined to be the receipt of the article. On the other hand, the abnormal behavior detection device according to the second embodiment detects and stores the position information of the article housed in the refrigerator, and by using this, the movement of the article in the refrigerator is stored in the article. Can not be determined. Except for this point, the abnormal behavior detection device according to the second embodiment is the same as the abnormal behavior detection device 1 according to the first embodiment. Therefore, in the second embodiment, FIGS. 1 to 7 are also referred to.

FIG. 8 is a diagram showing an example of warehousing / delivery status information f2 of the article storage 100 according to the second embodiment. In the second embodiment, the classification unit 11 detects the article included in the image and outputs the area information of the detected article and the name of the article. The area information of the article includes the coordinates (xc, yc) of the left end position of the rectangular area Reccc of the article detected in the image and the size (wc, hc) of the rectangular area Reccc.

The management unit 12 stores the warehousing / delivery status information f2, which indicates the warehousing / delivery status of the goods for each article. For example, as shown in FIG. 8, the management unit 12 stores the article name f2a of each article, the area information f2b of the article, and the warehousing time information f2c of the article as the warehousing / delivery status information f2.

The warehousing time stored by the management unit 12 as the warehousing / delivery status information f2 may be a time input by the target person using an operation unit such as a button provided on another device. When the abnormal behavior detection device 1 is a computer connected to a refrigerator, the time is set by an RTC (real time clock) composed of a clock built in the computer or an integrated circuit in which the function is implemented. May be obtained from.

The determination unit 13 repeatedly purchases articles belonging to the same group H1 to Hm during a predetermined time, and as a result, the number of articles belonging to groups H1 to Hm J1 to Jm is a predetermined threshold value. When the increase is larger, it is determined that the subject's behavior includes abnormal behavior, and information indicating that is output.

Next, the process for the abnormal behavior detection device 1 to determine whether or not the behavior of the target person includes the abnormal behavior will be described. The classification unit 11 classifies one or more articles into one or more groups H1 to Hm from the acquired image, and indicates the area information f2b of the detected articles and the classified groups H1 to Hm (for example, "" "A beer", "B jam", etc.) f2a and the warehousing time information f2c are output to the management unit 12. The management unit 12 generates and stores the warehousing / delivery status information f2 including the area information f2b of the article, the information f2a indicating the classified groups H1 to Hm, and the warehousing time information f2c from the classification unit 11.

Next, the classification unit 11 acquires an image from the image pickup device 110, detects an article, and classifies the detected article into one or more groups H1 to Hm. The management unit 12 determines whether or not the article included in the newly acquired image is the same as the article included in the images acquired up to the previous time.

The management unit 12 refers to the area information f2b of the article included in the warehousing / delivery status information f2 for the images acquired up to the previous time and the information f2a indicating the groups H1 to Hm to which the article belongs, and these are detected this time. Check whether it matches the one based on the warehousing / delivery status information f2. The management unit 12 can determine whether or not the article included in the newly acquired image is the same as the article included in the images acquired up to the previous time. If they match, the management unit 12 determines that it is not the warehousing of a new article, and does not add the warehousing time information f2c to the warehousing / delivery status information f2. If they do not match, the management unit 12 determines that the goods are warehousing, and adds warehousing time information f2c and the like to the warehousing / delivery status information f2.

Further, the management unit 12 refers to the area information f2b of the article included in the warehousing / delivery status information f2 stored in the management unit 12 and the information f2a indicating the groups H1 to Hm to which the article belongs, and the article detected this time. It is examined whether or not the area information f2b of the above matches the information f2a indicating the groups H1 to Hm to which the article belongs. Even if the area information f2b does not completely match, the management unit 12 sets the article as long as the difference between the area information f2b and the position of the area information of the article detected this time is within a predetermined allowable range. Judge that it is not a new item. Then, the management unit 12 does not add the information about the article determined not to be a new article to the warehousing / delivery status information f2. For example, the management unit 12 provides an allowable range of ± 5 pxel in the x direction or the y direction with respect to the coordinates (xc, yc) at the left end position of the rectangular area in the area information of the article, and at the same time, the area information of the article. A tolerance of ± 5% may be provided for each of the width and height of the rectangular size of. In this case, the management unit 12 can determine that the article is not a new article, that is, a slight movement of the article, even if there is a difference within the permissible range. By doing so, the management unit 12 avoids erroneously determining an article that has moved slightly when the door is opened or closed or when trying to pick up another article as a newly detected article. can.

When the management unit 12 determines that the article is not a new article, the management unit 12 may update the area information f2b of the article in the warehousing / delivery status information f2. By doing so, the management unit 12 can update the area information f2b to the position on the image where the article currently exists.

The determination unit 13 acquires the warehousing / delivery status information f2 from the management unit 12 and determines whether or not the behavior of the target person includes an abnormal behavior. When it is determined that the behavior of the subject includes an abnormal behavior, the determination unit 13 outputs information indicating that the behavior of the subject includes the abnormal behavior.

For example, the determination unit 13 acquires the warehousing / delivery status information f2 from the management unit 12, and repeatedly purchases articles belonging to the same group H1 to Hm within a predetermined time, and as a result, the number J1 to 1 When Jm is increased, it is determined that the behavior of the subject includes abnormal behavior. When the number J1 to Jm increases, for example, when there is an increase exceeding a predetermined threshold value, as in the case of the first embodiment.

As described above, if the abnormal behavior detection device according to the second embodiment is used, the abnormal behavior of the subject can be detected at an early stage before being examined by a specialist, and the subject or a related person can be notified of the subject. Can inform that the behavior of is including abnormal behavior. As a result, countermeasures can be taken at an early stage.

Further, in the abnormal behavior detection device according to the second embodiment, as in the case of the first embodiment, the determination unit 13 determines in advance as a threshold value for determining whether the behavior of the target person includes the abnormal behavior. The number of times (also referred to as "second reference number of times") TH_REPEAT2 is further provided, and the same group of goods are repeatedly purchased within a predetermined time, and as a result, the number of times the number J1 to Jm increases is the number of times. When the predetermined reference number of times TH_REPEAT2 is exceeded, it can be determined that the behavior is abnormal. By doing so, it is possible to detect when abnormal behavior due to a disease or the like occurs repeatedly.

Embodiment 3.
The abnormal behavior detection device according to the first and second embodiments determines that a new article has been received when the article stored in the refrigerator is temporarily taken out and returned to the original position in a short time. .. For example, when the jam stored in the refrigerator is taken out and the jam is returned to the refrigerator after a few minutes. On the other hand, the abnormal behavior detection device according to the third embodiment can not determine that the operation of temporarily taking out the article housed in the refrigerator and returning it is the receipt of a new article. Except for this point, the abnormal behavior detection device according to the third embodiment is the same as the abnormal behavior detection device according to the first and second embodiments. Therefore, in the second embodiment, FIGS. 1 to 7 are also referred to.

The management unit 12 stores the warehousing / delivery status information indicating the warehousing / delivery status of the goods for each article. For example, the management unit 12 stores the article name of each article, the area information of the article, the warehousing time of the article, and the warehousing time of the article as warehousing / delivery status information.

When the article detected in the previously acquired image cannot be detected in the newly acquired image, the management unit 12 determines that the article has been delivered and stores the delivery time. That is, the management unit 12 sets the warehousing time without deleting the information from the warehousing / delivery status information for the goods that are already included in the warehousing / delivery status information and could not be detected by the newly acquired image. Add and memorize.

The management unit 12 retains the information about the goods whose warehousing time is stored in the warehousing / delivery status information only for a preset period. The management unit 12 deletes both the warehousing information and the warehousing information related to the goods when the set period has passed, counting from the warehousing time.

When the warehousing / delivery status information does not store the information indicating the warehousing, and only the information representing the warehousing is stored, the management unit 12 continues to store the information.

When the management unit 12 tries to store the warehousing / delivery status information exceeding a preset threshold value in the memory area for storing the warehousing / delivery status information, the management unit 12 deletes the oldest warehousing / delivery status information in order. May be set to.

When the determination unit 13 repeatedly purchases the same articles of the same group H1 to Hm within a predetermined time, and as a result, the number of articles J1 to Jm belonging to the group H1 to Hm increases. In addition, it is determined that the behavior of the subject includes abnormal behavior, and information indicating that is output. However, in the determination unit 13, after the number J1 to Jm decreases during a predetermined time, a situation occurs in which the number J1 to Jm increases by the same number as the decreased number, or such a situation occurs. If it occurs repeatedly, it is not determined that the number J1 to Jm is increasing.

For example, in the time-series warehousing / delivery situation shown schematically in FIG. 9, the warehousing at time t2 and the warehousing at time t3 occur in a short period of 3 minutes. As described above, as a situation in which warehousing and warehousing occur in order in a short time, for example, a situation in which "B jam" is taken out from the refrigerator 100 for breakfast and returned to the refrigerator 100 after use can be considered. That is, time t3 does not correspond to an increase in the number of articles J1 to Jm due to additional purchase. Therefore, in the determination unit 13, after the number J1 to Jm decreases during a predetermined time (also referred to as “second time”), the number J1 to Jm increases by the same number as the decreased number. When a situation occurs, or when such a situation occurs repeatedly, it is not determined that the number J1 to Jm is increasing.

As described above, if the abnormal behavior detection device according to the third embodiment is used, the abnormal behavior of the subject can be detected at an early stage before being examined by a specialist, and the subject or a related person can be notified of the subject. Can inform that the behavior of is including abnormal behavior. As a result, countermeasures can be taken at an early stage.

Further, since the situation in which the article is taken out from the refrigerator 100 to consume a part and then returned to the original state is not used for determining the presence or absence of abnormal behavior, the accuracy of the determination can be improved.

Embodiment 4.
FIG. 10 is a functional block diagram schematically showing the configuration of the abnormal behavior detection device 4 according to the fourth embodiment. In FIG. 10, components that are the same as or correspond to the components shown in FIG. 2 are designated by the same reference numerals as those shown in FIG. The abnormal behavior detection device 4 according to the fourth embodiment is different from the abnormal behavior detection device 4 according to the first, second, and third embodiments in that it has a person recognition unit 18.

The person recognition unit 18 detects a person from the image captured by the image pickup device, identifies which individual the person is, and outputs the identified result to the management unit 12a. As a method of recognizing a person, a method of identifying from the similarity of feature amounts of images, which is generally used, can be used.

The management unit 12a further stores information that can identify the person who has received the goods in the warehousing / delivery status information.

The determination unit 13 acquires the warehousing / delivery status information from the management unit 12a, and the same person repeatedly purchases the same group of goods from the warehousing time and the person information of each goods within a predetermined time, and as a result. When the number J1 to Jm increases, it is determined that the behavior of the subject includes abnormal behavior.

As described above, if the abnormal behavior detection device 4 according to the fourth embodiment is used, the abnormal behavior of the target person can be detected at an early stage before being examined by a specialist, and the target person or related persons can be targeted. It can inform that a person's behavior includes abnormal behavior. As a result, countermeasures can be taken at an early stage.

Further, even when the refrigerator 100 is used by a plurality of people, it is possible to detect that the behavior of the subject includes abnormal behavior. As a result, the accuracy of determination can be improved.

Modification example.
It is possible to appropriately combine the configurations of the abnormal behavior detection devices according to the first to fourth embodiments.

1, 2, 4 Abnormal behavior detection device, 11 Classification unit, 12, 12a Management unit, 13 Judgment unit, 14 Database unit, 15 Storage unit, 16 Time acquisition unit, 17 Notification unit, 18 Person recognition unit, 91 Processor, 92 Memory, 93 storage, 94 communication device, 100 refrigerator, 101 door, 102 storage room, 110 image pickup device, 111 open / close detector.

The abnormal behavior detection device according to one aspect of the present invention includes one or more articles housed in the storage room based on an image obtained by an image pickup device that images the inside of the storage room of the article storage. The article storage is based on the results of comparison between the classification unit that classifies into the above groups, the management unit that acquires the number of articles belonging to each of the one or more groups, and the number and a predetermined threshold. a determination section for determining whether or not behavior of the subject contains an abnormal behavior of using the refrigerator, have a, the determining unit, the number of times that the number exceeds the predetermined threshold is predetermined When the number of times exceeds the first reference number, it is determined that the behavior of the subject includes abnormal behavior .

The abnormal behavior detection method according to another aspect of the present invention is a method executed by the abnormal behavior detection device, and is based on an image obtained by an imaging device that images the inside of the storage chamber of the article storage. A step of classifying one or more articles housed in the above into one or more groups, a step of acquiring the number of articles belonging to each of the one or more groups, and the number and a predetermined threshold. This is a determination step for determining whether or not the behavior of the subject who uses the article storage includes abnormal behavior based on the result of the comparison, and is the number of times the number exceeds the predetermined threshold. Is characterized by having the determination step of determining that the behavior of the subject includes an abnormal behavior when the number of times exceeds a predetermined first reference number of times.

Claims (15)

A classification unit that classifies one or more articles housed in the storage room into one or more groups based on an image obtained by an image pickup device that images the inside of the storage room of the goods storage.
A management unit that acquires the number of articles belonging to each of the one or more groups, and
Based on the result of comparison between the number and a predetermined threshold value, a determination unit for determining whether or not the behavior of the subject who uses the article storage includes abnormal behavior, and a determination unit.
An abnormal behavior detection device characterized by having. The abnormal behavior detection device according to claim 1, wherein the classification unit classifies the one or more articles into the one or more groups based on the respective attributes of the one or more articles. The abnormal behavior detection device according to claim 1, wherein the classification unit classifies the one or more articles into the one or more groups based on the trade name of each of the one or more articles. .. The abnormal behavior detection device according to any one of claims 1 to 3, wherein the predetermined threshold value is set for each of the one or more groups. Any one of claims 1 to 4, wherein the determination unit determines that the behavior of the subject includes an abnormal behavior when the number exceeds the predetermined threshold value. Abnormal behavior detection device described in. The determination unit determines that the behavior of the subject includes an abnormal behavior when the number of times the number exceeds the predetermined threshold value exceeds the predetermined first reference number of times. The abnormal behavior detection device according to any one of claims 1 to 4, wherein the abnormal behavior detection device is characterized. It also has a time acquisition unit that acquires time information.
The management unit acquires the warehousing time, which is the time when the goods are warehousing in the goods storage.
The determination unit is a second article that belongs to the first group that is the same group as the first article that has been warehousing during the first predetermined time from the warehousing time. Is further stored, and when the number of articles belonging to the first group housed in the article storage increases, it is determined that the behavior of the subject includes abnormal behavior. The abnormal behavior detection device according to any one of claims 1 to 5. It also has a time acquisition unit that acquires time information.
The management unit acquires the warehousing time, which is the time when the goods are warehousing in the goods storage.
The determination unit is a second article that belongs to the first group that is the same group as the first article that has been warehousing during the first predetermined time from the warehousing time. When the number of times the number of articles belonging to the first group, which is stored in the article storage, increases, exceeds the predetermined second reference number. The abnormal behavior detection device according to any one of claims 1 to 5, wherein it is determined that the behavior of the subject includes abnormal behavior. It also has a time acquisition unit that acquires time information.
The management unit acquires the delivery time, which is the time when the goods are delivered into the goods storage.
The determination unit is a fourth article that belongs to a second group that is the same group as the third article that has been delivered during the second time predetermined from the delivery time. However, the abnormal behavior detection device according to any one of claims 1 to 8, wherein it is not determined that the number of the items is increasing when the number of the items is the same as that of the third item. .. A person recognition unit that identifies the target person and outputs person information indicating the target person is further provided.
The determination unit repeatedly purchases articles belonging to the same group by the same person within a predetermined time from the warehousing time of each article stored by the management unit and the person information, and belongs to the same group. The abnormal behavior detection device according to any one of claims 1 to 9, wherein when the number of articles increases, the behavior of the subject is determined to be abnormal behavior. The abnormal behavior detection device according to any one of claims 1 to 10, further comprising a notification unit that transmits the result determined by the determination unit to a pre-registered destination. The classification unit has recently purchased an article when detecting a group of articles housed in the containment chamber from an image captured by the image pickup device from information about a group of articles purchased at a store or via the Internet. The abnormal behavior detection device according to any one of claims 1 to 11, wherein the threshold value used at the time of detection is lowered. An article storage facility comprising the abnormal behavior detection device according to any one of claims 1 to 12. A step of classifying one or more articles housed in the storage room into one or more groups based on an image obtained by an imaging device that images the inside of the storage room of the goods storage.
The step of obtaining the number of articles belonging to each of the one or more groups, and
Based on the result of comparison between the number and a predetermined threshold value, a step of determining whether or not the behavior of the subject using the article storage includes abnormal behavior, and
An abnormal behavior detection method characterized by having. A process of classifying one or more articles housed in the storage room into one or more groups based on an image obtained by an imaging device that images the inside of the storage room of the goods storage.
The process of acquiring the number of articles belonging to each of the one or more groups, and
Based on the result of comparison between the number and a predetermined threshold value, a process of determining whether or not the behavior of the subject who uses the article storage includes an abnormal behavior, and
An abnormal behavior detection program characterized by having a computer execute.

Images