JP6905850B2 - Image processing system, imaging device, learning model creation method, information processing device - Google Patents

Description

The present invention relates to an image processing system, an imaging device, a learning model creation method, and an information processing device.

There is known a monitoring method in which an image is captured and recorded by an imaging device such as a surveillance camera, and a person concerned later confirms the image to confirm the presence or absence of an abnormality and what kind of abnormality has occurred. However, as a person concerned, there are cases where the video cannot be confirmed without noticing the necessity of confirming the video, and the presence or absence of an abnormality can only be confirmed from the video after the fact. In addition, the presence or absence of an abnormality cannot be confirmed unless the person concerned sees all the images.

Therefore, a technique for analyzing an image in real time to detect an abnormality has been devised (see, for example, Patent Document 1). Anomaly detection refers to detecting a phenomenon that deviates from the normal state that can occur on a daily basis. Patent Document 1 discloses a method of suppressing congestion by determining a suspicious person before collecting an image captured by a camera and transmitting only the target image.

However, Patent Document 1 is a technique for determining an abnormality when a resident other than a pre-registered resident is detected, and if the registered contents are not appropriate, the abnormality detection result may not be accurate.

In response to such inconvenience, a technique has been devised in which an information processing device learns various normal images by machine learning and abnormally detects an abnormality from an image captured at a place where a camera is installed (for example, a patent document). See 2.). Patent Document 2 discloses a method of selecting information suitable for learning by having a learning executor or a learning agent further select information from the information narrowed down by the online learning unit.

Japanese Unexamined Patent Publication No. 2013-222216 Japanese Unexamined Patent Publication No. 2016-173682

However, in the conventional technology, since it is necessary for the newly installed camera to learn a normal image at the installation location in order to detect an abnormality, there is a problem that it is difficult to start the abnormality determination immediately after the installation. was there. For example, when an image processing system uses a machine-learned learning model to detect a suspicious person, a learning model that matches the scene captured by the camera is required, but the newly installed camera has an image used for learning. Since there is no data, the abnormality cannot be determined until the images used for learning are accumulated.

In view of the above problems, an object of the present invention is to provide an image processing system capable of detecting an abnormality at an early stage after installation.

The present invention is an image processing system that detects anomalies from an image captured by an image pickup device, and has similar features to an image acquisition means that acquires the image from a plurality of first image pickup devices installed at different installation locations. A classification means for classifying the images to be classified into clusters, a learning means for constructing an abnormality determination means for learning the images classified into the clusters and detecting an abnormality for each cluster, and the first imaging device. Determines the cluster in which the image of the first imaging device similar to the image captured by the second imaging device installed at a different installation location is classified, and is constructed from the image classified into the cluster. The abnormality determining means determined by the determining means has a determining means for determining the abnormality determining means, and determines the presence or absence of an abnormality from the image captured by the second imaging device .

According to the present invention, it is possible to provide an image processing system capable of detecting an abnormality at an early stage after installation.

This is an example of a diagram for explaining image learning and abnormality detection by machine learning. It is an example of a figure explaining the outline of image processing. This is an example of a system configuration diagram of an image processing system. This is an example of a hardware configuration diagram of a camera device. It is a block diagram which showed the schematic hardware configuration of an information processing apparatus. This is an example of a functional block diagram illustrating each function of the image processing system. This is an example of a flow chart showing a clustering processing procedure. A flow chart showing the procedure for the cluster setting unit to extract local features and global features is shown. It is a figure which shows typically the classification by the K-means method. It is an example of the figure explaining clustering. It is an example of the figure explaining the degree of separation R. It is an example of the figure explaining the learning method of SAE. It is a figure which shows typically the learning model which determines the presence or absence of an abnormality from an image. This is an example of a diagram for explaining the learning procedure of FIG. 13 in more detail. This is an example of a functional block diagram of an information processing device related to abnormality determination. It is an example of the figure schematically explaining the abnormality detection of the existing installation place. It is an example of the figure schematically explaining the abnormality detection of a new installation place. This is an example of a diagram schematically explaining the processing performed by the determination result processing unit. This is an example of a diagram for explaining a configuration when the camera device determines an abnormality.

Hereinafter, as an example of the embodiment for carrying out the present invention, the image processing system and the learning model creation method performed by the image processing system will be described with reference to the drawings.

<Specific example of abnormality detection>
In explaining the present embodiment, a specific example of abnormality detection will be described. Abnormality refers to a phenomenon that deviates from normality that can occur on a daily basis.

FIG. 1A is an example of a diagram illustrating normal learning of the image 6 by machine learning.
(1) A large number of images 6 (normal images 6) containing no abnormalities are prepared. It is assumed that this image 6 is a moving image. Not including anomalies means that the content in the picture is not subject to the alarm. Whether it is normal or not depends on the location where the camera is installed and the object in the picture. For example, in the image 6 in which a pedestrian is shown, the image in which the pedestrian walks is a normal image, and the image 6 in which the pedestrian gets over a wall is abnormal. Further, in the image 6 in which the car is shown, the image 6 in which a person gets on and off, the car stops, parks, and starts is a normal image 6, and the image 6 in which the action of releasing the door lock is shown is abnormal. .. In the case of a bicycle, as with a car, different images can be obtained depending on whether it is normal or abnormal.
(2) The learning unit 34 uses machine learning such as deep learning or SVM (support vector machine) to learn the range in which the shape and movement of the object shown in the normal image 6 are normal, and constructs a learning model. .. The learning model estimates this normal range, and outputs whether or not the shape and movement of the input object are normal.
(3) An example of a learning model will be described. For example, in the image 6 in which a pedestrian is shown, the background is stationary while the pedestrian moves, so the learning unit 34 automatically generates the posture, movement speed, and the like of the pedestrian. Then, a normal range such as the posture and moving speed of the pedestrian is estimated. It should be noted that the learning unit 34 does not need a teacher signal such as a pedestrian, and does not need to recognize that the moving object is a pedestrian (person). The learning unit 34 merely quantifies the normal behavior pattern (here, the moving speed) of a certain shape (here, a pedestrian) as such.

FIG. 1B is an example of a diagram illustrating an abnormality determination performed by the identification device 5.
(4) An image 6 in which a person climbs over a wall is imaged as an abnormal image 6. People rarely get over the wall and are not included in normal image 6.
(5) The identification device 5 determines an abnormality by comparing it with the learning model created in (3).
(6) For example, it is determined that the degree of agreement between the postures of people is 20% and the degree of agreement between the movement speeds is 60%. If the degree of agreement with the normal image 6 is low, the identification device 5 determines that there is a high possibility of abnormality.

In this way, by learning a large amount of normal images 6 by the learning unit 34, it is possible to construct an identification device that determines that the image 6 is abnormal as in the case of a human determination. Therefore, the image processing system of the present embodiment can detect an abnormality in real time before an intruder invades the facility and dispatch a guard to take appropriate measures.

However, on the other hand, the presence or absence of an abnormality cannot be determined until a large amount of normal images 6 are learned. As described above, whether or not the image 6 is normal depends on the place where the camera is installed and the object in which the camera is captured. Therefore, the normal image 6 used for learning by another camera can be used for learning the newly installed camera. Even if it is diverted as it is, an appropriate learning model is not always constructed.

<Outline of image processing of this embodiment>
Therefore, in the image processing system of the present embodiment, images 6 collected from a large number of existing installation locations are clustered and learned in similar scenes, and a learning model in which the scene is similar to the image 6 in the new installation location is newly developed. Used for the learning model of the installation location. Since the learning model of the existing installation location can be used, monitoring can be started immediately after the camera is installed in the new installation location.

FIG. 2 is an example of a diagram illustrating an outline of image processing of the present embodiment. Hereinafter, the preparation phase before the installation of the camera device and the abnormality determination phase based on the image 6 captured by the new camera device will be described separately.

-Preparation phase (1) There are already a plurality of existing installation locations 8a, 8b, 8c, 8d, etc. of the camera device 10. When distinguishing a plurality of existing installation locations 8, the reference numeral 8 will be described by adding an alphabet (the number of existing installation locations is not particularly limited). The image 6 captured by these camera devices 10 (an example of the first image pickup device) is stored in the image storage unit 32. The image storage unit 32 stores the images 6 captured by the camera devices 10 at all the existing installation locations 8.
(2) The cluster setting unit 33 clusters the accumulated images 6 for each similar scene. By clustering, for example, images in which the location of the camera device 10 and the objects in the image are similar are classified into the same cluster.
(3) Machine learning unit 1-n
The machine learning units 1 to n machine-learn the clustered normal images 6. In FIG. 2, there are as many machine learning units 1 to n as the number of clusters (n), but the number of machine learning units 1 to n may be at least one. The machine learning units 1 to n each build a learning model. Each learning model is used by the abnormality determination unit 36.
(4) Next, the cluster determination unit 35 determines which cluster the image 6 captured by the camera device 10 (an example of the second imaging device) of the new installation location 7 is classified into the image 6 of the existing installation location 8. Determine if it is classified and decide to apply the learning model of the cluster to which it is classified. In this way, it can be determined immediately after the appropriate learning model is installed in the newly installed camera device 10.

-Abnormality determination phase (5) The abnormality determination unit 36 corresponding to the camera device 10 at the existing installation location 8 determines whether the image 6 is normal or abnormal using the learning model. For example, a learning model suitable for the existing installation location 8a is automatically determined by the clustering result of the image 6 of the existing installation location 8a. The fact that the learning model of the machine learning unit 1 is provided to the existing installation locations 8a and 8b in FIG. 2 indicates that the images 6 of the existing installation locations 8a and 8b are clustered in the same cluster.
(6) The abnormality determination unit 36 corresponding to the camera device 10 at the new installation location 7 also determines whether the image 6 is normal or abnormal using the learning model determined in (4).
(7) The determination result processing unit 37 acquires the determination result of the presence or absence of an abnormality from each abnormality determination unit 36, gives an instruction to dispatch a guard, and transmits the image 6 to the customer.

In this way, the image 6 of the newly installed camera device 10 is clustered into a cluster similar to the image of the existing installation location 8, so that the learning model learned from the image 6 of the camera device 10 of the existing installation location 8 Is available, so that it is possible to determine the presence or absence of an abnormality immediately after the new camera device 10 is installed. Further, with respect to the existing installation location 8, there is a possibility that a learning model more appropriate than the learning model learned from the image of one installation location can be created.

<Terminology>
A cluster means a group, and in the present embodiment, it is a group of images having similar scenes. In addition to clusters, they may be referred to as classes, groups, categories, and so on.

Learning refers to extracting useful rules, rules, knowledge representations, judgment criteria, etc. from data in order to realize functions similar to those of human learning ability on a computer.

<System configuration example>
FIG. 3 shows an example of a system configuration diagram of the image processing system 100. The image processing system 100 mainly includes a camera device 10 and an information processing device 30 connected via a network N. One or more camera devices 10 are installed in one existing installation location 8. “Existing” means an installation location where a sufficient number of images 6 have already been captured for learning. In addition, one or more camera devices 10 are installed at the new installation location 7. The new installation location 7 is a location where the camera device 10 has not been installed in the past and the camera device 10 is newly installed.

The camera device 10 is an imaging device that captures several or more images 6 per second, and can capture so-called moving images. The existing installation location or new installation location is mainly a location where the intrusion of a third party can be monitored. For example, the area around a facility, the area around a private residence, a parking lot, a forest, a seaside area, an exclusion zone, etc. are included, but are not limited to these. In addition to being installed outdoors, it may be installed indoors such as facilities, offices, private residences, hotels, stores, laboratories, warehouses, and factories.

It is preferable that the camera device 10 has not only an image capturing function but also an image processing function for detecting a moving object, but it is not essential in the present embodiment. The camera device 10 captures several or more images 6 per second and transmits them to the monitoring center.

The network N is constructed by a telephone line laid at the installation location of the camera device 10, a LAN, a provider network of a provider that connects the LAN to the Internet, a line provided by a line operator, and the like. When the network N has a plurality of LANs, the network N is called a WAN and may include the Internet. The network N may be constructed either wired or wireless, and may be a combination of wired and wireless. Further, when the camera device 10 is connected to a wireless public line network such as 3G, 4G, 5G, LTE (Long Term Revolution), it can be connected to the provider network without going through a wired telephone line or LAN. ..

An information processing device 30 is installed in the monitoring center 9. The information processing device 30 builds a learning model and performs abnormality determination. The determination result of abnormality is transmitted to the customer's mobile terminal or the observer's terminal together with the image 6. The customer or the guard confirms the image 6 determined to be abnormal, and dispatches a guard to the installation site or reports to the police if necessary.

The information processing device 30 is a device called a general computer, a PC (Personal Computer), a server, or the like. Although one information processing device 30 is shown in FIG. 3 for convenience of explanation, the information processing device 30 is not limited to one, and a plurality of information processing devices 30 share the functions described in the present embodiment. You may have.

<Hardware configuration example>
<< Hardware configuration example of camera device >>
FIG. 4 is an example of a hardware configuration diagram of the camera device 10. The camera device 10 includes an image pickup unit 101, an image processing IC 103, a ROM 105, a CPU 106, a RAM 107, and a communication device 108.

The image pickup unit 101 is a camera having a lens, an aperture, a shutter (mechanical / electronic), a solid-state image sensor such as CMOS or CCD, a memory, and the like. The image 6 may be black and white or color. The imaging unit 101 periodically images a predetermined range under the control of the CPU 106, and sends the image 6 to the image processing IC 103. The image processing IC 103 is an integrated circuit that performs image processing on the image 6 to detect a moving object.

The CPU 106 uses the RAM 107 as a working memory to execute a program stored in the ROM 105 and controls the entire camera device 10. That is, the imaging by the imaging unit 101 is controlled. Further, the communication device 108 is controlled to transmit the image 6 to the monitoring center 9.

The communication device 108 is called a network interface or an Ethernet (registered trademark) card, and provides a function of connecting to a network. It may be connected to a wireless LAN access point or a base station of a mobile phone network.

<< Hardware configuration example of information processing device 30 >>
FIG. 5 is a block diagram showing a schematic hardware configuration of the information processing device 30. The information processing device 30 can be implemented as a personal computer, a workstation, or an appliance server. The information processing device 30 includes a CPU 201 and a memory 202 that enables high-speed access to data used by the CPU 201. The CPU 201 and the memory 202 are connected to other devices or drivers of the information processing device 30, such as the graphics driver 204 and the network driver (NIC) 205, via the system bus 203. ..

The LCD (display device) 206 is connected to the graphics driver 204 to monitor the processing result by the CPU 201. A touch panel may be integrally arranged on the LCD 206. In this case, the user can operate the information processing device 30 by using a finger as an operation means.

Further, the network driver 205 connects the information processing device 30 to the network N at the transport layer level and the physical layer level to establish a session with the camera device 10 and the like.

An I / O bus bridge 207 is further connected to the system bus 203. A storage device such as HDD 209 is connected to the downstream side of the I / O bus bridge 207 by IDE, ATA, ATAPI, serial ATA, SCSI, USB, etc. via an I / O bus 208 such as PCI. .. You may have SSD (Solid State Drive) instead of HDD 209 or together with HDD 209.

The HDD 209 stores a program 209p that controls the entire information processing apparatus 30. The information processing device 30 receives and learns the operation of the observer by executing the program 209p. The program 209p may be distributed from a server that distributes the program, or may be distributed in a state of being stored in a portable storage medium such as a USB memory or an optical storage medium.

Further, an input device 210 such as a keyboard and a mouse (called a pointing device) is connected to the I / O bus 208 via a bus such as USB, and receives inputs and commands by the operator.

The information processing device 30 may support cloud computing. Cloud computing refers to a usage pattern in which resources on a network are used without being aware of specific hardware resources. In this case, the hardware configuration shown in FIG. 5 does not need to be housed in one housing or provided as a group of devices, and it is preferable that the information processing device 30 includes the hardware configuration. Indicates an element.

<Example of functional configuration of image processing system 100>
FIG. 6 is an example of a functional block diagram illustrating each function included in the image processing system 100. Note that FIG. 6 mainly describes the functions related to the construction of the learning model. Further, it is assumed that the functions of the camera device 10 are common or different at each installation location, but there is no problem in the explanation of the present embodiment. Therefore, FIG. 6 shows only one camera device 10.

<< Functional configuration of camera device >>
The camera device 10 includes an image acquisition unit 11, an image processing unit 12, and a communication unit 13. Each of these functions is a function or means realized by the CPU 106 shown in FIG. 4 executing a program and cooperating with the hardware of the camera device 10. It is not necessary to clearly distinguish the functions realized by hardware or software, and some or all of these functions may be realized by a hardware circuit such as an IC.

Further, the camera device 10 has a storage unit 19 constructed by the ROM or RAM 107 shown in FIG. An image DB 191 is constructed in the storage unit 19. The image DB 191 does not have to be directly possessed by the camera device 10, and may be located at any location on the network accessible to the camera device 10.

The image acquisition unit 11 periodically acquires (images) the image 6 regardless of the presence or absence of an abnormality. A moving image can be captured by capturing images at sufficiently short time intervals. The imaging interval does not necessarily have to be constant, and the imaging interval may be changed depending on the time zone, the moving object detection result, or the like. The image acquisition unit 11 is realized by the CPU 106 of FIG. 4 executing a program to control the image pickup unit 101 and the like.

The image processing unit 12 performs image processing effective for clustering. For example, it is determined whether the imaging time is daytime or nighttime from the average brightness of the images 6 and the imaging time, and a label is attached to each image 6. Alternatively, processing such as noise removal, trimming, and high resolution may be performed. The image processing unit 12 stores the captured image 6 in the image DB 191. In the image DB 191, the old image 6 is overwritten and a constant new image 6 is always held. The image processing unit 12 is realized by the CPU 106 of FIG. 4 executing a program or the like.

The communication unit 13 transmits the image 6 stored in the image DB 191 to the information processing device 30. In order to guarantee the real-time property of abnormality detection, it is preferable that the stored image 6 is immediately transmitted to the information processing apparatus 30. Therefore, the communication unit 13 continuously transmits the images 6 to the information processing device 30, but the images 6 may be collectively transmitted after a certain amount of the images 6 are accumulated. The communication unit 13 is realized by the CPU 106 of FIG. 4 executing a program to control the communication device 108 and the like.

<< Functional configuration of information processing device >>
The information processing device 30 includes a communication unit 31, an image storage unit 32, a cluster setting unit 33, a learning unit 34, and a cluster determination unit 35. Each of these functions is a function or means realized by the CPU 201 shown in FIG. 5 executing the program 209p and cooperating with the hardware of the information processing apparatus 30. It is not necessary to clearly distinguish the functions realized by hardware or software, and some or all of these functions may be realized by a hardware circuit such as an IC.

Further, the information processing device 30 has a storage unit 39 constructed from the memory 202 or HDD 209 shown in FIG. An image storage DB 391 and a learning model DB 392 are constructed in the storage unit 39. These DBs do not have to be directly possessed by the information processing device 30, and may be provided at any location on the network accessible to the information processing device 30.

The communication unit 31 receives the image 6 from the camera device 10. The communication unit 31 is realized by the CPU 201 shown in FIG. 5 executing the program 209p to control the network driver 205 and the like.

The image storage unit 32 associates the image 6 received by the communication unit 31 with the identification information of the camera device 10 (or the identification information of the installation location if there is only one camera device 10 at the installation location), and the image storage DB 391. Remember in. The image storage unit 32 is realized by the CPU 201 shown in FIG. 5 executing the program 209p or the like.

The cluster setting unit 33 extracts a feature amount from the image 6 stored in the image storage DB 391, and clusters the image 6 transmitted from each camera device 10 into clusters having similar scenes using the feature amount. The scene can be said to be a scene in a broad sense in which the image 6 is captured. For example, it is highly possible that the images of the camera device 10 installed in the parking lot are similar in the images 6 of different parking lots. When clustering, determine the best number of clusters. Further, since the characteristics tend to be different between daytime and nighttime, there is a high possibility that the clusters of image 6 of the same camera device 10 will be separated into daytime and nighttime. Furthermore, the above label can be used to ensure that clusters can be clustered separately during the day and at night. The cluster setting unit 33 is realized by the CPU 201 shown in FIG. 5 executing the program 209p or the like.

The learning unit 34 further includes a machine learning unit 1, a machine learning unit 2, ..., And a machine learning unit n (n is a natural number). 1 to n of the machine learning units 1 to n correspond to the number of clusters (n) formed by clustering. However, one machine learning unit (for example, the machine learning unit 1) may learn the image 6 of all the clusters, and the machine learning units 1 to n may be one. The machine learning unit 1 stores the learning model C1 constructed by learning in the learning model DB 392, the machine learning unit 2 stores the learning model C2 constructed by learning in the learning model DB 392, and the machine learning unit n stores the learning constructed by learning. The model Cn is stored in the learning model DB 392. In the following, for convenience of explanation, it may be explained that the "learning unit 34" simply constructs the learning model. The learning unit 34 is realized by the CPU 201 shown in FIG. 5 executing the program 209p or the like.

The cluster determination unit 35 determines the cluster in which the image 6 captured by the camera device 10 at the new installation location 7 is clustered among the clusters generated by the cluster setting unit 33. For example, select the cluster with the closest features. The correspondence between the camera device 10 and the learning model (cluster) is registered in the learning model DB 392. Even when the camera device 10 does not label daytime or nighttime, it is preferable to determine the corresponding clusters in daytime and nighttime by utilizing the fact that the feature amounts are different between daytime and nighttime. It should be noted that the information processing apparatus 30 can also determine whether the image 6 is captured during the daytime or at night. The cluster determination unit 35 is realized by the CPU 201 shown in FIG. 5 executing the program 209p or the like.

Further, the cluster determination unit 35 determines a cluster in which the image 6 captured by the camera device 10 at the existing installation location 8a is clustered. In this case as well, it is preferable that the cluster determination unit 35 determines the corresponding clusters during the daytime and at nighttime.

表１は、学習モデルＤＢ３９２に格納される情報を模式的に示す。学習モデルＤＢ３９２は、クラスタＩＤに対応付けて、学習モデル、カメラＩＤ、及び、日中／夜間の各項目を有する。クラスタＩＤは、最終的にクラスタリングされた各クラスタを特定する情報である。ＩＤはIdentificationの略であり識別子や識別情報という意味である。ＩＤは複数の対象から、ある特定の対象を一意的に区別するために用いられる名称、符号、文字列、数値又はこれらのうち１つ以上の組み合わせをいう。他のＩＤについても同様である。カメラＩＤは各カメラ装置１０を特定する情報であり、学習モデルと対応付けられている。日中／夜間の項目は各カメラ装置１０の日中の画像６と夜間の画像６のそれぞれがどの学習モデルと最も近いかを示す。これにより、既存設置場所８及び新規設置場所７の各カメラ装置１０が撮像した画像６をどの学習モデルで異常判定すればよいか決定される。

Table 1 schematically shows the information stored in the learning model DB 392. The learning model DB 392 has a learning model, a camera ID, and daytime / nighttime items in association with the cluster ID. The cluster ID is information that identifies each cluster that is finally clustered. ID is an abbreviation for Identification and means an identifier or identification information. An ID refers to a name, a code, a character string, a numerical value, or a combination of one or more of these, which is used to uniquely distinguish a specific object from a plurality of objects. The same applies to other IDs. The camera ID is information that identifies each camera device 10, and is associated with the learning model. The daytime / nighttime item indicates which training model each of the daytime image 6 and the nighttime image 6 of each camera device 10 is closest to. As a result, it is determined which learning model should be used to determine the abnormality of the image 6 captured by each camera device 10 of the existing installation location 8 and the new installation location 7.

<Necessity of clustering>
Explain the need for clustering. First, as a premise, the type of installation location where the camera device 10 is installed is limited, and the normal behavior (behavior pattern of a pedestrian or the like) that can be considered at that location may also be limited. Table 2 shows an example of normal behavior at each installation location.

表２に示すように、設置場所が駐車場、家の玄関、建物外周のそれぞれで正常行動は幾つかに限定される。このため、画像６をクラスタリングすることで特徴が類似した画像を集めることができ、類似した画像から学習モデルを構築することで、異常判定の精度を向上できると期待できる。

As shown in Table 2, the normal behavior is limited to some in each of the parking lot, the entrance of the house, and the outer circumference of the building. Therefore, by clustering the images 6, images having similar characteristics can be collected, and by constructing a learning model from the similar images, it can be expected that the accuracy of abnormality determination can be improved.

<Clustering processing>
Next, clustering will be described in detail with reference to FIG. 7. FIG. 7 is an example of a flow chart showing a clustering processing procedure. Hereinafter, each step of FIG. 7 will be described in order.

(S1)
The cluster setting unit 33 acquires a background image from which noise is removed from each image 6. For removing noise, for example, a known smoothing filter may be used.

(S2)
The cluster setting unit 33 extracts local features and global features from the background image. The local features and the global features will be described with reference to Table 3.

表３は、局所的特徴量と大域的特徴量の利用目的と判定できないことがそれぞれまとめられたものである。局所的特徴量では画像６の一部（局所）の特徴が得られるが、シーンを分類することが得意でない。例えば、車が何台かあっても道路なのか駐車場なのかの特徴が得られない。大域的特徴量はベランダなのか駐車場なのかという画像全体の特徴が得られるが、駐車場と住宅のガレージを分類できない可能性がある。

Table 3 summarizes the facts that cannot be determined as the purpose of use of the local feature amount and the global feature amount. With the local feature amount, a part (local) feature of the image 6 can be obtained, but the scene is not good at classifying. For example, even if there are several cars, it is not possible to obtain the characteristics of whether it is a road or a parking lot. Although it is possible to obtain the characteristics of the entire image as to whether the global features are balconies or parking lots, it may not be possible to classify parking lots and residential garages.

In this way, since the local features and the global features are in a complementary relationship, it is effective to extract both the local features and the global features in order to properly classify the scene.

FIG. 8 shows a flow chart showing a procedure in which the cluster setting unit 33 extracts the local feature amount and the global feature amount.

S2-1: First, the cluster setting unit 33 detects the local feature amount. In this embodiment, SIFT (Scale-Invariant Feature Transform) features are acquired as an example of local features. The SIFT feature quantity is a feature quantity that is robust against lighting changes, rotation, enlargement, and reduction using a scale space, and a plurality of 128-dimensional feature points can be acquired from one image 6. The cluster setting unit 33 first detects a key point. The key point is that it appears characteristically on the image 6 even if the scale changes. For detection, a Gaussian filter having a different resolution is applied stepwise to create an image 6 having a different resolution, and an extreme value of a DoG (Difference of Gaussian) image is searched. The key point is the central part of the local area obtained by the search.

S2-2: Next, the cluster setting unit 33 detects the strength of the concentration gradient and the direction of the gradient with respect to the vicinity of the key point (extracts features). The gradient strength and gradient of each key point are SIFT features. The cluster setting unit 33 calculates SIFT features for all the images 6 to be classified.

S2-3: Next, the cluster setting unit 33 classifies the local features for all the images 6 to be classified. All images 6 are all images 6 of the existing installation location 8.

FIG. 9A schematically shows the classification by the K-means method. FIG. 9A shows each feature point in three dimensions, but actually has a dimension of 128 dimensions (SIFT feature amount) × number of feature points obtained from one image × number of images. One point in FIG. 9A represents one feature point. In the K-means method, each feature point is classified into each cluster with an arbitrarily given number of clusters. First, the number of clusters k is calculated by the following formula.
Number of clusters k = √ {128 dimensions (SIFT feature amount) x number of feature points obtained from one image x number of images}
Next, the cluster setting unit 33 classifies all the images 6 into clusters having this number of clusters. The initial center of gravity of the cluster is randomly given, each feature point is assigned to the cluster with the closest center of gravity, the center of gravity is recalculated, and the process of assigning each feature point to the cluster with the closest center of gravity is performed again. Repeat until it is gone. FIG. 9B schematically shows each feature point classified into the three clusters 71.

S2-4: Next, the cluster setting unit 33 is visually
Quantize by determining words. That is, the center (center of gravity) of each cluster is determined to be visual words 72. As a result, k visual words, which is the number of clusters, are specified. FIG. 9C schematically shows three visual words 72 when k = 3 as an example. The number of visual words 72 is the same as the number of clusters, and the number of dimensions of each visual words 72 is 128 dimensions (SIFT feature amount) × the number of feature points obtained from one image × the number of images.

S2-5: Next, the cluster setting unit 33 is visually
All the images 6 to be clustered are converted into a feature vector having words72 as a dimension. Search for the closest visual words 72 for each local feature of one image 6 and vote for that visual words 72. As a result, a histogram of one image 6 can be obtained. FIG. 9D schematically shows a histogram. The horizontal axis of FIG. 9D is visual words, and the vertical axis is the number of local features (SIFT features) determined to be closest to each visual words in a certain image 6. This histogram becomes the feature vector of each image 6.

S2-6: Next, the cluster setting unit 33 detects the global feature amount. In this embodiment, the GIST feature amount is acquired as an example of the global feature amount. The GIST feature amount is a feature amount that divides the image 6 into small areas and applies Gabor filters of various directions and frequencies to the small areas to describe scene information, and has 960-dimensional features. Since one GIST feature amount can be obtained from one image 6, it is not necessary to create a histogram.

S2-7: Next, the cluster setting unit 33 combines the local feature amount and the global feature amount to generate the final feature amount. Therefore, the number of dimensions is the number of visual words + 960 dimensions.

S2-8: The cluster setting unit 33 outputs the feature amount.

(S3, S4)
Next, the cluster setting unit 33 clusters the feature amount into n clusters. Here, n is not a fixed value, and the best number of clusters is dynamically determined.

FIG. 10 is an example of a diagram illustrating clustering. FIG. 10A schematically shows a set of each image 6 converted into a feature amount in step S2. In order to facilitate the calculation, one image 6 may be used for one camera device 10.

For clustering, the K-means method used for classifying local features may be used, or an EM algorithm using a Gaussian mixture model or a method such as LDA (Latent Dirichlet Allocation) may be used. The number of clusters cannot be determined by the K-means method and the EM algorithm (it is necessary to give them as appropriate). On the other hand, in LDA, clustering and the number of clusters can be determined at the same time. In this embodiment, the optimum number of clusters is determined by using the Stacked Auto Encoder described later as an example of clustering by the K-means method or the EM algorithm.

In order to determine the best number of clusters, the cluster setting unit 33 performs clustering by changing the number of clusters i. In FIG. 10, clusters of i = 5, 6, 100 are schematically shown. The cluster setting unit 33 performs clustering with the number of clusters being i = 5 to 100. FIG. 10B schematically shows the results of clustering with the number of clusters i = 5, 6 and 100.

When setting the number of clusters, the number of clusters i may be increased by one, or the number of clusters i may be increased by, for example, 10 and evaluated by changing the number of clusters i smaller around the number of clusters where the degree of separation is improved. You may.

In order to evaluate the number of clusters, the cluster determination unit 35 calculates the degree of separation R each time clustering is performed. FIG. 11 is an example of a diagram for explaining the degree of separation R. In this embodiment, SAE (Stacked Auto Encoder) is used as an example for calculating the degree of separation R. In FIG. 11, a case where the number of clusters i = 4 will be described as an example. It is assumed that 2500 images 6 are classified into each cluster by clustering. A part of the cluster image 6 (for example, 2000) is used for learning, and the rest (for example, 500) is used for identification.

FIG. 12 is an example of a diagram illustrating a SAE learning method. First, the autoencoder is a network having a feature expressive power that can restore the original pattern even if the feature is compressed. Therefore, the autoencoder can compress the features for learning, and if the learning is successful, the data in the input layer can be restored even if it is compressed. SAE is a network in which autoencoders are combined (stacked). 12 (a) is an example of SAE, and FIGS. 12 (b), (c) and 12 (d) are examples of an autoencoder. Normally, if there are two or more intermediate layers, the error will converge to an inappropriate minimum solution and the error back propagation will not work (this is called the vanishing gradient problem). In SAE, this is not calculated continuously for two layers, but by stacking one layer at a time, it is difficult to fall into the vanishing gradient problem. If the learning of SAE is successful, SAE can restore image 6. However, since SAE cannot be restored when an abnormal image is input, it is assumed that the image 6 is only a normal image.

As shown in FIG. 12A, the feature (node) decreases toward the input layer x, the hidden layer y, the hidden layer z, and the output layer s, which corresponds to the compression of the feature, and the output layer s, the hidden layer z', Restoring the number of nodes to the hidden layer y'and the input layer x'corresponds to restoration to the original pattern. When learning the parameters of such a neural network by SAE, the parameters (weights) between the input layer x and the hidden layer y are learned by the autoencoder shown in FIG. 12 (b).

Next, in order to obtain the parameters (weights) of the hidden layer y and the hidden layer z in FIG. 12 (a), the autoencoder shown in FIG. 12 (c) is used to hide the y in FIG. 12 (b). The parameters (weights) of the layer y and the hidden layer z are learned. Next, in order to obtain the parameters (weights) of the hidden layer z of FIG. 12 (a) and the output layer s, the autoencoder shown in FIG. 12 (d) is used to hide the z of FIG. 12 (c). The parameters (weights) of the layer z and the output layer s are learned. With the above, the parameters (weights) of SAE shown in FIG. 12A are learned.

It will be described back to FIG. The 500 images 6 for identification are represented by x, the SAE is represented by the function f (), and the output of the SAE is f (x). The score S is defined below. i is the cluster number. The score S is the square of the L2 norm (distance) in the space of the feature amount of the image 6.

この計算を各クラスタから抽出した１枚の画像６に対して行う。Ｓ_ciは学習がうまくいくほど小さくなる。

This calculation is performed on one image 6 extracted from each cluster. S _ci becomes smaller as learning is successful.

Next, the degree of separation R of all clusters is defined as follows.

学習がうまくいくこととクラスタリングがうまくいくことは相関するので、分離度Ｒが小さい方が好ましい。したがって、クラスタ設定部３３は幾つかクラスタ数ｉを変えて分離度Ｒを算出し、分離度Ｒが最も小さい時のクラスタ数ｉでクラスタリングすると判定する。つまり、各クラスタのスコアＳを合計した値が最も小さくなるようにクラスタ数を決定する
なお、ＳＡＥ以外でクラスタ数を決定する方法としては、ＡＩＣ（Akaike’s Information Criterion）、ＢＩＣ（Bayesian Information Criterion）などがある。

Since there is a correlation between successful learning and successful clustering, it is preferable that the degree of separation R is small. Therefore, the cluster setting unit 33 calculates the degree of separation R by changing the number of clusters i, and determines that clustering is performed by the number of clusters i when the degree of separation R is the smallest. That is, the number of clusters is determined so that the total value of the scores S of each cluster is the smallest. As a method for determining the number of clusters other than SAE, AIC (Akaike's Information Criterion), BIC (Bayesian Information Criterion), etc. There is.

(S5)
Next, the learning unit 34 creates a learning model for determining an abnormality from the clustered images 6. In order to determine the abnormality, the learning unit 34 learns only the normal image 6.

FIG. 13 schematically shows a learning model for determining the presence or absence of an abnormality from the image 6. In this embodiment, a learning model that combines deep learning and SVM will be described as an example. As shown in FIG. 13, a patch region in which one image 6 is evenly divided is input to the input layer 301 of the multi-layer perceptron for deep learning. In the learning phase, a value (for example, 1) indicating that the image 6 is normal is input to the output layer 303. In a multi-layer perceptron, an error backpropagation method is generally used for learning the weights of the input layer 301, the intermediate layer 302, and the output layer 303. The number of nodes in the output layer 303 is appropriately designed as a number capable of appropriately extracting the features of the image 6. The output value of each node of the output layer 303 is quantified into a value representing the characteristics of the moving body, and the details will be described with reference to FIG.

The SVM305 is originally a supervised learning algorithm that discriminates between two classes, but the SVM can also be used as a one-class SVM. In the 1-class SVM, the 2-class discrimination is replaced by the region discrimination problem for estimating the high-density region of the data, so that the teacher signal becomes unnecessary. The Lagrange's undetermined constant method is used to calculate the support vector, but details are omitted. Since the region of the normal feature vector can be specified as shown in the figure, the abnormality determination unit 36 can determine that the possibility of abnormality is higher as the region deviates from this region.

The learning is performed for each image 6 of the same cluster clustered in step S3. Therefore, a learning model learned to be normal for each scene can be obtained.

FIG. 14 is an example of a diagram for explaining the learning procedure of FIG. 13 in more detail. FIG. 14A schematically shows a plurality of images 6 of each cluster 71. The cluster 71 in FIG. 14A is a cluster in which the image 6 of the parking lot is classified, but is clustered for each scene such as movement of a car or walking. It should be noted that each is an image 6 of only normal behavior. As shown in FIG. 14B, the learning unit 34 divides one image 6 into patch regions (divided evenly in the vertical and horizontal directions) and inputs them to the input layer 301 of the multi-layer perceptron shown in FIG. Dividing into patch areas has the advantage that it is not necessary to consider the structure of the scene. For example, a car leaving the garage has the same characteristics regardless of which direction it moves. In FIG. 14B, the patch area 304 showing the moving vehicle is shown in a thick frame. When the features are extracted for each patch area, when a car or the like moves, the same car is detected in different patch areas of different images 6. Therefore, the learning unit 34 can track the same car between different images depending on the extracted features.

The learning unit 34 acquires the features of each patch region of the image 6 at time t from the features obtained from the output layer 303 of the multi-layer perceptron, and the learning unit 34 obtains the features of the past images 6 such as times t-1, t-2, and so on. Matches with features. This makes it possible to determine to which patch area the car has moved. For example, in the previous image 6 (one frame of the moving image), it can be seen that the car in the patch area at the left end has moved to the right by one patch area. From the above, as shown in FIG. 14C, the shape (feature amount) and movement (moving speed) of the moving vehicle can be quantified. Features representing the shape of a car are extracted by deep learning. The moving speed may be calculated in easy-to-use units such as n-patch area / 1 image, n-patch area / 1 second, n meters / 1 second, and the like. In this way, the shape (feature amount) and movement (movement speed) are quantified and input to the SVM305. If an abnormality occurs, there is some movement, so it is only necessary to extract the features from the patch area where the moving object is detected (movement speed is greater than zero).

As described above, the learning unit 34 learns all the images 6 for each cluster and creates learning models C1 to Cn.

<Functions when anomaly is detected>
When the learning models C1 to Cn are obtained, the information processing apparatus 30 starts abnormality determination for the moving image. FIG. 15 is an example of a functional block diagram of the information processing apparatus 30 related to abnormality determination. The information processing device 30 of FIG. 15 includes a communication unit 31, an abnormality determination unit 36 (referred to as abnormality determination units 1 to n when distinguishing), an abnormality determination unit x, and a determination result processing unit 37. First, the abnormality determination units 1 to n have learning models C1 to Cn, respectively. n is the number of clusters in steps S3 and S4 of FIG. The cluster determination unit 35 can determine the cluster closest to the image 6 of the existing installation location 8 by obtaining the local feature amount and the global feature amount from the image 6 of each existing installation location 8. Table 1 shows the correspondence between the cluster and the existing installation location 8 (camera ID). The association between the existing installation location 8 and the learning model may be performed once before the start of the abnormality determination.

The communication unit 31 allocates the images 6 transmitted from the existing installation locations 8 to the abnormality determination units 1 to n based on the correspondence between the learning models C1 to Cn and the camera ID. At this time, it is more preferable to refer to the label of daytime or nighttime and allocate it to the abnormality determination unit 36 for daytime or the abnormality determination unit 36 for nighttime. The association between the new installation location 7 and the learning model may be performed once before the start of the abnormality determination.

The abnormality determination unit x determines the abnormality of the image 6 of the camera at the new installation location 7. The abnormality determination unit x has a learning model Cx. Similarly, the cluster determination unit 35 can determine the cluster closest to the image 6 of the new installation location 7 by obtaining the local feature amount and the global feature amount from the image 6 of the new installation location 7. The discrimination between daytime and nighttime may be the same as in the abnormality determination units 1 to n.

The abnormality determination units 1 to n and the abnormality determination unit x use the assigned learning model to calculate the accuracy indicating how close the image 6 is to the normal image 6. The accuracy takes a value of 0 to 1, for example, and the closer to normal, the higher the value.

The determination result processing unit 37 acquires the accuracy calculated by the abnormality determination units 1 to n and the abnormality determination unit x, and determines whether or not there is an abnormality. For example, if the accuracy is less than the threshold value, it may be determined that there is an abnormality. If it is determined that there is an abnormality, a guard will be dispatched or the customer will be notified that there is an abnormality.

<Abnormality detection of existing installation location>
FIG. 16 is an example of a diagram schematically illustrating abnormality detection at the existing installation location 8. In FIG. 16, the existing installation location 8a is taken as an example, but the same applies to the other existing installation locations 8.
(1) The cluster determination unit 35 determines a learning model suitable for each of the daytime and nighttime images 6 by clustering the images 6 captured by the camera device 10 at the existing installation location 8a.
(2) After the start of abnormality detection, the camera device 10 at the existing installation location 8a transmits a moving image to the information processing device 30. The video shall be labeled during the day or at night. The information processing device 30 may determine daytime or nighttime.
(3) The abnormality determination unit 36 corresponding to the existing installation location 8a determines the presence or absence of an abnormality in the image 6 by using the learning model assigned during the daytime or nighttime of the existing installation location 8.
(4) The determination result (accuracy) by the abnormality determination unit 36 is sent to the determination result processing unit 37.

In this way, even for the image 6 of the existing installation location 8a, the abnormality can be determined by using the learning model created from the clustered image 6, so that the abnormality can be determined more accurately.

<Abnormal detection of new installation location>
FIG. 17 is an example of a diagram schematically illustrating abnormality detection at the new installation location 7.
(1) The cluster determination unit 35 determines a learning model suitable for each of the daytime and nighttime images 6 by clustering the images 6 captured by the camera device 10 at the new installation location 7.
(2) After the start of abnormality detection, the camera device 10 at the new installation location 7 transmits a moving image to the information processing device 30. The video shall be labeled during the day or at night. The information processing device 30 may determine daytime or nighttime.
(3) The abnormality determination unit 36 of the new installation location 7 determines the presence or absence of an abnormality in the image 6 by using the learning model assigned during the daytime or nighttime of the new installation location 7.
(4) The determination result (accuracy) by the abnormality determination unit 36 is sent to the determination result processing unit 37.

In this way, when the camera device 10 is newly installed at the new installation location 7, an abnormality can be determined using the learning model obtained by clustering the images 6 of the existing installation location 8, so that the camera device 10 can be used. From the beginning of the new installation, it will be possible to accurately determine abnormalities.

<Judgment result processing example>
The processing of the determination result processing unit 37 will be described with reference to FIG. FIG. 18 is an example of a diagram schematically explaining the processing performed by the determination result processing unit 37.
(1) The image 6 is input to the abnormality determination unit 36 as described above. The abnormality determination unit 36 sends information about the installation location of the camera device 10 (for example, a customer ID and a customer name) and a moving image used for determining the abnormality to the determination result processing unit 37 together with the abnormality determination result.
(2) The determination result processing unit 37 compares the determination result of the abnormality with the threshold value, and determines whether or not there is a high possibility of the abnormality. If it is determined that there is a high possibility of abnormality, at least a part of the video and the determination result are transmitted to the customer and the guards. It is more preferable that the observer visually confirms the moving image before transmission. The destinations are a guard center that serves as a command tower for dispatching guards, a waiting area where guards are stationed, guards around the site, customers, and so on.

FIG. 18 shows screen examples of mobile terminals 74a and 74b owned by a customer, a guard, or the like. On the screen of the mobile terminal 74a, a message 401 of "abnormality 80%", a moving image display field 402, a moving image playback button 403, and the like are displayed. Customers and security guards can visually view messages and videos and take appropriate action. On the mobile terminal 74b, one of the plurality of camera devices 10 is highlighted. In this way, when a plurality of camera devices 10 are installed in one installation location, it is easy for customers and security guards to identify the location where the abnormality occurs by displaying which camera device 10 has detected the abnormality. Become.
(3) Customers and security guards report to the police as necessary, and security guards go to the site to check. If the guard goes to the site but is normal, the guard notifies (feeds back) the determination result processing unit 37 to that effect. At the time of notification, the moving image (or information that can identify this moving image) transmitted to the mobile terminals 74a and 74b is transmitted to the determination result processing unit 37. Since the learning unit 34 can learn that the moving image is normal as a teacher signal, the accuracy of abnormality detection can be improved.

<How to use the learning model in the camera device>
In the present embodiment, the monitoring center determines the abnormality, but the camera device 10 can also determine the abnormality.

FIG. 19A shows a procedure in which the camera device 10 at the new installation location 7 uses the learning model Cx to determine an abnormality.
(1) The camera device 10 at the new installation location 7 captures the image 6. As long as there is at least one image 6, it does not have to be a moving image.
(2) The camera device 10 transmits the image 6 to the information processing device 30.
(3) The information processing device 30 stores the image 6.
(4) The information processing device 30 determines which cluster the image 6 of the new installation location 7 applies to.
(5) The information processing device 30 transmits the learning model Cx determined to be applicable to the camera device 10. Since the camera device 10 uses this learning model Cx to determine the abnormality, it is not necessary to transmit the moving image to the information processing device 30 in the abnormality determination phase.

FIG. 19B shows a procedure for determining an abnormality using the image 6 captured by the camera device 10 at the new installation location 7. After a certain amount of time has passed, the camera device 10 at the new installation location 7 can also capture an image 6 sufficient for learning.
(1) The camera device 10 at the new installation location 7 captures the image 6. Since it is an image 6 for determining an abnormality, it is preferable that there are a plurality of images 6 (moving images).
(2) The camera device 10 transmits the image 6 to the information processing device 30.
(3) The information processing device 30 stores a plurality of images 6.
(4) When the information processing device 30 accumulates a certain amount of images 6, it learns the normal images 6 and creates a learning model Cx (Ver2.0).
(5) The information processing device 30 transmits the created learning model Cx (Ver2.0) to the camera device 10 at the new installation location 7.
(6) The camera device 10 can determine an abnormality using this learning model Cx (Ver2.0). It may be used in combination with the original learning model Cx. When used in combination, if it is determined that either the original learning model Cx or the learning model Cx (Ver2.0) is abnormal, the abnormality is transmitted to the monitoring center. As a result, false reports can be reduced.

As described with reference to FIG. 19, the camera device 10 at the new installation location 7 can determine the abnormality immediately after it is installed, and after a lapse of time, the abnormality can be determined by the learning model of the image 6 captured at the installed location.

<Other application examples>
Although the best mode for carrying out the present invention has been described above with reference to examples, the present invention is not limited to these examples, and various modifications are made without departing from the gist of the present invention. And substitutions can be made.

In the present embodiment, the abnormality detection is performed based on the image 6, but the information processing apparatus 30 can also identify the sound and detect the abnormality. In this case, the camera device 10 has a microphone or the like that collects sound. The sound data collected by the camera device 10 is transmitted to the information processing device 30 together with the image 6. The information processing device 30 learns normal sound data and determines whether or not it is abnormal. In this case as well, the abnormality can be detected immediately after the installation at the new installation location 7.

Further, although the camera device 10 and the information processing device 30 are separate bodies in the system configuration diagram of FIG. 2, as described with reference to FIG. 19, the camera device 10 and the information processing device 30 may be integrated.

Further, although it has been explained that the daytime and nighttime images can be classified into different clusters even with the camera device 10 at the same location, the images may be further classified into different clusters for each time zone. In addition, it may be classified into different clusters according to the weather and the season. In this way, the cluster setting unit 33 can classify each image into an appropriate cluster according to the environment in which the camera device 10 captures the image.

Further, the configuration examples shown in FIGS. 6 and 15 are divided according to the main functions in order to facilitate understanding of the processing by the camera device 10 and the information processing device 30. The present invention is not limited by the method of dividing the processing unit or the name. The processing of the camera device 10 and the information processing device 30 can be divided into more processing units according to the processing content. It is also possible to divide one processing unit so as to include more processing.

Further, although there is one information processing device 30 in FIGS. 6 and 15, a plurality of the same information processing devices 30 may exist, and the functions of FIGS. 6 and 15 are distributed to the plurality of information processing devices 30. You may be.

The communication unit 31 and the image storage unit 32 are examples of image acquisition means, the cluster setting unit 33 is an example of classification means, the learning unit 34 is an example of learning means, and the abnormality determination unit 36 is an abnormality determination means. As an example, the abnormality determination unit 36 using the learning model Cx (Ver2.0) is an example of the second abnormality determination means, and the cluster determination unit 35 is an example of the determination means.

6: Image 7: New installation location 8: Existing installation location 9: Monitoring center 10: Camera device 30: Information processing device 100: Image processing system

Claims (10)

An image processing system that detects anomalies from images captured by an image pickup device.
An image acquisition means for acquiring the image from a plurality of first image pickup devices installed at different installation locations, and an image acquisition means.
A classification means for classifying the images having similar characteristics into clusters, and
A learning means for constructing an abnormality determination means for detecting an abnormality by learning the images classified into the cluster for each cluster, and a learning means.
The cluster in which the image of the first image pickup device similar to the image taken by the second image pickup device installed in the installation location different from the first image pickup device is classified is determined and classified into the cluster. It has a determination means for determining the abnormality determination means constructed from the image, and
The abnormality determination means determined by the determination means is an image processing system that determines the presence or absence of an abnormality from the image captured by the second imaging device. The classification means extracts a local feature amount and a global feature amount from the image captured by the first imaging device, and puts the image having similar local feature amount and the global feature amount at the installation location. The image processing system according to claim 1 , which is classified into the same cluster regardless of the above. The classification means classifies the image from which the local feature amount and the global feature amount are extracted into several numbers of the clusters, and calculates the degree of separation at which the classification becomes appropriately small.
The image processing system according to claim 2 , wherein the number having the smallest degree of separation is determined, and the images captured by the first image pickup apparatus are classified into the clusters of the number. The classification means calculates the difference between the image captured by the first image pickup device input to the neural network constructed by the Stacked Auto Encoder and the output value output by the neural network for each cluster, and of each cluster. The image processing system according to claim 3 , wherein the number is determined so that the total value of the differences is the smallest. The classification means classifies the image captured by the first imaging device at the same installation location into the clusters different depending on the environment in which the image is captured.
The image processing system according to any one of claims 1 to 4 , wherein the learning means constructs the abnormality determination means from the images classified into the clusters different according to the environment. The learning means extracts the features of the image divided into regions by deep learning.
Based on the characteristics of the area, the shape and movement of the object shown in the image are quantified.
The image processing system according to any one of claims 1 to 4 , wherein the abnormality determination means for classifying the shape and movement of the target as normal is constructed by an SVM (support vector machine). Image acquisition means for acquiring images from multiple other imaging devices installed at different installation locations, and
A classification means for classifying the images having similar characteristics into clusters, and
A learning model is acquired from an information processing device having a learning means for constructing an abnormality determination means for learning an abnormality by learning the images classified into the clusters for each cluster, and abnormality detection is performed from the captured image. It is an imaging device to perform
The image is transmitted to the information processing apparatus to acquire the abnormality determination means constructed from the cluster in which the images having similar characteristics of the images are classified.
An image pickup apparatus characterized in that it determines the presence or absence of an abnormality in an image captured by the abnormality determination means acquired from the information processing apparatus. A second abnormality determination means constructed from the image captured by the image pickup device is acquired from the information processing device.
The imaging device according to claim 7 , wherein the abnormality determination is performed by both the second abnormality determination means and the abnormality determination means. It is a learning model creation method performed by an image processing system that detects anomalies from images captured by an image pickup device.
A step in which the image acquisition means acquires the image from a plurality of first image pickup devices installed at different installation locations, and
A step in which the classification means classifies the images having similar characteristics into clusters, respectively.
A step in which the learning means constructs an abnormality determination means for each of the clusters by learning the images classified into the clusters and detecting an abnormality.
The determining means determined the cluster in which the image similar to the image captured by the second imaging device installed at the installation location different from the first imaging device was classified, and was constructed from the cluster. The step of determining the abnormality determination means and
A learning model creation method in which the determined abnormality determination means has a step of determining the presence or absence of an abnormality from the image captured by the second imaging device. An information processing device that creates a learning model that detects anomalies from images captured by the image pickup device.
An image acquisition means for acquiring the image from a plurality of first image pickup devices installed at different installation locations, and an image acquisition means.
A classification means for classifying the images having similar characteristics into clusters, and
A learning means for constructing an abnormality determination means for detecting an abnormality by learning the images classified into the cluster for each cluster, and a learning means.
The abnormality determination means constructed from the cluster is determined by determining the cluster in which the image similar to the image captured by the second image pickup device installed at an installation location different from the first image pickup apparatus is classified. anda determining means for determining a,
The abnormality determination means determined by the determination means is an information processing device that determines the presence or absence of an abnormality from the image captured by the second imaging device.

Images