JP2019096179A - Behavior monitoring system - Google Patents

Abstract

To make it possible to monitor whether irregularity by fraudulent behaviors is set from a behavior of a user by inputting images of monitored areas captured by a monitoring camera to a multilayer type neural network.SOLUTION: A monitoring camera 14 is configured to capture an office room 12 targeted for monitoring, and a behavior determination unit 16 of a behavior monitoring system 10 is configured to: cut out images of a monitoring segment in which equipment such as a desk, shelf and the like set in monitored images is arranged; input the cut-out image by a multilayer neural network learned by images indicative of allowed behaviors in which a user having an equipment use authority handles the equipment; and determine irregularity behaviors of a third party. When the irregularity behavior of the third party is determined by the behavior determination unit 16, the behavior determination unit is configured to: transmit a warning signal to a client device 24 of an entering/existing room management facility to let the client device know the irregularity behavior thereof; and reproduce motion images of the monitored area before and after a prescribed time including a point of time of determining the irregularity from recorded images of a recording image device 20; and cause the motion image to be displayed.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a behavior monitoring system that monitors suspicious behavior of an occupant by using a neural network from an image obtained by capturing a room in a facility with a surveillance camera.

  Conventionally, in facilities such as office buildings, entry and exit management facilities have been installed, and a reader terminal such as a card reader installed outside the door of a room entrance, a user ID read from a card held by the user When it is determined that the information matches the user identification information (hereinafter referred to as (user ID)), an authentication signal is sent to the room entry and exit management control device, and the control signal from the room entry and exit management control device The lock is unlocked and managed so that a third party other than the person concerned does not enter the facility (Patent Document 1).

  There is also known a surveillance camera system that monitors the behavior of a user who has entered the facility using human flow line tracking technology. The surveillance camera system stores flow line data for tracking the behavior of a person moving in the monitoring area in a flow line database, and stores camera image data obtained by photographing a person moving in the monitoring area with a camera image database From the flow line data of the person stored in the flow line database, a section in which the person action corresponding to the condition data representing the specific person action is performed is detected, and the person in the detected section is photographed By extracting and reproducing the captured camera image data, it is possible to confirm the behavior of the fraudster (Patent Document 2).

JP 2003-101999 A JP, 2009-284167, A

  However, in such a conventional entry and exit management facility, there is a problem of cohabitation that another person enters together when the user unlocks the electric lock using a card and enters the room. A third party other than the registered user may enter the room by sharing, and it is difficult to monitor the behavior, so there is a possibility of a security leak.

  In addition, in places where many users work, such as office rooms, authorized administrators and persons in charge use information terminal devices such as personal computers to carry out, for example, bank settlement transactions, and confidential information stored in bookcases. The business using the stored file is performed on a daily basis, for example, when an authorized administrator or person in charge temporarily takes a seat for a meeting etc., the information terminal is operated by an unauthorized third party. There is a possibility that the device may be tampered with, or confidential files stored in the bookcase may be viewed or taken out illegally.

  In order to prevent such fraudulent activity, for example, it is conceivable to monitor the image of the surveillance area taken by the surveillance camera, but it is necessary to always look at the image artificially, and the discovery of fraudulent activity is It is difficult, and there is a problem that it will be noticed only when unlawful banking or leakage of confidential information is discovered later.

  An object of the present invention is to provide an actor monitoring system capable of monitoring an image of a monitoring area captured by a monitoring camera in a multi-layered neural network and monitoring whether an abnormality due to a fraud has occurred. Do.

(Action monitoring system)
The present invention relates to a behavior monitoring system
An imaging unit for imaging a monitoring area to be monitored;
An action determination unit that identifies a person in the monitoring area and determines and outputs an abnormal behavior of the identified person;
Is provided,
The action determination unit is characterized by being configured by a multi-layered neural network learned by an image showing permission actions of a person normally present in the monitoring area.

(Person identification department)
The behavior monitoring system further includes a person specifying unit that specifies a person whose abnormal behavior has been determined by the behavior determining unit and outputs person specifying information.

(Functional Configuration 1 of Multilayer Neural Network)
The multi-layered neural network of the action determination unit is composed of a feature extraction unit and a recognition unit,
The feature extraction unit is configured of a convolutional neural network that extracts and outputs feature amounts according to the person from the input image of the person,
The recognition unit is configured by an all-connected neural network including a plurality of all connected layers that input a feature value output from the convolutional neural network and estimate whether or not the person is a permitted person.

(Learning control of multilayer neural network functional configuration 1)
The behavior determination unit
A learning information storage unit in which an image indicating a permitted action of a person normally present in the monitoring area is stored in advance;
Back propagation based on an error between an estimated value output from all coupled neural networks and a predetermined expected value when an image stored in a learning information storage unit is read out and input to a convolutional neural network as a supervised learning image A learning control unit for learning all coupled neural networks and convolutional neural networks by
Is provided.

(Functional configuration 2 of multi-layered neural network)
The multi-layered neural network of the action determination unit includes an image analysis unit and an action recognition unit.
The image analysis unit
A convolutional neural network that extracts feature quantities according to a person from an image of the person that is input, and outputs the extracted feature amount from a predetermined intermediate layer;
A recursive neural network that receives feature quantities output from a convolutional neural network and generates and outputs an image description of an image of a monitoring region;
Configured by
The action recognition unit
A dictionary in which a word indicating a predetermined abnormal behavior is registered;
A determiner that determines an abnormal behavior of a person by comparing words that make up the image description output from the image analysis unit with the words registered in the dictionary;
It consists of

(Determination of abnormal behavior by person identification)
The image analysis unit identifies a person from the image of the person and outputs person identification information;
A word indicating a predetermined abnormal behavior is registered according to a person usually present in the monitoring area,
The determiner compares the words forming the image description with the words indicating abnormal behavior in the person of the person specifying information to determine abnormal behavior of the person.

(Learning control of multilayer neural network functional configuration 2)
The behavior determination unit
A learning information storage unit in which a pair of an image indicating permission behavior of a person normally present in the monitoring area and a predetermined image explanatory text indicating an outline of the image is stored in advance;
The image stored in the learning information storage unit is read out and input to the convolutional neural network as an unsupervised learning image to be learned by back propagation, and the learning image is input and output to the learned convolutional neural network A learning control unit for inputting an image description sentence, which is a pair of the feature amount and the learning image, as unsupervised learning information into a recursive neural network and learning by back propagation;
Is provided.

(Cooperation with room entry and exit management equipment)
Furthermore,
An entry and exit management that performs control to identify a user who enters the monitoring area in the facility and unlocks the electric lock provided on the door of the entrance when the user identification information matches the user identification information registered in advance. Equipment,
An entry imaging unit for imaging a user who enters a room when the user is identified by the entry / exit management facility;
Is provided,
The person identification unit stores the user image captured by the entry imaging unit in conjunction with the control for unlocking the electric lock by the entry / exit management facility, and the user image and the action determination unit determine an abnormal action. User information of the matched user is collated with the image of the person and output as person identification information.

(Warning and recording of abnormal behavior)
Furthermore, a recording device for recording a moving image of the monitoring area imaged by the imaging unit is provided.
When the behavior determination unit determines an abnormal behavior of a person, it transmits an alarm signal to a predetermined external device for notification and also monitors the image captured by the imaging unit for a predetermined time before and after the time when the abnormal behavior is determined. The video of the area is reproduced from the recording device and displayed.

(Multi-monitoring)
The action determination unit sets one or more monitoring sections in which the equipment to be monitored is arranged in the image of the monitoring area captured by the imaging unit, cuts out the image of the monitoring section, and inputs the image to the action determination section Thus, the abnormal behavior of the person is determined and output for each monitoring section.

(Periodical activity monitoring)
The action determination unit inputs an image of the monitoring section every predetermined period, determines and outputs an abnormal action of the person.

(User training image size normalization)
The behavior determination unit normalizes and inputs an image of the monitoring section into an image of a predetermined size.

(Basic effect)
The present invention relates to an action monitoring unit for capturing an image of a monitoring area to be monitored, an action determining unit for identifying a person in the monitoring area, and determining and outputting an abnormal action of the specified person, in the action monitoring system. And a person specifying unit that specifies a person whose abnormal behavior has been determined by the behavior determining unit and outputs the person specifying information, and the behavior determining unit is an image showing permission behavior of the person normally present in the monitoring area. Since it is composed of learned multi-layered neural network, the output of multi-layered neural network matches the image of the authorized user handling the learned equipment with some features but matches some features If not, or if all features do not match, anomalous behavior indicating that the equipment is not handled by an authorized user is determined, and the anomalous behavior caused by a third party's fraudulent activity is monitored. Can.

(Effect of person identification department)
In addition, since the behavior monitoring system further includes a person specifying unit that specifies a person whose abnormal behavior has been determined by the behavior determination unit and outputs person specifying information, the person whose abnormal behavior is determined is who A person who can be identified and whose abnormal behavior has been determined can be promptly and appropriately dealt with.

(Effect by functional configuration 1 of multi-layered neural network)
The multi-layered neural network of the action determination unit is composed of a feature extraction unit and a recognition unit, and the feature extraction unit is a convolutional neural network that extracts and outputs feature quantities according to the person from the input image of the person. A convolutional neural network is configured, and the recognition unit is composed of an all-combined neural network including a plurality of all coupled layers that input a feature value output from the convolutional neural network to estimate whether or not the person is a permitted person. The feature of the user image handling the equipment is automatically extracted, and the feature of the person is extracted from the input information that becomes the user image by the pre-processing, for example, ridgelines of eyes, mouth, ears, etc. in the face. The features of the user image for handling the equipment are extracted without the need for preprocessing to be extracted, and learning has been performed by the recognition unit to be performed subsequently The variable behavior of the person and can be estimated with a high degree of accuracy the abnormal behavior of different third party is handling the equipment.

(Effect of learning control of multi-layered neural network functional configuration 1)
Also, the behavior determination unit reads out a learning information storage unit in which an image indicating a permitted action of a person normally present in the monitoring area is stored in advance, and an image stored in the learning information storage unit to perform supervised learning. A learning control unit for learning all coupled neural networks and convolutional neural networks by back propagation based on an error between an estimated value output from all coupled neural networks and a predetermined expected value when input to a convolutional neural network as an image; Since it is provided, for example, at the stage before starting the operation of the system, the image of one or more monitoring sections set as the monitoring target out of the monitoring images captured by the monitoring camera for the working hours of the office room For example, it is cut out in a one-minute cycle and stored as a learning image, and this is repeated for about one month As a result, a sufficient amount of learning images necessary for learning in the multi-layered neural network can be obtained, and the image of the monitoring section is input to the multi-layered neural network for learning, and the system operation is started. It is possible to accurately determine the abnormal behavior of the third party in the monitoring section by the already-described multi-layered neural network.

(Effect of functional configuration 2 of multilayer neural network)
In addition, the multi-layered neural network of the action determination unit is composed of an image analysis unit and an action recognition unit, and the image analysis unit extracts feature quantities according to the person from the input person's image and extracts from a predetermined intermediate layer The action recognition unit comprises a convolutional neural network to output and a recursive neural network to which feature amounts output from the convolutional neural network are input to generate and output an image description of the image of the monitoring area, and the action recognition unit A dictionary in which a word indicating an abnormal action is registered, and a determiner that determines an abnormal action of a person by comparing a word forming an image explanatory sentence output from the image analysis unit with a word registered in the dictionary;
Therefore, by analyzing the images of the monitoring area, for example, one or more monitoring sections captured by the monitoring camera, the features of the user who handles the fixtures and the items present in the monitoring section are extracted, and the image explanatory text The third party's abnormal behavior and judgment in the monitoring section when a word is generated and the word extracted from the generated image description matches or is compared with a predetermined word indicating an abnormal behavior of the third party of the dictionary Can be notified.

(Effects from the determination of abnormal behavior by person identification)
Further, the image analysis unit identifies a person from the image of the person and outputs person specifying information, and a word indicating a predetermined abnormal action is registered according to the person normally present in the monitoring area, and the determination unit Since the abnormal behavior of the person is determined by comparing the words constituting the image description and the word indicating the abnormal behavior in the person of the person specifying information, it is possible to monitor whether or not the behavior permitted by the specified person Thus, it is possible to determine and issue an action other than the permitted action, for example, an abnormal action such as an operation of a PC without use authority or a carry-out prohibition file.

(Effect of learning control of multilayer neural network functional configuration 2)
Further, the action determination unit includes a learning information storage unit in which a pair of an image indicating permitted behavior of a person normally present in the monitoring area and a predetermined image explanatory sentence indicating an outline of the image is stored in advance; The features stored in the storage unit are read out and input to the convolutional neural network as unsupervised learning images and learned by back propagation, and the learning images are input and output in the learned convolutional neural network A learning control unit for inputting an image description sentence as a pair of a quantity and a learning image into the recursive neural network as unsupervised learning information and learning by back propagation is provided. Therefore, the functional configuration 1 of the multilayer neural network is provided. Similar to the effect of learning control 1, the storage and system of the learning image and the image The learning of the multi-layer neural network using the training information learning image in operation and by the storage of the description, it is possible to abnormal behavior and accurately determine to broadcast a third party.

(Effect by cooperation with room entry and exit management equipment)
Furthermore, a user who enters the monitoring area in the facility is identified, and when the user identification information matches the pre-registered user identification information, control is performed to unlock the electric lock provided on the door of the entrance. An exit management facility and an entry imaging unit for imaging a user who enters the room when the user is identified by the entry / exit administration facility are provided, and the person specifying unit performs control to unlock the electric lock by the entry / exit administration facility In conjunction, the user image captured by the room entry imaging unit is stored, and the user image is collated with the user image and the image of the person whose abnormal behavior has been determined by the behavior determination unit, and the user information of the matched user is identified Since the information is output as information, the person whose abnormal behavior has been determined is a user who has entered the room by unlocking the electric lock by reading an IC card or the like by the room access control facility and is imaged by the room entry imaging unit Is identified, that person Since the entry / exit management information is the person identification information of the person whose abnormal behavior has been determined, it is possible to identify who the employee for whom the person whose abnormal behavior has been determined is or who is the visitor. It is possible to respond promptly and appropriately to the determined person.

  In addition, it is possible to detect that the user does not enter the room to be the monitoring area from the occupancy management information by the entry / exit management facility, and to suspend the determination operation of the behavior determination unit at night or on holidays. The time zone can be used to learn the multi-layered neural network of the action determination unit.

  In addition, when the presence of the user of the monitoring section performing the activity monitoring can be detected from the management information of the entry / exit management facility and the use start of the user terminal is detected from the network connection to the LAN line of the user terminal, etc. The learning image of the monitored area is stored for a predetermined time, and learning of the multi-layered neural network of the behavior determination unit can be performed at the pause time.

(Abnormal behavior warning and the effect of recording and playback)
Furthermore, a recording device for recording a moving image of the monitoring area imaged by the imaging unit is provided, and the behavior determination unit transmits an alarm signal to a predetermined external device for notification when it determines an abnormal behavior of a person. In the case where the abnormal behavior of the third party is determined, for example, since the moving image of the monitoring area captured by the imaging unit before and after the predetermined time including the time point of determining the abnormal behavior is reproduced from the recording device Send an alarm signal to the terminal of the user who is permitted to handle the equipment to be monitored and the equipment to be monitored to notify the abnormal behavior of the person who is to display the alarm, and confirm the situation by the video reproduction that became the basis Make it possible to

(Effect by multi-monitoring)
In addition, the behavior determination unit sets one or a plurality of monitoring sections in which the equipment to be monitored is disposed in the image of the monitoring area captured by the imaging unit, extracts the image of the monitoring sections, and performs multi-layered neural network Since the abnormal behavior of the person is determined and output for each monitoring section by inputting to the network, a large number of users perform work using information terminal devices and files, for example, as in an office room. If the user has a seat on the surveillance screen captured by the surveillance camera, the user may set multiple rectangular surveillance zones, for example, in places where equipment is shown on the surveillance screen, as needed. Abnormal behavior can be monitored in multiple ways.

(Effect of periodical action monitoring)
In addition, since the action determination unit inputs the image of the monitoring section to the multi-layered neural network at a predetermined cycle to determine and output the abnormal action of the person, the moving image captured by the monitoring camera is output. Since the frame image is read every predetermined period such as 1 to 2 minutes without missing the action monitoring, the action monitoring with reduced processing load of the multilayer neural network is made possible by inputting to the multilayer neural network.

(Effect of size normalization of user learning image)
Also, since the behavior determination unit normalizes the image of the monitoring section into an image of a predetermined size and inputs it to the multi-layered neural network, the image size of the monitoring section varies depending on the distance from the monitoring camera to the monitoring section. By normalizing the extracted images of the monitoring section to the same vertical and horizontal sizes, it is possible to improve the determination accuracy in the case of being input to the multilayer neural network.

An explanatory view showing an outline of the behavior monitoring system together with the entry and exit management equipment An explanatory diagram showing an example of an office layout to be monitored Explanatory drawing showing the setting of the monitoring section for the monitoring image The block diagram which showed 1st Embodiment of the action monitoring apparatus of FIG. 1 by functional configuration An explanatory view showing a functional configuration of the multilayer neural network of FIG. 4 Flow chart showing action monitoring control by the action determination unit of FIG. 4 A flowchart showing learning control by the learning control unit of FIG. 4 The block diagram which showed 2nd Embodiment of the action monitoring apparatus of FIG. 1 by functional configuration A block diagram showing the functional configuration of the convolutional neural network and the recursive neural network of FIG. A flowchart showing behavior monitoring control by the behavior determination unit of FIG. 8 A flowchart showing learning control by the learning control unit of FIG. 8

Behavior monitoring system
(System overview)
FIG. 1 is an explanatory view showing an outline of a behavior monitoring system together with a room entry and exit management facility. As shown in FIG. 1, a monitoring camera 14 functioning as an imaging unit is installed in an office room 12 of a facility such as a building, and a moving image is captured by the monitoring camera 14 inside the office room 12 where a plurality of users are present. doing.

  The surveillance camera 14 captures an RGB color image at, for example, 30 frames / second and outputs it as a moving image. Also, one frame has a pixel arrangement of 4056 × 4056 pixels, for example.

  An activity monitoring apparatus 10 is installed at a management center or the like of the office room 12. The behavior monitoring device 10 is provided with a behavior determination unit 16, a person identification unit 17, a learning control unit 18, and a recording device 20, and signal lines from the monitoring camera 14 installed in the office room 12 are input in parallel to each other. There is.

  The action determination unit 16 includes a multi-layered neural network, and the multi-layered neural network is a user terminal or the like existing in one or more monitoring sections set in the image of the office room 12 serving as the monitoring area. An image of equipment such as an information terminal and a bookshelf and a user who is permitted to use equipment to be monitored (hereinafter referred to as a "user with authority to use equipment") has been learned in advance by an image that handles equipment.

  During system operation, the behavior determination unit 16 performs behavior determination operation when, for example, the user of the office room 12 is present, and the image of the monitoring area captured by the monitoring camera 14 is displayed in a multi-layered neural network To determine whether the abnormal behavior of a third party (person) other than the user who has the authority to use the equipment, that is, whether or not the person who normally exists in the monitoring area is the authorized action of the person, The alarm signal is sent to a predetermined management device including the user terminal of the person to make a notification, and the moving image of the monitoring area for the predetermined time before and after including the time when the abnormal behavior recorded by the recording device 20 is determined Reproduction is possible.

  The person specifying unit 17 specifies a person whose abnormal action has been determined by the action determining unit 16 and outputs person specifying information. For example, the person specifying unit 17 stores the user image captured by the entry camera 25 in conjunction with the control of unlocking the electric lock 28 by the entry / exit management facility to be described later, and stores the user image with the user image. The behavior determination unit 16 collates the image with the person whose abnormal behavior has been determined, and outputs the user information of the matched user as person identification information.

  The learning control unit 18 selects, for example, an image of one or a plurality of monitoring sections set as a monitoring target from among images of the monitoring area captured by the monitoring camera 14 in the working time zone of the office room 12 before starting operation of the system. For example, it is cut out in a one-minute cycle and stored as a learning image, and this is repeated, for example, for about one month to collect a sufficient amount of learning images necessary for learning in a multi-layered neural network. By inputting to a network and learning by deep learning, when starting operation of the system, it is possible to accurately determine the abnormal behavior of the third party in the monitoring section by the learned multi-layered neural network .

  In addition, during operation of the system, for example, when a user having authority to use equipment starts using the user terminal at the start of work, the learning control unit 18 monitors for a predetermined time as in the case before the start of operation of the system. The learning image stored in the area is stored in the learning image of the area, and the user is not present in the office room 12, and the learning is stored in the vacant time such as night or holiday when the action judging operation of the action judging unit 16 is paused. Even if there is a change in the clothes, the hair style, etc. of the user who has the authority to use the equipment to be monitored, the image can be input to the multi-layered neural network of the action determination unit 16 and learned. It is possible to accurately determine the abnormal behavior of the third party in the monitoring section by reflecting it on the multi-layered neural network.

(Overview of entry and exit management system)
A facility in which the behavior monitoring system of the present embodiment is installed is provided with a room entry and exit management facility. As shown in FIG. 1, a door 30 is provided at the entrance of the office room 12 to be monitored by the occupants, and an electric lock 28 is provided on the door 30. The card reader 26 is disposed in the vicinity of.

  The card reader 26 and the electric lock 28 are connected to the entry and exit management control device 22 by a transmission line. The entry and exit management control device 22 is disposed, for example, on each floor of a building. A center device 23 and a client device 24 are disposed at a management center of the facility, etc., and are connected to each other by a LAN line 21 and connected to an entry management control device 22 on each floor.

  In addition, an entrance camera 25 functioning as an entrance imaging unit is installed at the entrance of the office room 12, and the user who enters the room is imaged in conjunction with the control for unlocking the electric lock 28, and the imaged user image is input / output The user image is transmitted to the client device 24 via the management control device 22, stores the user image corresponding to the user management information, and sends the user image to, for example, the person specifying unit 17 provided in the action monitoring device 10, The determination unit 16 collates the image with the person whose abnormal action has been determined, and outputs the user information of the matched user as person identification information.

  Management control of entry and exit management equipment is as follows. The card reader 26 reads personal information such as a user ID from a magnetic card or non-contact IC card carried by the user, for example, and collates it with personal information registered in advance, and determines an authentication success based on the collation match. Is sent to the room entry and exit management controller 22.

  When the entry / exit management control device 22 receives an authentication signal from the card reader 26, the entry / exit management control device 22 outputs a control signal to the electric lock 28 of the door 30 provided at the corresponding entrance to unlock and control entry. When leaving the room, authentication by the card reader 26 is not necessary, and the electric lock 28 can be unlocked and exited by operating the switch button provided inside the door 30 or the like.

  The center device 23 is a personal computer with a display, and displays management information such as a map of a building in which the room access management facility is installed. The client device 24 is a personal computer with a display, and is connected to the room entry and exit management controller 22 via the LAN line 21, and a magnetic card or contactless with the card reader 26 via the room entry and exit management controller 22. Performs various settings and processing such as registration, deletion, history search, etc. of personal information corresponding to the IC card, and counts people in the office room 12 for each authentication processing based on a card read signal from the card reader 26 It manages the number of users in the room and the names of users in the room (user ID).

  Since the action determination unit 16 provided in the action monitoring apparatus 10 is connected to the LAN line 21 and in the present embodiment, the client device 24 of the entry and exit management facility is used as a management apparatus of the action monitoring system. When an abnormal behavior of a third party other than the user having the right to use equipment is determined from the image of the monitoring section by the behavior determination unit 16, an alarm signal is transmitted to the client device 24 through the LAN line 21 to The alarm indicating the abnormal behavior of the person is notified, and the moving image of the monitoring area of the predetermined time before and after the time including the time when the abnormal behavior recorded by the recording device 20 is determined can be reproduced.

  Further, a user terminal 32 as a target of behavior monitoring installed in the office room 12 is connected to the LAN line 21. The user terminal 32 uses a personal computer with a display. Although the user terminal 32 is disposed in the office room 12, it is shown in a state of being taken out of the office room 12 in order to indicate the connection with the LAN line 21.

  In addition, the behavior determination unit 16 controls the suspension and activation (pause release) of the behavior determination operation using the occupancy management information of the office room 12 managed by the client device 24 of the entry / exit management facility.

  That is, the behavior determination unit 16 suspends the behavior determination operation when it is detected from the occupancy management information of the client device 24 that the user is not present at night or on a holiday in the office room 12 to be monitored. When the presence of the user is detected during the pause, the pause is canceled and the action determination operation is activated.

  As described above, when the behavior determination operation is suspended by the behavior determination unit 16 during the operation of the system, the learning control unit 18 uses the idle time due to the suspension and stores it during the system operation as before the system operation start. Control is performed to learn the multilayer neural network of the action determination unit 16 based on the learning image of the monitored area.

(Overview of office room monitoring)
FIG. 2 is an explanatory view showing an example of an office layout to be monitored, and FIG. 3 is an explanatory view showing setting of a monitoring section for a monitoring image.

  As shown in FIG. 2, in the office room 12 to be monitored by the behavior monitoring system, a plurality of desks and chairs are disposed inside, and equipment such as a personal computer is placed on the desk. In the desk arrangement of the office room 12, for example, desks are arranged in two rows in the vertical direction in the drawing in units of department or clerk, and a manager's seat is arranged on the upper side in the drawing.

  The surveillance camera 14 is arranged near the ceiling of the lower left corner of the office room 12 so as to look over the entire room so that the whole room can be reflected in the surveillance image. In addition, when the monitoring camera 14 can not image the whole room to one, you may install several monitoring camera 14 as needed. Also, a wide angle surveillance camera 14 having an imaging range of 180 ° or 360 ° may be installed as the surveillance camera 14.

  In addition, a card reader 26 is disposed outside the door 30 provided at the entrance of the office room 12, and when entering the room, the user causes the card reader 26 to read the card and unlock the electric lock.

  For the office room 12, monitoring sections 34-1 to 34-8 indicated by rectangular dotted lines to be monitored by the behavior monitoring system are set. For example, the monitoring section 34-1 is set in the seat of the manager of manager A who is the manager, the desk of the monitoring section 34-1 has the user terminal 32-1, and the user terminal 32-1 has the authority of using the manager A. There is. For this reason, the monitoring area 34-1 monitors the action by a third party other than the manager of department A who has the authority to use the equipment of the user terminal 32-1.

  Also, the monitoring area 34-2 is set to the bookshelf 35-2, and the bookshelf 35-2 stores files of confidential documents for which the manager of the department A is authorized to use. Therefore, the monitoring area 34-2 monitors an action such as taking out a file by a third party other than the manager A who has the authority to use the equipment of the bookshelf 35-2.

  Such setting of the monitoring section for the office room 12 is performed using the monitoring image 36 of the monitoring area captured by the monitoring camera 14 as shown in FIG. 3. The monitoring image 36 of FIG. 3 is an example, and does not correspond to the office room 12 of FIG.

  As shown in FIG. 3, the monitor image 36 is displayed on the client device 24 of the room entry and exit management facility of FIG. 1 functioning as a management device for behavior monitoring control, and in this state, for example, a monitoring section 34-10 to a dotted line rectangle. Set 34-12 in the screen.

  For example, the monitoring section 34-10 is set at the seat of the director X, the desk of the monitoring section 34-10 has the user terminal 32-10, and the user terminal 32-10 has the authority of using the director X. For this reason, the monitoring area 34-10 monitors the action by a third party other than the director of department X who has the right to use the equipment of the user terminal 32-10.

  Further, the monitoring section 34-11 is set to the seat of the employee Y handling confidential information, the desk of the monitoring section 34-11 has the user terminal 32-11, and the user terminal 32-11 has the use authority of the employee Y. Have Therefore, the monitoring area 34-11 monitors behavior by a third party other than the employee Y who has the authority to use the equipment of the user terminal 32-11.

  For the monitoring sections 34-10 to 34-12 set by the monitoring screen 36 in this way, in addition to the action monitoring control, an image of the monitoring section is cut out by the learning control unit 18 shown in FIG. Learning control is performed to learn the multilayer neural network of the action determination unit 16 based on the stored and stored learning image.

First Embodiment of Behavior Determination Device
FIG. 4 is a block diagram showing the first embodiment of the behavior monitoring device of FIG. 1 by a functional configuration.

  As shown in FIG. 4, the action determination unit 16 includes an image extraction unit 44, a multilayer neural network 46 and a time series determination unit 48, and the learning controller 18 includes an image extraction unit 50 and a control unit 52. , A learning image storage unit 54 functioning as a learning information storage unit, an operation unit 55 such as a mouse or a keyboard, and a display unit 56 provided with a display, and these functions are performed by a CPU of a computer circuit corresponding to neural network processing. It is realized by the execution of the program by.

(Function of behavior judgment unit)
The multi-layered neural network 46 of the behavior determination unit 16 is learned by the image of the equipment set in the monitoring section by the learning control unit 18 and the image by which the user having the equipment usage authority handles the equipment.

  The behavior determination unit 16 starts the behavior determination operation when detecting presence in the room from the occupancy management information of the office room 12 managed by the client device 24, and monitors the office room 12 captured by the monitoring camera 14. For example, an image is held in the image cutting out unit 44 at predetermined intervals necessary for behavior monitoring, which is set to about 1 minute, and the images of one or more monitoring sections displayed in the held monitoring image are cut out A multilayer neural network 46 is input to output an estimated value indicating the behavior of the user who has the right to use the equipment.

  For example, taking the monitoring image 36 of FIG. 3 as an example, the behavior determining unit 16 sets the monitoring sections 34-10 to 34 set in the monitoring image 36 in a state where the monitoring image 36 is held by the image cutting unit 44. The image of -12 is cut out and sequentially input to the multilayer neural network 46. For example, in the case of the cut-out image of the monitoring section 34-10, an estimated value indicating the behavior of the director X with authority to use the equipment of the user terminal 32-10. Output

  Further, in the case of only the user terminal 32-10, the section image X of the monitoring section 34-10 does not appear, and in the case of only the user terminal 32-10, the multilayer neural network 46 which has input this cutout image is the presence of the user terminal 32-10 Output an estimated value indicating.

  The estimated value indicating the behavior of the user who has the right to use the equipment output by the multi-layered neural network 46 when the cutout image of the monitoring section is input has a value of 0 to 1 and is an image of the user who has the right to use the equipment. If it is, the estimated value will be 1 or a value close to 1, and if it is an image of a third party, the estimated value will be a value close to 0 or 0.

  For this reason, the multi-layered neural network 46 sets the determination threshold to, for example, 0.5, and when the estimated value is equal to or less than the determination threshold 0.5, it is determined to be an abnormal action by a third party. Output.

  When the determination result of the abnormal behavior by the multilayer neural network 46 continues a plurality of times or for a predetermined time, the time series determination unit 48 determines the determination as the abnormal behavior by the third party and outputs the determination to the person specifying unit 17 Do.

  The person identification unit 17 stores the user image captured by the entry camera 25 in conjunction with the control for unlocking the electric lock 28 by the entry / exit management facility shown in FIG. The image of the person and the image of the person whose abnormal behavior has been determined by the behavior determination unit 16 are collated, and the user information of the matched user is acquired from the client device 24 of the room entry / exit management facility and used as person identification information. For example, an alarm signal including the image of the determined person and the person identification information is transmitted to the client device 24 of the entry and exit management facility, the alarm signal is transmitted, and the abnormal action of the third party is notified by an alarm. It is possible to reproduce a moving image of a monitoring area for a predetermined time before and after the time when the abnormal behavior recorded in the above is determined.

  Further, the alarm signal from the person specifying unit 17 is also transmitted to the user terminal 32 of the monitoring section in which the abnormal behavior is determined, and the user who took the seat comes back and operates the user terminal 32. The alarm of the third party's abnormal behavior is notified, and the moving image of the monitoring area for a predetermined time before and after the time including the time when the abnormal behavior recorded by the recording device 20 is determined can be reproduced. Confirm abnormal behavior and enable necessary measures.

  In addition, the multilayer neural network 46 of the behavior determination unit 16 monitors the multilayer neural network 46 when only the equipment is shown in the input image of the monitoring section and the user having the authority to use the equipment is not shown. A value of 1 or a value close to 1 is output as an estimated value indicating the presence of the target equipment.

  On the other hand, when the equipment to be monitored is taken out illegally, the image of the monitoring area where the equipment is not shown is input to the multilayer neural network 46, and the estimated value indicating the existence of the equipment to be monitored is 0. Or, the value becomes close to 0, and the threshold value is 0.5 or less, so an alarm signal is output to notify of an abnormality, and moving picture reproduction by the recording device 20 before and after a predetermined time including an abnormality occurrence time becomes possible. This makes it possible to monitor the theft of equipment to be monitored.

  In addition, when the image of the monitoring section is cut out from the monitoring image, the behavior determining unit 16 performs control to normalize the image to a predetermined size and input the image to the multilayer neural network.

  This is because, as shown in FIG. 3, in the monitoring sections 34-10 to 34-12 set in the monitoring image 36 captured by the monitoring camera 14, the image size decreases as the distance from the monitoring camera 14 increases. As a result, the images of a plurality of cut out monitoring areas have different vertical and horizontal sizes, and if input to the multi-layered neural network 46 as it is, the determination accuracy of the small size image becomes low, and monitoring is performed to prevent this. The cropped image of the section is normalized to the same size.

  Resizing is a well-known method for normalizing the cutout image of the monitoring section to a predetermined size. When reducing the image size, pixel thinning processing is performed, and when increasing the image size, pixel complementing processing is performed.

  Also in the case where the image of the monitoring section by the learning control unit 18 is cut out and stored as a learning image, similarly, control is performed to normalize and store the image cut out from the monitoring section into an image of a predetermined size.

(Multilayer neural network)
FIG. 5 is an explanatory view showing a functional configuration of the multilayer neural network shown in FIG. 4, schematically shown in FIG. 5 (A), and schematically shown in detail in FIG. 5 (B).

  As shown in FIG. 5A, the multilayer neural network 46 of the present embodiment is composed of a feature extraction unit 58 and a recognition unit 60. The feature extraction unit 58 is a convolutional neural network, and the recognition unit 60 is a fully combined neural network.

  The multilayer neural network 46 is a neural network that performs deep learning (deep learning), is a neural network having a deep hierarchy in which a plurality of intermediate layers are connected, and performs expression learning that is feature extraction.

  A normal neural network requires an operation by artificial trial and error to extract a feature for determining a user who is usually present in an image from an image. By using a convolutional neural network as an input, the pixel value of the image is input, the optimum feature is extracted by learning, and input to the all-connected neural network of the recognition unit 60 to estimate the behavior of the user who has authority to use the equipment. Print a value.

  The all connection neural network of the recognition unit 60 is composed of an input layer 66, a total connection 68, a repetition of an intermediate layer 70 and a total connection 68, and an output layer 72, as schematically shown in FIG. 5B. .

(Convolutional neural network)
FIG. 5B schematically shows the structure of a convolutional neural network that constitutes the feature extraction unit 58. As shown in FIG.

  Convolutional neural networks are slightly different in characteristics from ordinary neural networks and adopt biological structures from visual cortex. The visual cortex contains a receptive field that is a collection of small cells that are sensitive to a small area of the visual field, and the behavior of the receptive field can be mimicked by learning weights in the form of a matrix. This matrix is called a weight filter (kernel) and is sensitive to similar subregions of an image, as well as the role played by the receptive field in biological terms.

  A convolutional neural network can express the similarity between the weight filter and the small area by convolution, through which the appropriate features of the image can be extracted.

  As shown in FIG. 5B, the convolutional neural network first performs convolution processing on the input image 62 using the weight filter 63. For example, the weight filter 63 is a matrix filter with predetermined weighting of 3 × 3 in vertical and horizontal directions, and performs nineteen pixels of the input image 62 by performing a convolution operation while aligning the center of the filter with each pixel of the input image 62. A plurality of feature maps 64a are generated by convolution to one pixel of the feature map 64a which is a small area.

  Subsequently, pooling operation is performed on the feature map 64 a obtained by the convolution operation. The pooling operation is a process of removing feature amounts unnecessary for identification and extracting feature amounts necessary for identification.

  Subsequently, the convolution operation and the pooling operation using the weight filters 65a and 65b are repeated in multiple stages to obtain the feature maps 64b and 64c, and the feature map 64c of the last layer is input to the recognition unit 60, and normal total combination The recognition unit 60 using a neural network estimates the behavior of the user who has the authority to use the equipment.

  It should be noted that the pooling operation in the convolutional neural network is that the unnecessary feature amount is not necessarily clear in identifying the user who has the right to use the equipment, and the necessary feature amount may be deleted. You may not do this.

(Function of learning control unit)
For example, the control unit 52 of the learning control unit 18 shown in FIG. 4 is set as a monitoring target from the moving image of the monitoring area captured by the monitoring camera 14 during the working time zone of the office room 12 before starting operation of the system 1 Alternatively, one or more frame images of a plurality of monitoring sections are cut out, for example, in a one-minute cycle, and stored in the learning image storage unit 54 as a learning image. By repeating such storage of the learning image, for example, for about one month, a sufficient amount of learning images necessary for learning of the multilayer neural network 46 is collected.

  In this way, normalization of the image size is also performed on the learning image stored in the learning image storage unit 54 as in the case of the cutout image of the monitoring section by the action determination unit 16.

  Further, the control unit 52 performs control of cutting out an image of the monitoring area during system operation and storing the image in the learning image storage unit 54. Therefore, the control unit 52 detects the room occupancy of the user who has the right to use the equipment of the monitoring section from the room occupancy management information of the client device 24 and starts the use of the user terminal, for example, the user terminal for the LAN line 21 When the network connection is detected, control is performed to cut out an image of the monitoring area from the monitoring image in a one-minute cycle, for example, for a predetermined time such as 10 to 15 minutes, and store the image in the learning image storage unit 54.

  As described above, the learning image stored in the learning image storage unit 54 at the start of work in the system operation is controlled by the learning control unit 18 during an idle time such as night or holiday when the action determination operation of the action determination unit 16 is paused. Control is performed by sequentially reading out from the learning image storage unit 54 by the unit 52 and inputting it to the multilayer neural network 46.

  As a result, even if there is a change in clothes, hair style, etc. of the user who has the right to use equipment shown in the image of the monitoring section during system operation, learning of the multilayer neural network 46 is performed by the image of the changing monitoring section. It is performed, and the decrease in the estimated value indicating the behavior of the user who has the authority to use the equipment is suppressed, and it is prevented in advance that the behavior of the user who has the authority to use the equipment is erroneously judged as the abnormal behavior of the third party. To be possible.

(Learning of multi-layered neural network)
The learning of the multilayer neural network 46 by the learning control unit 18 shown in FIG. 4 is performed as follows.

  A neural network composed of an input layer, a plurality of intermediate layers, and an output layer provides a plurality of units in each layer and combines them with a plurality of units in other layers, and weights (bias) and bias values are set for each unit The vector product of a plurality of input values and weights is calculated and the bias values are added to obtain the sum, which is passed through a predetermined activation function and output to the unit of the next layer. Forward propagation is performed where the value propagates until reaching.

  To change the weights and biases of such neural networks, we use a learning algorithm known as backpropagation. In back propagation, supervised learning when the data set of input value x and expected output value (expected value) y is given to the network and unsupervised learning when the input value x is given only to the network In this embodiment, supervised learning is performed.

  When performing backpropagation in supervised learning, for example, a function of mean square error is used as an error for comparing values of estimated value y * and expected value y, which are the results of forward propagation passed through the network Do.

  In back propagation, using the magnitude of the error between the estimated value y * and the expected value y, the value is propagated from the back to the front of the network while correcting the weight and bias. The amount corrected for each weight and bias is treated as a contribution to the error, calculated by the steepest descent method, and the value of the weight and bias is changed to minimize the value of the error function.

The procedure of learning by back propagation for a neural network is as follows.
(1) The input value x is input to the neural network, and forward propagation is performed to obtain the estimated value y *.
(2) Calculate an error with an error function based on the estimated value y * and the expected value y.
(3) Perform back propagation in the network while updating weights and biases.

  This procedure is repeated using combinations of different input values x and expected values y until the error of weight and bias of the neural network is minimized to minimize the value of the error function.

[Behavior judgment control]
FIG. 6 is a flowchart showing action determination control by the action determination unit of FIG. As shown in FIG. 6, the action determination unit 16 sets a monitoring section on the image of the monitoring area captured by the monitoring camera 14 in step S1, for example, as shown in FIG.

  Subsequently, the process proceeds to step S2, and the behavior determination unit 16 determines the presence or absence of a room occupant from the room occupancy management information of the office room 12 managed by the client device 24. When it is determined that there is no room occupant The process proceeds to step S3 to suspend the action determination operation. On the other hand, when the presence of a room occupant is determined in step S2, the process proceeds to step S4 to start or continue the action determination operation, and the process proceeds to step S5.

  When the action determination unit 16 determines that the predetermined monitoring timing, which is, for example, a one-minute cycle, is reached in step S5, the process proceeds to step S6, and the image cutting is performed with the frame image of the moving image captured by the monitoring camera 14 as a monitoring image. The image is stored in the output unit 44, and the image of the monitoring section is cut out from the monitoring image to normalize the image size, and input to the multilayer neural network 46 in step S7 to output an estimated value indicating the behavior of the user who has the right to use the equipment. Do.

  Subsequently, the action determination unit 16 determines that the estimated value indicating the action of the user having the right to use the equipment output from the multilayer neural network 46 by the input of the image of the monitoring section in step S8 is equal to or less than a predetermined threshold (or less than the threshold) If it is the action of the user who has the right to use the equipment, the threshold value is exceeded (or more than the threshold value), so when it is determined, the process returns to step S2 and the same process is repeated.

  On the other hand, the estimated value indicating the behavior of the user having the right to use the equipment output from the multi-layered neural network 46 by the motion judging unit 16 at step S8 input of the image of the monitoring section is less than the predetermined threshold (or less than the threshold) If it is determined to be, the process proceeds to step S9 to determine that it is an abnormal action of the third party, and in step S10, for example, the client device 24 of the room access control facility or the user terminal of the monitoring section where the abnormal action is determined The alarm signal is sent to alert the alarm, and the process proceeds to step S11, and instructs the recording device 20 to reproduce, for example, five minutes before and after the monitored moving image including the determination time of the abnormal action, Send to the terminal of the person and display it.

[Learning control]
FIG. 7 is a flow chart showing occupancy learning control by the learning control unit of FIG. As shown in FIG. 7, the control unit 52 of the learning control unit 18 determines from the occupancy management information of the client device 24 in step S21 whether the user having the right to use the equipment for the monitoring area is an occupancy or not When it is determined, the process proceeds to step S22 to determine whether the user terminal arranged in the monitoring area is connected to the LAN line 21. When the connection is determined, the start of use of the user terminal is recognized and the process proceeds to step S23. . By the processes of steps S21 and S22 as described above, the control unit 52 detects an action at the start of work when a user having authority to use equipment in the monitoring area comes to work and starts work.

  For example, when the control unit 52 proceeds to step S23 and determines that the predetermined cutout timing which is a one-minute cycle has been reached, the control unit 52 proceeds to step S24 to cut out the image of the monitoring section and normalize the image size. In step S26, the process is repeated until it is determined that the predetermined learning time has elapsed, such as 10 to 15 minutes from the start of work.

  Subsequently, when the control unit 52 determines in step S27 that the action determination unit 16 is paused at night, on holidays, etc., the control unit 52 proceeds to step S28 and reads the image of the monitoring section stored in the learning image storage unit 54 The learning control to adjust the weight and bias of the multilayer neural network 46 by back propagation so that the estimated value indicating the behavior of the user who has the authority to use the equipment becomes 1 is input to the formula neural network 46 in step S29. The process is repeated until the pause release of the determination unit 16 is determined.

  Subsequently, when it is determined in step S29 that the pause determination (activation) of the action determination operation in the action determination unit 16 is determined in step S29, the control unit 52 proceeds to step S30 to suspend or end learning of the multilayer neural network 46, and from step S21. Repeat the process.

Second Embodiment of Behavior Monitoring Device
FIG. 8 is a block diagram showing a second embodiment of the behavior monitoring apparatus of FIG. 1 by a functional configuration.

  As shown in FIG. 8, the action determination unit 16 provided in the action monitoring device 10 according to the second embodiment includes an image cutout unit 44, an image analysis unit 74, an action recognition unit 76, and a time series determination unit 48. The image analysis unit 74 is provided with a convolutional neural network 78 and a recursive neural network 80, and the behavior recognition unit 76 is provided with a determiner 82 and a thesaurus dictionary 84. The thesaurus dictionary 84 stores abnormal behavior determination words for identifying a third party who does not have the right to use the equipment.

  The learning control unit 18 includes an image cutting unit 50, a control unit 52, a learning data set storage unit 100 functioning as a learning information storage unit, an operation unit 55 such as a mouse and a keyboard, and a display unit 56 including a display. Be done. These functions are realized by execution of a program by a CPU of a computer circuit corresponding to neural network processing.

(Function of behavior judgment unit)
As shown in FIG. 8, the image cutout unit 44 of the action determination unit 16 reads and holds the monitoring image of the office room 12 or the like captured by the monitoring camera 14 for each predetermined cycle, and is set as the held monitoring image An image of one or more monitoring sections is cut out, the image size is normalized, and the image size is output to the image analysis unit 74.

  A convolutional neural network 78 provided in the image analysis unit 74 extracts and outputs feature amounts of the image of the input monitoring section. Recursive neural network 80 inputs the feature amount output from convolutional neural network 78, and an action description sentence of an input image (also referred to as "image explanation text") explaining the action of a user who has authority to use equipment or a third party. Generate and output).

  The determiner 82 of the action recognition unit 76 also uses one or more words constituting the action explanatory text output from the recursive neural network 80 of the image analysis unit 74 and the equipment use authority stored in the thesaurus dictionary 84. If the word of the behavioral description matches or resembles the abnormal behavior determination word of the thesaurus dictionary 72, the third party who does not have the authority to use the equipment is compared with the abnormal behavior determination word indicating the behavior of the third party. The abnormal behavior indicating the behavior is determined and output to the time series determination unit 48.

  When the determination result of the abnormal action by the action recognition unit 76 continues a plurality of times or for a predetermined time, the time series determination unit 48 determines the determination as the abnormal action by the third party, and the person identification shown in FIG. 1 Output to the unit 17.

  In the person identification unit 17, a user image captured by the entry camera 25 is stored in conjunction with control of unlocking the electric lock 28 by the entry / exit management facility, and the stored user image and action recognition are stored. The part 76 collates the image of the person whose abnormal behavior has been determined, and acquires the user information of the matched user from the client device 24 of the entry / exit management facility as person identification information, and the person whose abnormal behavior has been determined An alarm signal including an image and person identification information, for example, transmitted to the client device 24 of the room entry and exit management facility to transmit an alarm signal to notify an alarm of an abnormal behavior of a third party, and an abnormality being recorded by the recording device 20 The moving image of the monitoring area of a predetermined time before and after the time point when the action is determined is reproduced and displayed on the client device 24 or the like.

(Function of learning control unit)
The folding neural network 78 and the recursive neural network 80 provided in the image analysis unit 74 of the action determination unit 16 are a pair of a learning image stored in advance in the learning data set storage unit 100 of the learning control unit 18 and its action explanation sentence. Are learned by the control unit 52 using a large number of learning data sets.

  The learning image stored in the learning data set storage unit 100 is, for example, the monitoring section 34-10 to 34-12 shown in FIG. 3, for example, captured by the monitoring camera 14 as in the first embodiment of FIG. It is an image which has been cut out and the image size has been normalized.

  Also, in the learning data set storage unit 100, action explanatory sentences are prepared corresponding to the stored many learning images, and are stored as a large number of learning data sets consisting of pairs of learning images and action explanatory sentences.

  For example, for the learning image cut out from the monitoring section 34-10 shown in FIG. 3, an action explanatory text such as "the director X is operating the user terminal" is prepared, and the monitoring section In the learning image where the director X cut out from 34-10 has taken a seat, an action explanatory sentence (image explanatory sentence) such as "the user terminal is placed on a desk" is prepared, for example. It is stored in the learning data set storage unit 100 as a pair with the image.

  The control unit 52 is a learning convolutional neural network composed of an input layer, a plurality of intermediate layers and an output layer of all couplings, that is, the convolutional neural network of the feature extraction unit 55 shown in FIG. A multi-layered neural network composed of a combined neural network is prepared for learning, and first, a large number of learning images stored in the learning data set storage unit 100 are read out and input as learning images to the multi-layered neural network for learning. And learn by back propagation method (back propagation method).

  Subsequently, the control unit 52 sets the weights and biases obtained in the learned convolutional neural network in the convolutional neural network 78 having no output layer shown in FIG. The learning image is input to the embedded neural network 78 to extract the feature amount, and the action description sentence paired with the extracted feature amount and the input learning image is input to the recursive neural network 80, and the back propagation method is performed. To learn.

  In the image analysis unit 78 composed of the convolutional neural network 78 learned by the learning control unit 18 and the recursive neural network 80 in this manner, for example, the part length X having the right to use the equipment shown in FIG. When an image of the monitoring section 34-10 operating 10 is input, an action description such as “the director X operates the user terminal” is output. In this case, the behavior recognition unit 76 determines that there is a mismatch with the abnormal behavior determination word, and is not recognized as a third party abnormal behavior.

  On the other hand, when an image of the monitoring section 34-10 where, for example, a third party other than the director X operates the user terminal 32-10 is input to the image analysis unit 78, for example, An action description such as “operating a terminal” is output, and the action recognition unit 76 determines, for example, a match with an abnormal action determination word such as “suspicious person”, “use terminal”, or “operation”. Anomalous behavior can be recognized and informed.

[Multilayer neural network of image analysis unit]
FIG. 9 is an explanatory view showing functional configurations of a convolutional neural network and a recursive neural network provided in the image analysis unit of FIG.

(Convolutional neural network)
As shown in FIG. 9, the convolutional neural network 78 comprises an input layer 85 and a plurality of intermediate layers 86. A typical convolutional neural network combines an input layer, a plurality of intermediate layers, and an output layer after the last intermediate layer 86 to estimate the output from the feature quantities of the image as shown in FIG. Although provided, since the present embodiment only needs to extract the feature amount of the input image, the fully coupled neural network in the latter stage is not provided.

  The convolutional neural network 78, as shown in FIG. 5, is slightly different in characteristics from a normal neural network, and takes in a biological structure from the visual cortex. The convolutional neural network 78 can represent the similarity between the weight filter and the small area by convolution and through this operation the appropriate features of the image can be extracted.

  The convolutional neural network 78 performs a convolution process on the input image input to the input layer 85 using a weight filter, and a feature map is generated on the intermediate layer 86. Subsequently, pooling operation is performed on the feature map of the intermediate layer 86 obtained by the convolution operation.

  Subsequently, the feature map is generated up to the last intermediate layer 86 by repeating the convolution operation and the pooling operation using the weight filter for each intermediate layer 86, and in the present embodiment, in the arbitrary intermediate layer 86. The generated feature map is input to the recursive neural network 80 as a feature of the input image.

  The convolutional neural network 78 performs learning by inputting the learning image stored in the learning data set storage unit 56 by the learning control unit 18 shown in FIG. 8, and groups similar images by this learning. Images with clustered feature quantities can be generated.

(Recursive neural network)
A recursive neural network 80 shown in FIG. 9 predicts an action description by inputting the feature quantities of the image extracted using the convolutional neural network 78 together with the word vector.

  The recursive neural network 80 of the present embodiment uses a Long Short-Term Memory-Langage Model (LSTM-LM) which is a deep learning model corresponding to time series data.

  A model of a normal recursive neural network is a model that is composed of an input layer, a hidden layer, and an output layer, and is a model that performs time series analysis using past history by using information of the hidden layer as an input of the next time. On the other hand, the LSTM model calculates the probability that each word will be selected as the t-th word from t-1 words that will be in the past context. That is, the LSTM model takes three inputs of word information of time 1 to t-1 that is hidden one time ago, time t-1 that is a prediction result of one time earlier, and external information, and sequentially Repeat the next word prediction to generate sentences.

The recursive neural network 80 shown in FIG. 9 converts the feature vectors of the image extracted by the convolutional neural network 78 into a matrix input to the LSTM hidden layer 88, and the word S stored in word units in the register 90. converting the ₀ to S _N-1 output vector WeS ₀ ~WeS vector converter 92 to be converted to _N-1, N-1 stage LSTM hidden layer 88 of, LSTM hidden layer 88 to the appearance probability p ₁ ~p _N The probability conversion unit 94 is configured with a cost calculation unit 96 that calculates the cost from the probability of outputting a word by the cost functions log P ₁ (s ₁ ) to log p _N (S _N ) to minimize the cost.

(Learning of recursive neural networks)
The learning targets of the recursive neural network 80 are the vector conversion unit 92 and the LSTM hidden layer 88, and for extraction of feature quantities from the convolutional neural network 78, the learned parameters (weights and biases) are used as they are.

The learning data is the word sequence {St} (t = 0,... N) of the learning image I and its action explanatory text, and the following procedure is performed.
(1) The image I is input to the convolutional neural network 78, and the output of a specific intermediate layer 86 is extracted as a feature vector.
(2) Input the feature vector to the LSTM hidden layer 88.
(3) The word string St is sequentially input from t = 0 to t = N-1, and a probability _{pt + 1} is obtained in each step.
(4) Minimize the cost obtained from the probability pt + 1 (St + 1) of outputting the word St + 1.

(Generation of image description)
In the case of generating the action explanation of the input image using the trained convolutional neural network 78 and the recursive neural network 80, the feature extracted from the monitoring section is input to the convolutional neural network 78 and the feature amount generated is A vector is input to the recursive neural network 80, and word sequences are arranged in descending order of the product of the word appearance probability to generate an action description. The procedure is as follows.

(1) The image is input to a convolutional neural network 78, and the output of a specific intermediate layer 86 is extracted as a feature vector.
(2) Input a feature vector from the LSTM input layer 87 to the LSTM hidden layer 88.
(3) The start symbol <S> of the sentence is converted into a vector using the vector conversion unit 92 and input to the LSTM hidden layer 88.
(4) Since the appearance probability of a word is known from the output of the LSTM hidden layer 88, the top M (for example, M = 20) words are selected.
(5) The word output in the previous step is converted into a vector using the vector conversion unit 92 and input to the LSTM hidden layer 88.
(6) From the output of the LSTM hidden layer 88, the product of the probabilities of the words output so far is obtained, and the top M word strings are selected.
(7) Repeat the processes of (5) and (6) until the word output becomes a terminal symbol.

  As described above, the behavior determination unit 16 cuts out and analyzes the image of the monitoring section set in the monitoring image captured by the monitoring camera 14 to thereby allow an abnormal action of a third party who does not have the authority to use the equipment of the monitoring section. Can be determined and reported.

  Note that the convolutional neural network 78 and the recursive neural network 80 according to the present embodiment may be unsupervised learning or supervised learning.

[Behavior judgment control]
FIG. 10 is a flowchart showing action determination control by the action determination unit of FIG. As shown in FIG. 10, the action determination unit 16 sets a monitoring section on the image of the monitoring area captured by the monitoring camera 14 in step S31, as shown in FIG. 3, for example.

  Subsequently, the process proceeds to step S32, and the behavior determination unit 16 determines the presence or absence of a room occupant from the room occupancy management information of the office room 12 managed by the client device 24. When it is determined that there is no room occupant The process proceeds to step S33 to suspend the action determination operation. On the other hand, when it is determined in step S32 that there is a room occupant, the process proceeds to step S34 to start the action determination operation, and the process proceeds to step S35.

  When the action determination unit 16 determines at step S35 that arrival at a predetermined monitoring timing which is, for example, a one-minute cycle is reached, the process proceeds to step S36, and the image cutting is performed using the frame image of the moving image captured by the monitoring camera The image is stored in the output unit 44, and the image of the monitoring section is cut out from the monitoring image to normalize the image size, and input to the multilayer neural network 46 in step S37 to output an action explanation to the determiner 82 of the action recognition unit 76 .

  In step S38, the determiner 82 compares the words constituting the action explanatory text with the abnormal behavior determination words registered in the socilas dictionary 84, and proceeds to step S40 when part or all of the words match the abnormal behavior determination words. The alarm signal is transmitted by transmitting an alarm signal to the client device 24 of the room entry and exit management equipment or the user terminal of the monitoring section in which the abnormal behavior of the third party is determined in step S41. Further, the process proceeds to step S42 to instruct the recording apparatus 20 to reproduce a monitored moving image, for example, five minutes before and after including the determination time point of the abnormal action, and transmit it to the client device 24 or the user terminal for display.

[Learning control]
FIG. 11 is a flowchart showing occupancy learning control by the learning control unit of FIG. As shown in FIG. 11, the control unit 52 of the learning control unit 18 determines the occupancy of the user (authorized user) who has the right to use the equipment for the monitoring section from the occupancy management information of the client device 24 in step S51. Then, in step S52, the presence or absence of connection to the LAN line 21 of the user terminal arranged in the monitoring section is determined, and when connection is determined, the start of use of the user terminal is recognized, and the process proceeds to step S53. By the processes of steps S51 and S52, the control unit 52 detects an action at the start of work when a user having authority to use equipment goes to work and starts work.

  For example, when the control unit 52 proceeds to step S53 and determines that the predetermined cutting timing which is a one-minute cycle has been reached, the process proceeds to step S54 to cut out the image of the monitoring area and normalize the image size. It is determined whether or not there is a person in the image cut out in.

  The determination of the person in the clipped image in step S55 includes, for example, storing in advance a background image in the absence of a person, taking a difference from the clipped image, and the person when the sum of luminance values of the difference image is equal to or more than a predetermined threshold It may be determined that there is.

  If it is determined in step S55 that there is a person in the cut-out image, the control unit 52 proceeds to step S56 and, for example, an action descriptive text of an authorized user such as "authorized user operating user terminal" In step S58, a pair of the monitoring section image and the action explanatory text is stored in the learning data set storage unit 100 as a learning data set.

  If the control unit 52 determines that there is no person in the cut-out image in step S55, the process proceeds to step S57, and for example, an action explanation such as "a user terminal is placed on a desk" without an authorized user A sentence is generated, and in step S58, a pair of the monitoring section image and the action explanatory sentence is stored in the learning data set storage unit 100 as a learning data set.

  The storage of the learning information in steps S53 to S58 is repeated until the control unit 52 determines in step S59 that a predetermined learning time, such as 10 to 15 minutes, has elapsed.

  Subsequently, when the control unit 52 determines that the action determination unit 16 is paused at night, a holiday, or the like in step S60, the process proceeds to step S61, and the monitoring section image and the action description stored in the learning data set storage unit 100. The data set is read out and input to a multi-layered neural network composed of a convolutional neural network 78 and a recursive neural network 80, and the weights and biases of the multi-layered neural network are adjusted so that an action description can be estimated from the input image. The learning control to be performed is repeated until it is determined in step S62 that the action determination unit 16 has been canceled.

  Subsequently, when the control unit 52 determines that suspension release (activation) of the action determination operation in the action determination unit 16 is determined in step S62, the process proceeds to step S63 to suspend or end learning of the multilayer neural network, and the process from step S51 repeat.

[Modification of the present invention]
(Abnormal judgment according to the room occupant)
In the second embodiment described above, the person specified as abnormal is specified by the person specifying unit 17. However, the embodiment is not limited to this. After identifying a person, it may be determined whether the action is permitted for the person. For each person normally present in the area and for a person not normally present in the area, an action dictionary is created regarding an abnormal action determination word as an action for determining an abnormality. During operation, a person is identified by comparing the image analysis unit or the user image with the monitoring image. It is determined whether the action of the person output by the image analysis unit is an action for determining an abnormality in the identified person. This makes it possible to monitor the behavior permitted by the person. For example, normal room occupants have no problem operating the PC, but if they are not ordinary room occupants, an alarm is issued because there is a possibility of security leaks. The department manager issues a warning if an employee tries to take out a file that can be taken out but the employee is not allowed to take it out.

(Learning function)
Although the room occupancy monitoring apparatus shown in the above embodiment is provided with a learning control unit for learning the multi-layer neural network of the action determination unit as an example, the learning of the multi-layer neural network has a learning function. It may be carried out using another computer equipment, and the resulting learned multi-tiered neutral network may be implemented and used in the action determination unit.

(Feature extraction)
Although the above embodiment inputs an image to a convolutional neural network to extract features of a user who is usually in the room, it does not use the convolutional neural network, and features such as contours and shadings from the input image Whether the user is usually in the room by performing pre-processing to extract a predetermined feature to extract a predetermined feature, and inputting the image from which the feature has been extracted into a fully connected neural network or a recursive neural network that functions as a recognition unit It is also possible to estimate a heel image description. As a result, the processing load of image feature extraction can be reduced.

(About the learning method)
Although the above embodiment performs learning by back propagation, the learning method of the multilayer neural network is not limited to this.

(Cooperation with room entry and exit management equipment)
Although the above embodiment controls operation and suspension of the action determination unit using the occupancy management information managed by the entry and exit management facility, the present invention is not limited to this, and is installed in a room to be monitored It is also possible to determine the presence or absence of the room by the human sensor, and control the operation and the pause of the behavior determination unit.

(Storage of learning information during system operation)
Although the above embodiment detects the start of use of the user terminal of the user who has the right to use the equipment in the monitoring area, the image of the monitoring area is cut out and stored as learning information for a predetermined time. The image of the monitoring section may be cut out and stored as learning information during the working hours of a weekday by artificial operation using a user terminal such as a security person in charge of the monitoring area.

(Others)
Further, the present invention is not limited to the above-described embodiment, includes appropriate modifications that do not impair the objects and advantages thereof, and is not limited by the numerical values shown in the above-described embodiment.

10: behavior monitoring device 12: office room 14: monitoring camera 16: behavior determination unit 17: person identification unit 18: learning control unit 20: recording device 21: LAN line 22: entering and leaving management control device 23: center device 24: client Device 25: Entry camera 26: Card reader 28: Electric lock 30: Door 32: User terminal 34-1 to 34-12: Monitoring section 36: Monitoring image 44, 50: Image cutting out part 46: Multilayer neural network 48 : Time series determination unit 52: control unit 54: learning image storage unit 55: operation unit 56: display unit 58: feature extraction unit 60: recognition unit 62: input image 63, 65a, 65b: weight filter 64a, 64b, 64c: Feature map 66, 85: input layer 68: coupling layer 70, 86: middle layer 72: output layer 74: image analysis unit 76: action recognition unit 78: convolution new Network 80: recursive neural network 82: determiner 84: thesaurus dictionary 87: LSTM input layer 88: LSTM hidden layer 90: word register 92: word vector converter 94: probability converter 96: cost calculator 100: learning data Set storage unit

Claims (12)

An imaging unit for imaging a monitoring area to be monitored;
An action determination unit that determines and outputs an abnormal action of a person in the monitoring area;
Is provided,
The behavior monitoring system according to claim 1, wherein the behavior determination unit is configured by a multi-layered neural network learned by an image indicating permission behavior of a person normally present in the monitoring area.
In the behavior monitoring system according to claim 1,
The behavior monitoring system further includes a person specifying unit that specifies a person whose abnormal behavior has been determined by the behavior determining unit and outputs person specifying information.
In the behavior monitoring system according to claim 1,
The multi-layered neural network of the action determination unit is composed of a feature extraction unit and a recognition unit,
The feature extraction unit is configured of a convolutional neural network that extracts and outputs feature amounts according to the person from the input image of the person,
The recognition unit is configured by an all connected neural network including a plurality of all connected layers that input the feature amount output from the convolutional neural network and estimate whether the person is the person of the permitted behavior or not. Behavior monitoring system.
In the behavior monitoring system according to claim 3,
The action determination unit
A learning information storage unit in which an image indicating a permitted action of a person normally present in the monitoring area is stored in advance;
When the image stored in the learning information storage unit is read out and input to the convolutional neural network as a supervised learning image, the error between the estimated value output from the fully combined neural network and a predetermined expected value A learning control unit that learns the fully coupled neural network and the convolutional neural network by back propagation based on;
An activity monitoring system characterized in that
In the behavior monitoring system according to claim 1,
The multi-layered neural network of the action determination unit includes an image analysis unit and an action recognition unit.
The image analysis unit
A convolutional neural network which extracts a feature amount according to the person from the input image of the person and outputs it from a predetermined intermediate layer;
A recursive neural network which receives the feature quantity output from the convolutional neural network and generates and outputs an image description of an image of the monitoring area;
Configured by
The action recognition unit
A dictionary in which a word indicating a predetermined abnormal behavior is registered;
A determiner that compares the words forming the image description output from the image analysis unit with the words registered in the dictionary to determine abnormal behavior of the person;
An activity monitoring system characterized by comprising:
In the behavior monitoring system according to claim 5,
The image analysis unit identifies a person from the image of the person and outputs person identification information.
The word indicating the predetermined abnormal behavior is registered according to the person normally present in the monitoring area,
The behavior monitoring system, wherein the determination unit determines the abnormal behavior of the person by comparing a word forming the image description sentence and a word indicating an abnormal behavior in the person of the person specifying information.
In the behavior monitoring system according to claim 5,
The action determination unit
A learning information storage unit in which a pair of an image indicating permission behavior of a person normally present in the monitoring area and a predetermined image explanatory sentence indicating an outline of the image is stored in advance;
The image stored in the learning information storage unit is read out and input as an unsupervised learning image to the convolutional neural network to learn by back propagation, and the learning image is input to the convolutional neural network after the learning. A learning control unit that inputs the image description, which is a pair of the feature value input and output and the learning image, as unsupervised learning information to the recursive neural network and learns by back propagation;
An activity monitoring system characterized in that
In the behavior monitoring system according to claim 2, further,
A user who enters a monitoring area in the facility is identified, and when the user identification information matches the user identification information registered in advance, a control is performed to unlock the electric lock provided at the door of the entrance. Management equipment,
An entry imaging unit for imaging a user who enters the room when the user is identified by the entry / exit management facility;
Is provided,
The person identification unit stores a user image captured by the room entry imaging unit in conjunction with control for unlocking the electric lock by the entry / exit management facility, and the user image and the action determination unit The room occupant monitoring system, wherein the user information of the user matched with the image of the person whose abnormal behavior has been determined is output as the person identification information.
In the behavior monitoring system according to claim 1,
Furthermore, a recording device for recording a moving image of the monitoring area imaged by the imaging unit is provided,
The action determination unit transmits an abnormality alarm signal to a predetermined external device for notification when it determines an abnormal action of the person, and the imaging unit before and after a predetermined time including a time point when the abnormal action is determined. An action monitoring system characterized by reproducing and displaying a moving image of a captured monitoring area from a recording device.
The behavior monitoring system according to claim 1, wherein the behavior determination unit includes one or more monitoring sections in which the equipment to be monitored is disposed in the image of the monitoring area captured by the imaging unit. An action monitoring system characterized by setting an abnormal action of the person for each of the monitoring sections by setting out and extracting an image of the monitoring section and inputting the image to the action determination unit.
The behavior monitoring system according to claim 1, wherein the behavior determining unit determines and outputs an abnormal behavior of the person by inputting an image of the monitoring section at a predetermined cycle. Monitoring system.
  The behavior monitoring system according to claim 1, wherein the behavior determination unit normalizes and inputs an image of the monitoring section into an image of a predetermined size.

Images