JP7478630B2 - Video analysis system and video analysis method - Google Patents

Description

The present invention relates to a video analysis system and a video analysis method that detects the state of people and objects from video captured in a surveillance area and detects the surveillance target based on the detection results.

In recent years, there has been an increasing need for video surveillance at event venues such as concert halls and amusement facilities, and at public facilities such as stations and airports. For example, in order to prevent terrorist acts using hazardous materials such as explosives or toxic liquids, security requires that actions such as monitoring people, detecting the act, and calling out to people who have committed or are about to commit an act, be taken against the handover of luggage or the abandonment of items inside and outside security areas. In addition, early detection of people struggling with each other, or of a person in the video falling or crouching, allows facility managers to quickly protect people in need of assistance within the facility, contributing to ensuring safety.

For example, the image monitoring device described in Patent Document 1 detects handover actions in a monitored area by calculating information about the posture of the human body from an image captured of the monitored area. Furthermore, the monitoring importance is calculated using the location of the action and the type of item being handed over, and the detection result of the handover action is output according to the monitoring importance. Also, the device describes a method of notifying the "monitoring staff at the monitoring center" of the output of the detection result of the handover action by using a screen display, an alarm lamp, an alarm sound, etc.

In addition, the image processing device described in Patent Document 2 discloses an image processing device that has an acquisition means for acquiring multiple videos distributed from multiple video distribution devices, an object position detection means for detecting the positions of multiple objects contained in each of the multiple videos, an orientation detection means for detecting the orientation of each object at the position of the multiple objects, a setting means for setting a priority for each of the multiple videos based on the orientation of each object, and a display means for displaying the multiple videos based on the priority, with the aim of reducing the burden of monitoring the videos and making it easier to capture the moment something happens.

JP 2017-028561 A JP2017-017441Publication

When using the above-mentioned conventional technology, there are cases where excessive alerts are issued for people who are not actually of high monitoring importance, and workers involved in monitoring work have to respond to the alerted events, increasing their workload. Furthermore, when outputting the results after detecting an event, if there are many actors with high monitoring importance, it becomes necessary to use the video analysis system to track these actors and recognize their behavior, etc., which increases the processing load due to the increase in the number of people. Therefore, the objective of this invention is to provide a video analysis system that can reduce the workload of responders in monitoring and reduce the processing load of the system by setting a monitoring importance for each of multiple people who interact within a monitoring area and accurately narrowing down the people with high monitoring importance.

One aspect of the present invention provides a video analysis system that detects events in a monitored area using video captured of the area, and includes an interaction detection unit that detects interactions, which are events that arise with the involvement of multiple people, based on the video, and outputs the type of interaction and the direction of interaction indicating how each of the multiple people involved in the interaction with the other people; a monitoring importance determination unit that compares the type and direction of the interaction with preset monitoring standard information to determine the monitoring importance of each of the multiple people involved in the interaction; and an output control unit that outputs the detection results of the event based on the monitoring importance.
The present invention also provides a video analysis method for detecting an event in a monitored area using video captured of the area, the video analysis method including an interaction detection step of detecting an interaction, which is an event occurring with the involvement of a plurality of persons, based on the video, and outputting a type of the interaction and a direction of the interaction indicating how each of the plurality of persons was involved with other persons in the interaction, a monitoring importance determination step of comparing the type and direction of the interaction with preset monitoring standard information to determine the monitoring importance of each of the plurality of persons involved in the interaction, and an output control step of outputting a detection result for the event based on the monitoring importance.

According to the present invention, it is possible to set a monitoring importance level for each of multiple people who interact within a monitoring area, thereby reducing the workload of the person in charge of monitoring and the processing load of the system.

1 is an explanatory diagram of a video monitoring system according to an embodiment of the present invention; 1 is a diagram showing an overall configuration of a video monitoring system according to an embodiment of the present invention. FIG. 1 is a block diagram of a video analysis system according to an embodiment of the present invention. FIG. 2 is a diagram showing a flowchart of the video analysis system according to the present embodiment. 4] FIG. 4 is a diagram showing a data structure of monitoring criteria information in the present embodiment. 11 is a diagram showing an example of a setting screen for monitoring criteria information in the present embodiment. FIG. 11 is a diagram showing an example of a setting screen for monitoring criteria information in the present embodiment. FIG. 4A to 4C are diagrams showing examples of displays on a video display unit in the present embodiment. 4A to 4C are diagrams showing examples of displays on a video display unit in the present embodiment. FIG. 11 is a diagram showing a display example of a search unit in the present embodiment. 4A to 4C are diagrams showing examples of displays on a video display unit in the present embodiment. 1 is a diagram showing a hardware configuration of a video monitoring system according to an embodiment of the present invention.

The following describes an embodiment of the video surveillance system according to the present invention. In this embodiment, the system detects coordinated actions by multiple people, such as the handing over of goods or scuffles, in a surveillance area, for the purpose of early detection of terrorist acts or dangerous acts in public facilities such as event venues, stations, and airports. According to the present invention, it is possible to set a surveillance importance for each person who has interacted and to set priorities for events that need to be dealt with, which promotes efficient and rapid response by surveillance personnel and on-site staff in the facility to events. In addition, when the number of staff is fewer than the number of people who need to be dealt with, it is possible to reduce the risk of missing people with high importance. Furthermore, it is possible to carry out processing such as person tracking and behavior recognition after an event occurs starting from people with high surveillance importance, which allows limited computer resources to be appropriately allocated.

In this embodiment, an "event" refers to a situation that is preset as a detection target in a certain monitoring area. In particular, in this embodiment, the detection target is an interaction, which is an event that occurs with the involvement of multiple people. For example, interactions include actions such as shaking hands, handing over luggage, fighting, and assault. Below, an example will be explained using the drawings.

Figure 1 is an explanatory diagram of a video surveillance system in this embodiment. As shown in Figure 1, the video surveillance system 1 is broadly divided into a filming system 2, a video analysis system 3, and a surveillance center system 4. The filming system 2 is composed of a camera unit installed in the area to be monitored. The video analysis system 3 analyzes the input video from the imaging device to determine interactions between people to be detected and the attributes of the people, and further determines the surveillance importance of each person by comparing information on the occurrence location and information on the direction of the interaction with preset surveillance criteria information. The surveillance center system 4 receives the analysis results from the video analysis system 3, and effectively displays them to surveillance staff and on-site staff, and searches for events related to interactions and people after they occur.

Here, the direction of interaction indicates which person performed a specific action related to the interaction to which person. For example, in the case of a handover, the direction of the interaction is set from the person who handed over the item (the person who performed the handover) to the person who received it (the person who received the handover). Similarly, in the case of an assault, the direction of the interaction is set from the assailant (the person who performed the assault) to the victim (the person who received the assault). In this way, the direction of the interaction is determined for each type of interaction. There are also interactions that are two-way, such as handshakes and scuffles.
The attributes of a person include whether the person is a regular person or a security guard, their age, and their gender.
The video surveillance system 1 uses the direction of interaction and the attributes of the people involved in the interaction to set the monitoring importance for each person, thereby reducing the workload of the surveillance personnel and the processing load of the system.

This point will be explained with a concrete example. According to the calculation method that calculates the monitoring importance of a person using the location of occurrence and the type of the delivered item, the degree of monitoring importance of the multiple people who performed the delivery and receipt is not taken into consideration. However, for example, if the item is a suspicious item in the monitored area, it is desirable to set the monitoring importance of the person to whom the item is delivered higher than that of the person who delivered it. Alternatively, if there are areas with different security levels within the monitored area, it is desirable to determine the monitoring importance differently for a delivery from an area with a low security level to an area with a high security level and a delivery in the opposite direction.

Also, for example, if the person who made the handover is a security guard or other security personnel in charge of maintaining security in the monitored area, the monitoring importance should not be set high. However, if such attributes are not taken into consideration, the monitoring importance of the person who made the handover and the security personnel who should really be monitored will be judged to be equal. With this method, it becomes necessary to respond to multiple people whose importance is the same as that presented by the system, which increases the burden on the monitors. Furthermore, if the number of personnel who respond to the handover, such as on-site staff and security guards, is smaller than the number of people who need to be responded to, there is a risk that important targets will be missed.
As described above, the video surveillance system 1 realizes accurate judgment of the importance of surveillance by using information on the direction and attributes of an action, in addition to the type of interaction and the location where it occurred.

The imaging system 2, the video analysis system 3, and the monitoring center system 4 will be specifically described below.
FIG. 2 is a diagram showing the overall configuration of the video monitoring system in this embodiment. The shooting system 2 is composed of one or more camera units 21 installed in the area to be monitored, and the captured images are sequentially input to the video input unit 31 of the video analysis system 3. The camera unit 21 is a monitoring camera arranged so as to capture the entire area to be monitored. In addition, if area setting for detecting an interaction is not required, the monitoring camera may be a mobile camera that is not fixed, and the type does not matter as long as it can capture the area to be monitored. On the other hand, if area setting is required, it is desirable to use a monitoring camera fixed to a wall, a pillar, etc., and to perform calibration setting in advance. In such a case, it is assumed to use a fixed camera that cannot be operated in pan-tilt-zoom (PTZ), but if the combination of those settings and calibration settings is adjusted in advance, a camera capable of PTZ operation may be used, and various areas may be monitored with the same camera.

The camera unit 21 and the video input unit 31 are connected by wired communication means or wireless communication means, and the camera unit 21 continuously transmits frame images to the video input unit 31. When the interaction recognition is a time series data analysis model that assumes the input of multiple frame images, it is desirable that the frame rate of the continuous transmission of frame images is equal to or higher than the required value for the interaction recognition. On the other hand, if the decrease in accuracy of the interaction recognition that occurs when the frame rate falls below the required value is acceptable, the frame rate may be lower than the required value. In this case, in the interaction recognition, a process for suppressing the decrease in accuracy, such as interpolation by interpolation or extrapolation of the time series data, may be performed. Furthermore, the camera unit 21 and the video analysis system 3 do not need to correspond one-to-one, and multiple camera units may be used as one video analysis system. Even in the case of execution of such multiple processes, the frame rate required by each process conforms to the above-mentioned constraints. The camera unit 21 may be equipped with some or all of the functions of the video analysis system described later.

The video analysis system 3 is composed of a video input unit 31, a video processing unit 32, and a storage unit 33. The video input unit 31 accepts video input from the camera unit 21 and transmits video data to the video processing unit 32. The video to be analyzed may not be video directly input from the camera unit 21, but may be video stored separately in a recorder. The location where the video is stored does not matter. The video processing unit 32 has a function of reading the monitoring criteria information stored in the storage unit 33 described later and analyzing the video input from the video input unit 31 to determine the monitoring importance of each individual who has interacted. The storage unit 33 stores the monitoring criteria information set in the management control unit 43 described later. The monitoring criteria information is used to determine the monitoring importance output from the video processing unit 32. In this embodiment, the video analysis system 3 is not limited to an on-premise system constructed on a server in an operation facility, but may be constructed on a server outside the facility, for example, by utilizing a cloud service.

The monitoring center system 4 is composed of a recording unit 41, a video display unit 42, a management control unit 43, and a search unit 44. The recording unit 41 has a function of holding information such as the interaction, the direction of the interaction, the person's attributes, the area where the interaction occurred, and the time of the interaction obtained by the video analysis by the video analysis system 3 as a database. The video display unit 42 displays information on the current behavior of the person who performed the interaction and information on some or all of the frames when the interaction was detected according to the monitoring importance. The management control unit 43 has a function of inputting setting information into the memory unit 33 by a monitor or on-site staff, etc., in order to save the monitoring criteria information used by the video processing unit 32. The search unit 44 has a function of searching for the relevant person from the information stored in the recording unit 41 using the person's attributes and interaction type as a query, and has a function of investigating information such as the relevant person's position at the current time and the movement trajectory within the facility up to that point.

Figure 12 is a hardware configuration diagram of a video monitoring system in this embodiment. In Figure 12, a camera unit 1102 is connected to a computer 1103 via a network. Furthermore, computer 1103 can communicate with computer 1104 via the network.

One or more camera units 1102 are installed in the monitoring area, and transmit video data to the computer 1103 as appropriate. The computer 1103 has a CPU (Central Processing Unit) as an arithmetic and control device, a RAM (Random access memory) as a main storage device, and a HDD (hard disk drive) as an auxiliary storage device. The computer 1103 realizes the functions of the video analysis system 3 by reading various programs from the HDD, expanding them in the RAM, and executing them with the CPU. The computer 1103 also communicates with the camera unit 1102 and the computer 1104 via a specified communication interface (IF). Although not shown in the figure, input/output devices such as a keyboard and a display are also connected to the computer 1103 via a specified IF.

Computer 1104 is equipped with a CPU as an arithmetic and control device, a RAM as a main memory device, and a HDD as an auxiliary memory device, and realizes the functions of monitoring center system 4 by reading various programs from the HDD, expanding them into the RAM, and executing them with the CPU. Computer 1104 is also connected to computer 1103 and input/output devices such as a keyboard and a display via a specified interface (IF).

Next, the video analysis system 3 will be described in detail with reference to FIG. 3. FIG. 3 is a block diagram of the video analysis system according to the present embodiment. Below, the video input unit 31, video processing unit 32, and storage unit 33 that constitute the video analysis system 3 will be described.

The video input unit 31 sequentially receives video from one or more camera units 21 and outputs the video to the downstream video processing unit 32. However, if the video processing unit 32 does not handle time series information, the input may be an image.

The video processing unit 32 is made up of a calculation unit 321 , an interaction detection unit 322 , a monitoring importance determination unit 323 , and an output control unit 324 .
The calculation unit 321 further includes a person detection unit 3211 and an attribute determination unit 3212 .

The person detection unit 3211 detects people from the still image of the current frame using the image or video received from the video input unit. Means of person detection include a means for determining using Haar-like features or R-CNN (Regions with CNN), or a means for determining an estimated area from a group of skeleton coordinates estimated for each person using a skeleton estimation means, and in this embodiment, any means is acceptable. In addition, the person detection unit 3211 tracks people after detecting them. In person tracking, it is sufficient that a rectangular image of a person and a person ID assigned to that person are associated in previous and subsequent frames, and a general person tracking method such as template matching or optical flow can be used.

Next, the rectangular image of the person obtained by the person detection unit is input to the attribute determination unit 3212, and the attributes of the person are determined. By using the attributes of the person, it can be used as information that contributes to the determination of the monitoring importance of each individual. Furthermore, attributes can be used as a query in the search unit 44 described above. Examples of attributes include security guards and staff in the facility, general facility users, age, gender, etc. If it is possible to distinguish between security guards and staff in the facility and general facility users, it is assumed that the interactions caused by security guards and staff are actions performed within the scope of their duties, so that it is expected that there is no need to set a monitoring importance level and no unnecessary alerts will be generated. In addition, by estimating the age and gender of general facility users, for example, if there is prior statistical information that indicates that people of a certain age group are likely to engage in suspicious behavior, it is possible to effectively monitor the video by setting the monitoring importance level for that specific age group higher. Furthermore, in amusement facilities, event facilities, etc., it is also effective to compare the facility users with pre-registered suspicious people or people who are prohibited from entering the facility. Attribute estimation methods include a method of converting a rectangular image of a person into image features such as HOG (Histograms of Oriented Gradients), SIFT (Scale-Invariant Feature Transform), or a vector output from an intermediate layer of a network of a trained deep learning model, and training a classifier such as SVM (Support Vector Machine) or a decision tree, or a method of end-to-end determination using a method based on CNN (Convolutional Neural Network). These determination means may be constructed as a classifier that classifies security guards, on-site staff, and general visitors as the first stage, and a classifier that determines the age and gender of general visitors as the second stage, or may be trained in the same classifier. Furthermore, by using a separate means for determining what items a person possesses, it would be effective to give an attribute that indicates that a person is carrying prohibited or dangerous items in order to appropriately express the attributes.

The interaction detection unit 322 uses the information obtained from the person detection unit 3211 to determine the presence or absence of an interaction, its type, and its direction. The determination method includes a means using image features as described above for any person pair, or a means using skeletal information. When skeletal information is used, a feature representing the posture of a person calculated from the skeleton estimation result by the person skeleton detection means, a feature calculated from the relative distance between any skeleton points of a certain person pair, or a time-series feature representing the movement amount of the skeleton per unit time or the change in relative distance from previous and next image frames may be used. Furthermore, attribute information obtained from the attribute determination unit 3212 may be used as a feature, and for example, a feature representing age or gender may be used. Alternatively, in addition to focusing on a person, a feature representing an item held by a person may be used. For example, when an item that has been determined to be owned by a certain person is determined to be owned by another person after a certain time, it can be interpreted that a handover action has occurred at the time when the ownership determination was switched. These features may be used not only alone but also in combination. For example, if interactions are determined using only posture-related features, there is a possibility that interactions between people located far away may be falsely detected, but by also using features related to relative distance, the number of false positives can be reduced. As described above, effective interaction detection can be achieved by using features alone or in combination.

When image information is used, a method based on CNN can be used. When skeletal information is used, a method of training a classifier such as SVM, decision tree, or LSTM (Long Short-Term Memory) can be used using features representing posture, relative distance, attributes, etc.

The monitoring importance determination unit 323 inputs the attributes of the person obtained from the attribute determination unit 3212, the type and direction of the interaction obtained from the interaction detection unit 322, and in this embodiment, the area information in which the person is located obtained from the person detection unit 3211, and sets the monitoring importance for each individual who has interacted by comparing it with the information set in the monitoring criteria information 331. The area information can be calculated by comparing the pre-set information of the area with the information of the person rectangle. For example, if an area is set on the image coordinates in a camera with a certain PTZ setting, the area in which the person is located can be determined depending on which area the estimated position of the person's feet is located in.

The output control unit 324 transmits the monitoring importance for each individual determined by the monitoring importance determination unit 323 to the monitoring center system 4. All events for which the monitoring importance has been calculated may be transmitted, or a threshold may be preset so that only events with a high monitoring importance are transmitted.

The memory unit 33 stores monitoring criteria information 331 for use by the monitoring importance determination unit 323. The monitoring criteria information 331 has three types of security level setting information: an interaction security level set for each interaction type, an attribute security level set for each attribute type, and an area security level set for each area type. It also has weight information for each of these security level settings and weight information for the performer or performed party set for each interaction type. The monitoring criteria information 331 can be set from the management control unit 43.

Next, the flow of processing in the video analysis system according to the present embodiment will be described with reference to the flowchart shown in FIG.
When an image is input from the imaging system to the image analysis system in step S1, human detection is performed in step S2.
Next, in step S3, the number of people is counted. If two or more people are detected on the screen, the process proceeds to step S4, and if only one or less people are detected, the process waits for the input of the next frame without performing the processes in and after step S4, and returns to step S1. Note that if the screen includes a mixture of areas where interaction detection is desired or permitted and areas where it is not desired or permitted, a partial mask process may be performed on the monitoring area immediately before step 2 in order to reduce the amount of calculation.

In step S4, an interaction judgment is performed. The judgment is performed for any person pair in the screen, but in order to reduce the amount of calculation, it is preferable not to perform the judgment process for person pairs that are more than a certain distance apart for each interaction type. For example, in detecting a handover action, it is not necessary to judge whether or not there is a handover between two people who are clearly out of reach of each other. When judging the relative distance between people before the behavior judgment, it is necessary to calculate the relative distance in the world coordinate system. Therefore, by using area information set in advance, or by using depth estimation technology using a stereo camera or a monocular camera, the position of the person in the world coordinate system is estimated without presetting, and the relative distance between the people is calculated. Furthermore, a separate setting table is prepared, and a judgment threshold is set collectively. For example, if the threshold is set to "3m", the behavior judgment process is not performed for person pairs whose relative distance in the world coordinate system exceeds the threshold of "3m". If the classifier is not a multi-class classifier that corresponds to the recognition of various types of interactions, but a two-class classifier that is trained for each interaction type, the threshold can also be set for each interaction type. In addition, when detecting interactions that cross area boundaries, it is preferable to determine interactions between people located in different areas rather than between people located in the same area, in order to reduce the amount of calculations and false positives.

If the interaction determination determines that an interaction has occurred, i.e., that an event has occurred, the process branches off at step S5 and proceeds to step S6 and subsequent steps. On the other hand, if it determines that no event has occurred, the process does not proceed to step S6 and subsequent steps, and the process waits for the input of the next frame and returns to step S1.

In steps S6 to S8, attributes are calculated for each person who caused the event. Next, in steps S9 to S11, a monitoring importance determination is made for each detected event. After performing output control in step S12 for the determined monitoring importance, the process waits for the input of the next frame and returns to step S1.

Note that the processing of the flow shown in this diagram does not necessarily have to be performed by a single process, and may be performed asynchronously using multiple processes to improve computational efficiency.

Next, referring to FIG. 5, an example of the monitoring criteria information settings in this embodiment is shown. The monitoring criteria information in this embodiment is composed of security level setting information in Table 51, weights for security level setting targets in Table 52, and performer weights for each interaction type in Table 53. From Tables 51 and 52, a weighted score for each individual is calculated taking into account the interaction type, person attributes, and occurrence area, and the monitoring importance is calculated using the weighted score and the performer weight in Table 53. This information is set by the management control unit 43, saved in the monitoring criteria information 331, and read by the monitoring importance determination unit 323. The setting contents and effects of each table are explained in detail below.

Table 51 is made up of three types of security level setting tables: table 511 for setting security levels for each interaction type, table 512 for setting security levels for each attribute type, and table 513 for setting security levels for each area type. In this embodiment, security levels are set in four stages from 3 points to 0 points, and are classified in descending order of points as "high level,""mediumlevel,""lowlevel," and "none level." A high level indicates an object requiring the highest level of attention, and no level indicates an object requiring no attention.
In the interaction security level settings shown in Table 511, the importance of each interaction type is set, for example, "handover" is set to 1 point and "assault" is set to 3 points. Similarly, in the attribute security level settings shown in Table 512, "staff" is set to 0 points and "prohibited persons" is set to 3 points, and in the area security level settings shown in Table 513, "inside the entrance gate" is set to 3 points and "concession stand" is set to 2 points.

For each security level, items that are not subject to setting are explicitly set as level 0; for example, in the interaction security level setting field, "handshake" and "hug" are set as level 0. In this embodiment, each item in tables 511, 512, and 513 is given a four-level score from 3 points to 0 points, but the number of levels is not limited to this embodiment, and it is desirable for the setting person to be able to set it freely.

Table 52 is a setting table that stores weights for each of the three security level setting targets. In the example of Table 52, the sum of the three weights is set to 100%, with interaction set to 30%, attribute set to 20%, and area set to 50%. In this embodiment, these weights are used to calculate weighted scores from the scores in parentheses in Table 51, and are used to calculate the monitoring importance for each individual. For example, if a "handover" action is performed by "general/young person" "inside the entrance gate", the weighted score given to "general/young person" is calculated as 1 x 0.3 + 2 x 0.2 + 3 x 0.5 = 2.2 points according to Table 52. By making it possible to set weights for each of the three security level setting targets, it becomes possible to flexibly respond to different demands for each facility. For example, if there is a demand to emphasize only the type of interaction and person attributes performed but not the area, it is possible to set the interaction to 70%, the attribute to 30%, and the area to 0%. Also, if one or more of the set targets receive a score of 0, the weighted score for that individual may be set to 0. For example, if a person whose attribute is determined to be "security guard" engages in any interaction, it is likely that the interaction was within the scope of their job duties, and it is generally considered inappropriate to target "security guards" as targets for monitoring. The above formula is an example of a formula for calculating the weighted score, and a different formula may be used.

Table 53 is a setting table for calculating the monitoring importance level by considering the direction of the interaction for the weighted scores calculated from Tables 51 and 52. Referring to Table 53, for example, for the "handover" action, the weight for the perpetrator, i.e., the person who hands over the item, is set to 20%. On the other hand, the weight for the victim, i.e., the person to whom the item is handed over, is set to 80%. As in this example, a setting in which the weight for the perpetrator is lower means that the victim is more important than the perpetrator. In the "handover" action, it is assumed that the person who receives the item is more important than the person who gives it away. In addition, the "scuffle" action, which is a two-way interaction, is set to 50% in Table 53 so that the weights for the perpetrator and victim are equal. And for the "assault" action, where the direction of the perpetrator, the assailant, and the victim, the victim, is clear, the weight for the perpetrator is set to 90%. As an example of calculating the monitoring importance from the weighted score, if a "general/young person" performs a "handover" action "inside the entrance gate", the weighted score given to the "general/young person" is 2.2 points according to Tables 51 and 52 as described above, but if the weight of the person who performs the "handover" action is set to 20%, the monitoring importance will be 2.2 x 0.2 = 0.44.

When a person who has caused an event with high monitoring importance subsequently causes an event with low monitoring importance, it is necessary to avoid the monitoring importance value being overwritten and estimated low. Therefore, for a person who has caused multiple interactions, it is desirable to have the monitoring importance calculated for multiple events added together, or to continue to use the value of the large event, until the monitoring importance value for that person is reset.

It is also possible to determine the monitoring importance of a person or to differentiate responses by using the type and direction of multiple interactions involving the same person in a comprehensive manner. For example, consider a case where an event in which pedestrians collide with each other is set as an interaction of type "pedestrian collision", and the person who collided is set as the actor and the person who was hit is set as the victim, and the direction is specified. When referring to the records of interactions in which a certain person was involved, if the person was frequently involved in "pedestrian collisions" and was always on the actor side, it is highly likely that this person intentionally collided. In this case, the monitoring importance can be set high, and security personnel can be dispatched immediately when the person is involved in a similar "pedestrian collision". On the other hand, if the person is frequently involved in "pedestrian collisions" but the direction is not constant (the actor and victim sides are about the same), it is possible that the person is in poor health. In this case, it is desirable to set the monitoring importance high and dispatch rescue personnel when behavior such as "crouching" is observed.

The monitoring importance calculated as above is transmitted to the output control unit 324. The output control unit can set a threshold for the monitoring importance in order to reduce the number of events transmitted to the monitoring center system 4. For example, if the threshold is set to 2.0, monitoring targets calculated to have a monitoring importance of 1.5 will not be transmitted, and individuals calculated to have a monitoring importance of 3.0 will be transmitted. Although not shown in this embodiment, targets for alerts, such as on-site staff or security guards, may be specified according to their monitoring importance scores.

Next, a GUI (Graphical User Interface) for setting the monitoring criteria information of FIG. 5 will be described with reference to FIG. 6 and FIG. 7. FIG. 6 and FIG. 7 are diagrams showing examples of setting screens for monitoring criteria information in this embodiment. FIG. 6 is a setting screen for creating Table 51, and FIG. 7 is a setting screen for creating Tables 52 and 53.

Figure 6 shows a GUI for setting the interaction type, person attributes, and area-related security levels. The following describes the setting of the interaction security level shown in area 611. The number of interaction security levels in area 611 is three, from 1 point to 3 points, but as described above, the number of levels is not limited to this embodiment, and it is preferable that the setter can freely set it. Similarly, it is preferable that the setter can freely set the magnitude of the security level. Furthermore, for interactions for which a security level is not set, the importance may be clearly indicated as 0 points. The setter can press the pull-down field shown in area 6111 and select a registered interaction from the list for each security level column. The selected interaction type is added to the "Registered Interaction" list in the lower row by pressing the "Add" button in area 6112. To delete a registered interaction after adding it, the check box in area 6113 corresponding to the interaction to be deleted is pressed, and area 6114 is pressed. By pressing the "Save Settings" button in area 6115, the information set in this area is reflected in the monitoring criteria information 331. To check the information set in this area, simply press area 6116. The attribute security level shown in area 612 and the area security level shown in area 613 can also be set in a similar manner. However, unlike interaction types and attribute types, whose options are fixed by the system, area types themselves can be added separately to accommodate areas not anticipated by the system.

Area 62 in FIG. 7 is a GUI for setting the weights of the security level setting targets, and table 52 is set by the settings in this area. In area 621, weights for interactions, attributes, and areas are set in percentages. By pressing the "Save Settings" button in area 622, the information set in this area is reflected in the monitoring criteria information 331. To check the information set in this area, simply press area 623.

Area 63 is a GUI for setting the executor weight for each interaction type, and this area is used to set Table 53. In area 631, the executor weight for each interaction type is set as a percentage. The weight for the executed party may be automatically calculated and displayed by inputting the "executor weight". Alternatively, the input may be made on the executed party side. The interactions for which setting is requested in area 631 are all actions that are registered in area 611 and have a score greater than 0 points, and the corresponding row is automatically added at the same time as the registration by pressing area 6115. By pressing the "Save Settings" button in area 632, the information set in this area is reflected in the monitoring criteria information 331. To check the information set in this area, simply press area 633. However, in order to avoid an inconsistency in which the executor weight is not registered in area 63 despite the fact that it has been registered in area 611, it is desirable to implement an implementation such as outputting a display to call attention to any errors in the registered items when the entire setting screen is closed.

Next, an example of a notification screen for a detected event to a monitor will be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of a display on the video display unit 42 in this embodiment, where area 7 indicates an output screen. Area 7 may be displayed on the entire notification screen, or may be displayed on a part of the notification screen.

The people displayed in area 7 on screen A (area 71), screen B (area 72), screen C (area 73), and screen D (area 74) are people for whom monitoring importance has been set and transmitted to the monitoring center system 4. In this embodiment, an example is shown in which images of all these people are displayed at the current time. The images used in this display are real-time images based on tracking of the person who caused the event. Person tracking is performed by the person detection unit 3211 as described above, and the images are transmitted to the image display unit 42 together with information on the monitoring importance. In addition, area 75 shows the people displayed on each screen in order of monitoring importance, and shows the detected event, the location of occurrence, the time of occurrence, and the current location. The column displayed in order of monitoring importance may display the actual values output by the monitoring importance determination unit 323.

In this embodiment, the area sizes of screens A to D are dynamically changed according to the determined monitoring importance. For example, when the monitoring importance of all people is reset and none of the people are displayed on the screen, person a who performed the "handover" act at time "09:20:00" is displayed on the largest screen. Next, if person b who performed the "assault" act at time "09:30:50" is determined to have a higher monitoring importance than the person who performed the "handover" act, the display area of person b becomes larger than the display area of person a. Note that it is desirable to display the people displayed on the screen after applying image processing such as superimposing a detection frame of the person or cropping the image in order to make it easier to distinguish them from other people photographed on the same screen. Also, due to the constraints of the screen size, if multiple events are occurring, the screen may be scrolled.

It is desirable to be able to switch the state of an event by selecting a line in the display screen or area 75, in addition to the real-time tracking video as described above. FIG. 9 is a diagram showing an example of the display of the image display unit in this embodiment, and shows the state in which one event in area 75 in FIG. 8 is selected. In FIG. 9, screen B (area 76) is selected, and the frame at the time of detection of an interaction caused by a person being tracked is displayed. Since an interaction is performed over a certain time span, it is desirable to display the frame with the highest accuracy of judgment between the start and end of the interaction. Alternatively, a short clip from the start to the end of the interaction may be made playable. In this way, the monitor can understand the situation under which the interaction was performed when the event occurred. Furthermore, when the staff has completed their response to the event confirmed in the above manner, when it is determined that no response is necessary, or when a false detection is obvious, the display screen or the line in area 75 can be selected and deleted.

In addition, the screen can be viewed not only by the subject of an alert, such as a monitor at a monitoring center, who is expected to use a large display, but also by staff or security guards responding at the scene, who can view part or all of area 7 at the scene by using a smartphone, tablet, AR goggles, etc.

Next, referring to FIG. 10, a search means using interaction type, attributes, etc. will be described. Events that have been dealt with by monitors or on-site staff in the video display unit 42, or events that have been determined to no longer require a response, are deleted from the display output. Therefore, to check the events thereafter, a search means for the events from the recording unit 41, which is a database of information on the events that have occurred, is required.

10 is a diagram showing an example of the display of the search section in this embodiment, in which an area 9 shows the entire search screen. Within the area 9, an area 91 shows a search input screen, and an area 92 shows an output screen.
In the area 91, the area 911 narrows down the events using the interaction type, attribute, and occurrence time as queries. Specifically, the pull-down column for each search item column in the area 911 is pressed, and a registered item is selected from the list. The selected item is added to the "registered items" list in the lower section by pressing the "Add" button. To delete a registered item after adding it, the check box of the item to be deleted is pressed, and the area 912 is pressed. Note that in the embodiment, an example of a search using the interaction type, attribute, and occurrence time as queries is shown, but a search using the direction of the interaction, area information, etc. as queries may also be performed. Also, input is not necessarily required for each column, and the columns may be left blank. For example, when a search is executed with all items left blank, all information stored in the storage unit 41 is output as a search result.

In area 92, area 921 displays information about the search results, such as the current location and the time of occurrence. As shown in area 921, if the person is confirmed to be staying within the monitoring area at the current time and tracking is possible, information about the current location may be displayed. Area 922 displays a frame image corresponding to the information in area 921 or a short clip video from the start to the end of an interaction. As described above, areas 921 and 922 are combined into one search result, and in this embodiment, the entire search result can be confirmed by scrolling. The search results may be switched to a grid of only the images or videos shown in area 922.

By using the search unit 44, events can be efficiently searched for in the recording unit 41, even if the event has been deleted from the display output by the video display unit 42. In addition, since it is possible to search for similar cases and check the number of occurrences, this can be useful in taking measures to deal with and prevent events that are expected to occur in the future.

As described above, the video surveillance system 1 detects interactions from surveillance video and determines the surveillance importance for each individual using the type and direction of the interaction, as well as the person's attributes and area information. According to the present invention, instead of assigning an equal surveillance importance to all people who have interacted within a surveillance area, a surveillance importance is set for each individual, making it easier for surveillance personnel to prioritize responses and enabling efficient responses to perpetrators.

Below, we will explain another embodiment of the video surveillance system 1 of the present invention. Note that we will omit the description of the invention that is common to the above-mentioned embodiment, and will explain the processing that is unique to this embodiment.

In the display example on the video display unit 42 in the embodiment described above, all events for which a monitoring importance level has been set at the current time are displayed in order of monitoring importance. However, a display method that focuses on only one particular event may also be considered.

11 is a diagram showing an example of display by the video display unit 42 in this embodiment. The information of each person is shown in detail for an interaction between two people for which a monitoring importance level is set. Area 81 shows a frame image in which an interaction is detected. The frame image shows the state of a handover between person 811 and person 812. Area 84 also shows information on the attributes, occurrence location, and current location of the two people, as well as the direction and occurrence location of the handover. As described in area 84, information on the two people is displayed separately on screen X (area 82) and screen Y (area 83). Regarding the screens inside areas 82 and 83, areas 821 and 831 display the images of each person captured in area 81 with the magnification adjusted so that the monitor can easily view them. Areas 822 and 832 also show the location of the event, the current location of the person, and the movement trajectory using a floor map. In this diagram, a circle indicates the location where the event occurred, a pentagon indicates the person's current location and direction of travel, and a dotted line indicates the movement trajectory connecting the location where the event occurred to the current location. Additionally, areas 823 and 833 allow the user to check the state of the person at the current time. The screens shown in areas 82 and 83 allow the user to grasp the movement trajectories of the performer and the performed person from the time the event occurred until the current time.

The following process is required to display the movement trajectory on the floor map. First, images of person 811 and person 812 are acquired, and then person identification is performed at regular time intervals or for regular frames for the frame images acquired from the camera. Next, to identify the position of the person on the floor map, pre-set area information is used, or depth estimation technology using a stereo camera or monocular camera is used to estimate the position of the person in the world coordinate system without pre-setting. By connecting the acquired position information and time information in chronological order, the movement trajectory of the person can be displayed on the floor map.

In addition, while this diagram shows the movement trajectories of both the perpetrator and the executed person from the occurrence of the event, if tracking of a person who did not cause the event has already been performed, or if tracking of that person can be performed after the event occurs by using video stored on a separately prepared storage medium, the movement trajectory of each person to the scene of the event may be displayed. For example, in the case of a handover act, the perpetrator who hands over an item displays the movement trajectory up to the occurrence of the event, and the executed person to whom the item is handed over displays the movement trajectory from after the event occurs, allowing video surveillance to focus on the movement trajectory of the item rather than the person.

As described above, according to this embodiment, the image display unit can display a single event. Specifically, it is possible to display the state of each person who generated an interaction at the time the event occurred, their state at the current time, and their movement trajectory, and it is also possible to display the movement trajectory of the perpetrators up until the event occurred. This allows the monitor to check at a glance not only whether an event occurred, but also the movements of the perpetrators before and after the event occurred, making it possible to easily and accurately grasp detailed information about the event.

As described in the above-mentioned embodiments, the disclosed video analysis system 3 is a video analysis system that detects events in a monitored area using video captured in the monitored area, and includes an interaction detection unit 322 that detects interactions, which are events that occur due to the involvement of multiple people, based on the video, and outputs the type of interaction and the direction of the interaction that indicates how each of the multiple people interacted with other people in the interaction, a monitoring importance determination unit 323 that compares the type and direction of the interaction with preset monitoring criteria information to determine the monitoring importance of each of the multiple people involved in the interaction, and an output control unit 324 that outputs the detection result of the event based on the monitoring importance. With this configuration and operation, a monitoring importance can be set for each of the multiple people who interacted within the monitored area, and the workload of the responder in monitoring and the processing load of the system can be reduced.

The system further includes a calculation unit 321 that detects images of people included in the video and calculates attribute features that represent the attributes of the detected people, and the monitoring importance determination unit 323 further uses the attribute features to determine the monitoring importance, making it possible to make a highly accurate determination that takes into account the attributes of people.

The interaction detection unit 322 also detects the skeleton of a person based on the video, and calculates at least one of the following: a posture feature representing the posture of the person calculated from the estimated result of the detected skeleton, a distance feature calculated from one or more distances between any parts of the person, a movement feature representing the amount of movement of the skeleton per unit time calculated from the difference between previous and next image frames based on the video, and an item feature expressing the ownership relationship of an item to the person, and detects the type and direction of the interaction based on the calculated feature. This configuration enables highly accurate determination taking into account the posture of the person, the distance between the people, the type of item, etc.

Furthermore, the monitoring importance determination unit 323 determines the monitoring importance using information about the positions of multiple people at the time an interaction occurs, making it possible to eliminate unreasonable events based on positional relationships and improve the accuracy of the determination.

The video analysis system 3 also has a storage unit 33 that stores information on the type of interaction, the direction of the interaction, and the security level for each area of occurrence as monitoring criteria information to determine the monitoring importance of each person who has generated the interaction to be detected. By storing this information in advance and using it appropriately, it is possible to realize simple and highly accurate determination.

In addition, by providing a search unit 44 that can search for the person who caused the interaction by searching the interaction detection record using the type of interaction and/or information about the person as a search query, the detected and accumulated interactions can be effectively utilized.

Furthermore, the output control unit 324 changes the size of the display on the display terminal in accordance with the monitoring importance of each person who has generated an interaction. This allows the importance to be easily recognized, and the amount of information can be controlled according to the importance.
In addition, for each person who generated an interaction, the movement trajectory before and after the interaction can be displayed on the screen, making it possible to check the behavior before and after the interaction.
Whether or not to generate trajectories of multiple people may be determined based on their importance, in which case it becomes possible to selectively output trajectories of important people.
Furthermore, the direction of the interaction may be clearly indicated by displaying on the screen information indicating that each of a plurality of people in a detected interaction is either the performer or the recipient of a predetermined action related to the interaction.
In addition, by displaying on the screen a frame image in which an interaction is detected while also displaying on the screen the current position and/or current video of the person involved in the interaction, it is possible to display the content of the interaction in relation to the person's current situation.

The present invention is not limited to the above-mentioned embodiment, and various modifications are included. For example, the above-mentioned embodiment has been described in detail to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the configurations described. In addition, it is possible to replace a part of the configuration of a certain embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of a certain embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration. In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware by designing them as integrated circuits, for example, in part or in whole. In addition, each of the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as programs, tables, files, etc. that realize each function can be placed in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

1...Video surveillance system,
2...photography system, 21...camera unit,
3: Video analysis system, 31: Video input unit, 32: Video processing unit, 321: Calculation unit, 3211: Person detection unit, 3212: Attribute determination unit, 322: Interaction detection unit, 323: Monitoring importance determination unit, 324: Output control unit, 33: Storage unit, 331: Monitoring criteria information,
4: monitoring center system, 41: recording section, 42: video display section, 43: management control section, 44: search section

Claims (13)

1. A video analysis system for detecting an event in a monitored area using a video of the monitored area,
an interaction detection unit that detects an interaction, which is an event occurring with the involvement of a plurality of persons, based on the video, and outputs a type of the interaction and an interaction direction indicating how each of the plurality of persons is involved with other persons in the interaction;
a monitoring importance determination unit for comparing the type and direction of the interaction with preset monitoring standard information to determine a monitoring importance level for each of a plurality of people involved in the interaction;
An output control unit that outputs the event detection result based on the monitoring importance. A calculation unit is further provided for detecting an image of a person included in the video and calculating an attribute feature amount representing an attribute of the detected person,
The video analysis system according to claim 1 , wherein the monitoring importance determination unit determines the monitoring importance by further using the attribute feature amount. The interaction detection unit detects a skeleton of the person based on the video;
A posture feature amount representing a posture of the person calculated from an estimation result of the detected skeleton; and
a distance feature calculated from one or more distances between any parts of the people; and
a movement feature amount that represents a movement amount of the skeleton per unit time, the movement feature amount being calculated from a difference between previous and next image frames based on the video;
and an item feature that expresses an ownership relationship of the item to the person, and the type of the interaction and the direction of the interaction are detected based on the calculated feature. The video analysis system according to claim 1, characterized in that the monitoring importance determination unit determines the monitoring importance using information about the positions of the multiple people at the time the interaction occurred. The video analysis system according to claim 1, further comprising a storage unit that stores, as the monitoring criteria information, information on the type of interaction, the direction of the interaction, and the security level for each area in which the interaction occurs, in order to determine the importance of monitoring for each person who has caused the interaction to be detected. The video analysis system according to claim 1, further comprising a search unit capable of searching for the person who caused the interaction by searching the interaction detection record using the type of the interaction and/or information about the person as a search query. The video analysis system according to claim 1, characterized in that the output control unit changes the size of the display on the display terminal according to the monitoring importance of each person who caused the interaction. The video analysis system according to claim 1, characterized in that the output control unit displays on a screen the movement trajectory of each person who caused the interaction, both before and after the interaction. The video analysis system according to claim 1, characterized in that the output control unit determines whether or not it is necessary to generate movement trajectories of the multiple people based on the monitoring importance. The video analysis system according to claim 1, characterized in that the output control unit displays on a screen information indicating that each of the multiple people in the detected interaction is a performer or a recipient of a predetermined action related to the interaction. The video analysis system of claim 1, wherein the output control unit displays on a screen a frame image in which the interaction is detected, and further displays on the screen the current position and/or current video of the person involved in the interaction. The video analysis system according to claim 3, characterized in that the interaction detection unit calculates the posture feature amount, the distance feature amount, the movement feature amount, and the item feature amount, and detects the type of interaction and the direction of the interaction between the detected people based on the posture feature amount, the distance feature amount, the movement feature amount, and the item feature amount. 1. A computer-implemented video surveillance method for detecting an event in a surveillance area using a video of the surveillance area, the method comprising:
an interaction detection step of detecting an interaction, which is an event occurring with the involvement of a plurality of persons, based on the video, and outputting a type of the interaction and an interaction direction indicating how each of the plurality of persons is involved with the other persons in the interaction;
a monitoring importance determination step of comparing the type and direction of the interaction with preset monitoring standard information to determine a monitoring importance level for each of a plurality of people involved in the interaction;
and an output control step of outputting the event detection result based on the monitoring importance.

Images