JP2019029747A - Image monitoring system - Google Patents

Abstract

To provide an image monitoring system which allows a watchman or the like to efficiently monitor images of a space where congestion of people can occur.SOLUTION: Predetermined image acquisition means 30 successively acquires monitor images. Congestion degree estimation means 50 estimates a congestion degree expressing a degree of congestion of objects in a space on the basis of the monitor images. Monitor importance calculation means 51 calculates duration time of the congestion of the objects in an area of interest set to the space and calculates monitoring importance increased with an increase in the congestion degree and an increase in the duration time with respect to the area of interest.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image monitoring system that generates information indicating the degree to which a monitor or the like should gaze about an object from a monitoring image obtained by capturing a space in which an object such as a person can be crowded.

  In an event venue or other space where congestion can occur, countermeasures such as placing a large number of guards in the crowded area are required to prevent accidents. Therefore, monitoring efficiency can be expected to improve by arranging monitoring cameras at various locations in the venue, estimating crowd density from the captured images, and displaying the congestion status of the monitoring range.

  As an image representation of the analyzed congestion situation, there is a heat map that displays colors according to the degree of congestion for each pixel or each block into which an image is divided. For example, in Non-Patent Document 1 below, a heat map (Figure 2 (e)) in which a color corresponding to an estimated value of the crowd density at each pixel is set for each pixel and a captured image (Figure 2 (a) )). In this example, an area with a particularly high crowd density is displayed in red, an area without people is displayed in blue, and an area having an intermediate crowd density is displayed in orange, yellow, green, and light blue.

V. Eiselein, H. Fradi, I. Keller, T. Sikora and JL Dugelay, "Enhancing human detection using crowd density measures and an adaptive correction filter," 2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance, Krakow, 2013 , pp. 19-24.

  However, the heat map display only displays the estimation result for each frame for each frame. Therefore, it is necessary to check the images of the cameras at each location where a high crowd density is estimated, and the monitoring efficiency is not always high.

  For example, when the crowd density at a certain moment is displayed using the heat map display, even if the crowd density is high, the crowd density is reduced at the next moment, or the crowd density is high. An area where the flow of people is proceeding smoothly is not necessarily a dangerous state. On the contrary, in a region where the crowd density state continues for a long time, or in a region where the flow of people is stagnant, there are many people who feel uncomfortable or those who are unwell, and are likely to be in a dangerous state. Therefore, even in areas where the crowd density is high as well, areas where the crowd density is high for a long time should be given priority over areas where the crowd density is temporarily high, and the flow of people is smooth. Priority should be given to areas where flow is delayed rather than areas. In the prior art, such priorities cannot be set, and the monitoring efficiency is not necessarily high.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide an image monitoring system in which a monitor or the like can efficiently monitor a space in which an object such as a person can be crowded.

  (1) The image monitoring system according to the present invention is a system that generates information representing a degree to which the object should be watched from a monitoring image obtained by photographing a space where a predetermined object may be crowded, and sequentially acquires the monitoring images. A congestion degree estimating means for estimating a degree of congestion representing the degree of congestion of the object in the space, and a duration of congestion of the object in the attention area set in the space from the monitoring image. And monitoring importance calculation means for calculating a higher monitoring importance for the attention area as the congestion level is higher and the duration is longer.

  (2) In the image monitoring system described in (1) above, the monitoring importance degree calculation unit further calculates a movement amount of the object in the attention area from the monitoring image, and the movement amount is calculated for the attention area. It can be set as the structure which calculates high monitoring importance, so that there are few.

  (3) Another image monitoring system according to the present invention is a system that generates information indicating a degree to which the object is to be watched from a monitoring image obtained by photographing a space in which a predetermined object may be crowded, wherein the monitoring image is Image acquisition means for sequentially acquiring, congestion degree estimation means for estimating the degree of congestion representing the degree of congestion of the object in the space from the monitoring image, and movement of the object in the region of interest set in the space from the monitoring Monitoring importance calculation means for calculating an amount, and calculating a higher monitoring importance for the attention area as the congestion degree is higher and the movement amount is smaller.

  (4) In the image monitoring system according to (1) to (3), the image acquisition unit sequentially acquires the monitoring images for each of a plurality of different monitoring areas set in the space, and The monitoring area may be the attention area.

  (5) In the image monitoring system described in (1) to (3) above, the space reflected in each of a plurality of image sections into which the monitor image is divided is set as the attention area, and the monitoring important for each image section A distribution image generation unit that calculates a degree and generates a distribution image expressing the distribution on the monitoring image can be provided.

  According to the present invention, it is possible to provide an image monitoring system in which a monitor or the like can efficiently monitor a space in which an object such as a person can be crowded.

1 is a block diagram showing a schematic configuration of an image monitoring system according to an embodiment of the present invention. It is a functional block diagram which shows the function of the image monitoring system which concerns on embodiment of this invention. It is a schematic diagram which shows an example of the information of the object model which the object model memory | storage means has memorize | stored. It is a schematic diagram which shows the example of the setting about the display method of the monitoring importance degree and congestion degree which the drawing setting memory | storage means has memorize | stored. It is a schematic diagram which shows the example of the distribution image which a distribution image generation means produces | generates. It is a schematic flowchart of the process which calculates the monitoring importance in the image monitoring system which concerns on embodiment of this invention. It is a general | schematic flowchart of the process which displays monitoring importance on a display part. It is a schematic diagram which shows the example of a display of the display part in multi display mode.

  Hereinafter, an image monitoring system 1 according to an embodiment of the present invention (hereinafter referred to as an embodiment) will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a schematic configuration of the image monitoring system 1. The image monitoring system 1 includes a photographing unit 2, a communication unit 3, a storage unit 4, an image processing unit 5, an operation unit 6, a display control unit 7, and a display unit 8.

  The imaging unit 2 is a monitoring camera, is connected to the image processing unit 5 via the communication unit 3, and captures a space (target space) in which a predetermined object can be crowded at predetermined time intervals and outputs a monitoring image. It is a photographing means. Here, the portion of the target space that is imaged by the imaging unit 2 is referred to as a monitoring area. In the present embodiment, a plurality of monitoring areas are set in the target space, and the imaging unit 2 is assigned to each monitoring area. That is, the imaging unit 2 includes a plurality of imaging units 2-1, 2-2,..., And the imaging unit 2 images a plurality of different monitoring areas set in the target space. Each imaging unit 2 is given in advance an identifier (area ID) that uniquely identifies the monitoring area captured by the imaging unit 2 and inputs the area ID to the image processing unit 5 together with the monitoring images to be sequentially output.

  For example, each photographing unit 2 is installed with a field of view overlooking a monitoring area on a pole installed at an event venue. The visual field may be fixed, or may be changed according to a schedule in advance or an instruction from the outside via the communication unit 3. For example, the imaging unit 2 captures the monitoring area with a frame period of 1/5 second to generate a color image. A monochrome image may be generated instead of the color image.

  The communication unit 3 is a communication circuit, one end of which is connected to the image processing unit 5, and the other end is connected to the photographing unit 2 and the display control unit 7 via a communication network such as a coaxial cable, a LAN (Local Area Network), or the Internet. Connected. The communication unit 3 acquires a monitoring image from the photographing unit 2 and inputs it to the image processing unit 5, and outputs information input from the image processing unit 5 to the display control unit 7.

  The storage unit 4 is a memory device such as a ROM (Read Only Memory) or a RAM (Random Access Memory), and stores various programs and various data. The storage unit 4 is connected to the image processing unit 5 and inputs / outputs such information to / from the image processing unit 5.

  The image processing unit 5 is configured by an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit). The image processing unit 5 is connected to the storage unit 4, operates as various processing units / control units by reading and executing a program from the storage unit 4, stores various data in the storage unit 4, and from the storage unit 4 read out. The image processing unit 5 is also connected to the photographing unit 2 and the display control unit 7 via the communication unit 3, and a person who is photographed by analyzing a monitoring image acquired from the photographing unit 2 via the communication unit 3. Is generated, and the generated information is output to the display control unit 7 via the communication unit 3.

  The operation unit 6 is an input device for the display control unit 7 and includes a keyboard and a mouse. The operation unit 6 is connected to the display control unit 7, receives an instruction operation by the monitor, and outputs the instruction operation to the display control unit 7.

  The display control unit 7 is constituted by a PC (Personal Computer) or the like, and is a storage unit (not shown) constituted by a memory device such as a ROM or RAM, and an interface circuit with a communication network to which the communication unit 3 is connected. A communication unit (not shown) and a control unit (not shown) configured by an arithmetic device such as a CPU, MCU, or IC are provided. The display control unit 7 is connected to the communication unit 3 via the communication network, and is also connected to the operation unit 6 and the display unit 8. The display control unit 7 receives information from the image processing unit 5 from the communication unit 3 and stores it, and receives an operation instruction from the monitoring person from the operation unit 6 and displays information corresponding to the operation instruction among the stored information. Output to unit 8.

  The display unit 8 is a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display, and is connected to the image processing unit 5 via the communication unit 3 and the display control unit 7, and information generated by the image processing unit 5. Is a display means for displaying. The monitor visually checks the displayed information to determine the occurrence of congestion, and takes measures such as changing the personnel assignment as necessary.

  FIG. 2 is a functional block diagram showing functions of the image monitoring system 1. The communication unit 3 functions as an image acquisition unit 30 and a gaze information output unit 31 and the like, and the storage unit 4 has a density estimator storage unit 40, a congestion level storage unit 41, a motion amount storage unit 42, an object model storage unit 43, and a drawing. It functions as the setting storage means 44 and the like. The image processing unit 5 functions as a congestion level estimation unit 50, a monitoring importance level calculation unit 51, an object region estimation unit 52, a distribution image generation unit 53, and the like.

  The image acquisition unit 30 sequentially acquires the monitoring image and the area ID from the imaging units 2-1, 2-2,... That are the imaging unit, and the acquired monitoring image together with the area ID, the congestion degree estimation unit 50, and the monitoring importance level. It outputs to the calculation means 51 and the object area estimation means 52.

  The density estimator storage unit 40 is an estimated density calculation function that learns the image features of each density image obtained by photographing a space where an object (person) exists at a predetermined density for each predetermined density. When input, information representing an estimator (density estimator) that calculates and outputs an estimated value (estimated density) of the density of an object photographed in the image is stored in advance. That is, the density estimator storage means 40 stores parameters such as the coefficient of the estimated density calculation function in advance as information on the density estimator.

  The congestion degree estimation means 50 estimates a congestion degree that represents the degree of congestion of an object in the target space based on the monitoring image. The congestion degree estimation unit 50 divides each of the monitoring images input from the image acquisition unit 30 into a plurality of image sections, and estimates a congestion degree that represents the degree of object congestion for each image section. Then, the estimated congestion level is stored in the congestion level storage unit 41 in time series in association with the area ID. The image section can be, for example, each pixel of the monitoring image.

  The congestion degree estimation process and the density estimator will be specifically described.

  The congestion degree estimation means 50 sets a window (estimation extraction window) at the position of each pixel of the monitoring image, and extracts an estimation feature amount from the monitoring image in each estimation extraction window. The estimation feature amount is a GLCM (Gray Level Co-occurrence Matrix) feature.

  It is desirable that the areas in the target space captured by each estimation extraction window have the same size. That is, preferably, the congestion degree estimation unit 50 performs monitoring image so that the area in the monitoring space captured by any pixel of the monitoring image has the same size by homography conversion using the camera parameters of the imaging unit 2. Then, the estimation feature quantity is extracted. As will be described later, in this embodiment, camera parameters are stored in the object model storage unit 43, and the congestion degree estimation unit 50 reads and uses them.

  The density estimator can be realized by a classifier that identifies multi-class images, and can be a discrimination function learned by a multi-class SVM (Support Vector Machine) method.

Density, for example, there is no human "Background" class is 0 people / m higher than ² is two / m ² or less "low density" class, higher than two / m ² 4 persons / m ² or less It can be defined as 4 classes of “medium density” class, “high density” class higher than 4 persons / m ² .

  That is, the density estimator applies the multi-class SVM method to the feature quantities of a large number of images (density images) belonging to the “background” class, “low density” class, “medium density” class, and “high density” class. This is an identification function for discriminating the images of each class from other classes. The parameters of the discriminant function derived by this learning are stored as a density estimator. The feature amount of the density image is the same type as the estimation feature amount and is a GLCM feature.

  The congestion degree estimation means 50 acquires the estimated density which is the output value by inputting each of the estimation feature quantities extracted corresponding to each pixel to the density estimator. In addition, when the monitoring image is deformed and the estimation feature amount is extracted, the congestion degree estimation unit 50 deforms the density distribution to the shape of the original monitoring image by homography conversion using the camera parameter.

  The degree of congestion is a value given in advance to each class of estimated density, and in this embodiment, the values corresponding to each class are “background”, “low congestion”, “medium congestion”, and “high congestion”. write.

  The congestion degree storage means 41 stores the congestion degree corresponding to each monitoring image in time series in association with the area ID of the monitoring image. In the present embodiment, the degree of congestion corresponding to each monitoring image is composed of the degree of congestion of each pixel of the monitoring image. The congestion degree storage unit 41 stores at least a congestion degree obtained from a monitoring image in a past period τ described later.

  The monitoring importance calculation means 51 calculates the monitoring importance, which is information indicating the degree to which the observer or the like should pay attention to the object on the basis of the monitoring image, and the calculated monitoring importance is obtained from the distribution image generation means 53 and the necessary elements. It outputs to the gaze information output means 31.

  The calculation of the monitoring importance is performed for each image section of the monitoring image, and further for each monitoring area (that is, for each photographing unit 2).

  The monitoring importance level is calculated based on the congestion level, the congestion duration, and the amount of movement of the object. That is, the higher the degree of congestion, the higher the congestion duration, the higher the degree of monitoring, and the higher the importance of monitoring for each image section of the monitoring image. Further, for each monitoring area, the higher the congestion degree, the higher the congestion duration time, the higher the importance, and the lower the object movement amount, the higher the monitoring importance degree is calculated.

An example in which the monitoring importance is calculated for each image section (that is, an example in which the monitoring importance is calculated for each attention area set corresponding to each image section in the target space) will be described. Here, the image section is assumed to be a pixel. For example, as shown in the following equations (1) to (3), the monitoring importance S _T at each time point at time t = T is a period [T−τ, T ] (That is, the average value D _T of the representative values _dt of the congestion degree in the pixel in question T−τ ≦ t ≦ T) and the magnitude (speed) of the motion vector in the pixel in the period [T−τ, T]. ) Weighted sum of v _t and average value V _T.

S _T = α · D _T + β · V _T (1)
D _T = (1 / τ) Σ _{T−τ ≦ t ≦ T} d _t (2)
V _T = (1 / τ) Σ _{T−τ ≦ t ≦ T} v _t (3)

  Here, the weighting factors α and β are constants larger than 0, and are determined in advance based on, for example, preliminary experiments.

The representative value of the congestion level is defined by density, the representative value of “background” and “low congestion” is 0 person / m ² , the representative value of “medium congestion degree” is 3 person / m ² , and “high congestion” A representative value of “degree” may be 6 people / m ² . The representative value of “background” may be 0 person / m ² , while the representative value of “low congestion” may be 1 person / m ² .

τ is determined in advance through experiments and the like. The _DT calculated by the equation (2) basically becomes a higher value as the degree of congestion is higher, and becomes a higher value as the duration of the degree of congestion within the period [T−τ, T] is longer. That is, in the example described above, which then sets an upper limit of the impact duration to severity S _T monitoring time tau, tau can be set in consideration of this point. For example, τ can be 1 minute.

  In addition, as described later, by calculating the representative value of the monitoring importance calculated for each image section for each object area, the monitoring importance is calculated for each attention area set corresponding to each object area in the target space. You can also

  Next, an example of calculating the monitoring importance for each monitoring area (that is, an example of calculating the monitoring importance for each attention area set corresponding to each monitoring area in the target space) will be described. The monitoring importance of each monitoring area can be obtained by integrating the monitoring importance calculated for each image section in the monitoring image obtained by photographing the monitoring area, and thereby the monitoring importance for each monitoring area is calculated.

  Here, as a method of integrating the monitoring importance in the monitoring area, it is possible to adopt a configuration in which the monitoring importance for each image section is averaged over the entire monitoring image. On the other hand, only a part of the image sections of the monitoring image is used. However, if the monitoring importance level is high, it is possible to increase the monitoring importance level of the corresponding monitoring area so as to encourage the monitoring staff to watch. This configuration is preferable in that monitoring omissions are easily suppressed. . Specifically, the maximum value is selected from the monitoring importance levels for each image section of the monitoring image corresponding to each monitoring area, and the selected maximum value is subjected to threshold processing to three levels “low”, “medium”, “ It is possible to determine the level of monitoring as the monitoring importance level of the monitoring area. Further, after the monitoring importance for each image section corresponding to each monitoring area is divided into three levels, the maximum level of the plurality of image sections may be determined as the monitoring importance of the monitoring area. Good.

The monitoring severity S _T of the aforementioned formula (1), the movement amount of the object is reflected by the motion vector. The motion vector that is the basic information of the amount of motion can be detected by optical flow analysis. For example, a motion vector is calculated between the current image and the previous image by optical flow analysis.

  The monitoring importance calculation means 51 sets a plurality of attention points on one image (reference image) of the current image and the immediately preceding image, and the image characteristics of each attention point are the most similar in the other image (comparison image). The most similar point is detected as a corresponding point, and a vector connecting the attention point and the corresponding point is calculated as a motion vector. Incidentally, of the attention point and the corresponding point, the immediately preceding image side is the start point of the motion vector, and the current image side is the end point.

  The point of interest and the corresponding point can be set at the center by dividing the monitoring image into a plurality of predetermined blocks. The size of the block is set at least smaller than the person who is the target object. Further, in consideration of the distance from the photographing unit 2 to the object, the block may be smaller as the distance is longer and larger as the distance is shorter. When an image section is a pixel, a motion vector detected for each block is used as a motion vector for each pixel included in the block.

  For example, the monitoring importance calculation means 51 calculates the similarity of the luminance distribution with the point of interest while moving the candidate point on the comparison image, and determines the candidate point with the highest similarity as the corresponding point. For this search, various steepest descent methods such as the Lucas-Kanade method can be applied. That is, the search is performed by sequentially moving the candidate points in the direction in which the difference in luminance is reduced, with the same coordinates as each point of interest as the initial value of the candidate points.

  In another method, each coordinate in a predetermined search range including the coordinates of the point of interest is set as a candidate point for searching. In this case, although the search range can be the entire comparison image, a distance on the image where the object can move in one time is estimated in advance, and a circle with the distance as a radius around the coordinates of the point of interest The inside should be the search range.

  A zero vector is also detected as a motion vector. In addition, when only the most similar point having a degree of similarity equal to or less than a threshold with respect to the attention point is found, it can be assumed that the motion vector corresponding to the attention point is not detected.

  The monitoring importance degree calculation unit 51 stores the motion vector for each image section detected from each monitoring image and each monitoring image in the motion amount storage unit 42. The motion vector is stored at least for the period τ that goes back in the past, and the monitoring image is stored at least one hour before.

  The object model storage unit 43 stores an object model that approximates the shape of the object in advance, and outputs the stored object model to the object region estimation unit 52.

  FIG. 3 is a schematic diagram showing an example of object model information stored in the object model storage unit 43. In the example of FIG. 3, the object model is a three-dimensional model composed of three spheroids corresponding to the head, torso, and legs of a standing person. By the way, the center of gravity of the head is the representative position of the person. The solid model can be simplified and the whole person can be represented by a single spheroid. For example, the human head, torso, both arms and both legs can be separated into separate spheroids. It can also be expressed as

  Further, the object model storage unit 43 stores the camera parameters of the photographing unit 2 in order to project the stereoscopic model onto the coordinate system of the monitoring image together with the stereoscopic model. The camera parameters include external parameters such as the installation position and imaging direction of the imaging unit 2 in the actual monitoring space, internal parameters such as the focal length, field angle, lens distortion, and other lens characteristics of the imaging unit 2 and the number of pixels of the imaging device. Information.

  The drawing setting storage unit 44 stores in advance settings regarding how to display the monitoring importance level and the congestion level included in the information when the information generated by the image processing unit 5 is displayed on the display unit 8. . FIG. 4 is a schematic diagram showing an example of the setting.

  The drawing setting storage means 44 stores in advance graphics (icons) that differ for each level in correspondence with the level of monitoring importance. FIG. 4A is an explanatory diagram showing an example of the icon, and two types of icons are stored corresponding to each level. Specifically, the two types of icons are facial expressions and emotions. The monitoring importance level “low” is associated with an icon that reminds you of a safe state, an expression of a smile, and an illustration of an eighth note used in comics or the like to express a pleasant mood. Remembered. In addition, the level of monitoring importance “medium” is stored in association with an unsatisfied facial expression and a spiral illustration known as a comic symbol when expressing a moist feeling, as a reminder of a state to watch out for, At the high level of importance of monitoring, an expression of a distressed face and an illustration known as a cross-shaped anger mark of a comic book are stored in association with each other as a reminder of a dangerous state.

  In addition, the drawing setting storage unit 44 stores in advance colors that differ depending on the degree of congestion corresponding to the degree of congestion. FIG. 4B is an explanatory diagram showing an example of setting the color. The colors corresponding to the congestion levels “low”, “medium”, and “high” are displayed by the monitor who visually observes the display on the display unit 8. It is set differently so that it can be identified. Considering that the danger increases as the congestion level increases, green, yellow, and red are set corresponding to the congestion levels “low”, “medium”, and “high”, respectively, and the drawing setting storage unit 44 stores the congestion level “ A green color code corresponding to “low”, a yellow color code corresponding to “medium”, and a red color code corresponding to “high” are stored.

  The object area estimation unit 52 estimates the area (object area) of each object from the monitoring image, and outputs the estimated object area to the distribution image generation unit 53. Specifically, the object area estimation means 52 estimates the area of each person being photographed as the object area.

  For example, the object region estimation unit 52 performs background difference processing on the monitoring image obtained from the photographing unit 2 using a background image that is generated in advance from a monitoring image at the time of unattended and stored in the storage unit 4, A pixel whose absolute value of the difference between pixel values is equal to or greater than a threshold value is detected, and a group of detected pixels is extracted as a change area.

  Subsequently, the object region estimation means 52 performs an edge image generation process. In other words, an edge operator is applied to the monitoring image at the position of each pixel in the change region, the edge intensity is calculated for each pixel, and binarized with a predetermined threshold value to generate an edge image.

  Subsequently, the object region estimation means 52 performs person position estimation processing by model matching. That is, the object region estimation unit 52 generates a shape model imitating a human shape by projecting the three-dimensional model stored in the object model storage unit 43 onto the coordinate system of the monitoring image, A plurality of patterns are arranged in a plurality of ways, the degree of coincidence with the edge image is calculated for each arrangement, and the arrangement having the maximum degree of coincidence is specified. Then, the object region estimation means 52 estimates each position of the shape model in the specified arrangement as a person position. Note that the degree of coincidence can be the coincidence ratio of the position of the contour pixel of the shape model and the edge pixel whose edge strength is equal to or greater than a predetermined threshold.

  At that time, the object region estimation means 52 refers to the two-dimensional information (congestion degree image) of the congestion degree corresponding to the monitoring image, and for each congestion degree, in the area where the congestion degree is estimated, the congestion is included in the change area. By arranging the number of shape models within the range indicated by the degree, it is possible to prevent excessive duplication of the shape models and improve the estimation accuracy of the person position. Further, the object region estimation means 52 refers to the congestion degree image, and places a shape model imitating the shape of a person's whole body in the region where the low congestion degree is estimated, and the region where the medium congestion degree is estimated. Decrease the degree of coincidence due to occlusion by placing a shape model simulating the shape of the upper body of the body and locating a shape model simulating the shape of the range from the human head to the shoulder in an area where high congestion was estimated It is also possible to improve the estimation accuracy of the person position.

  The distribution image generation unit 53 receives the monitoring importance for each image section from the monitoring importance calculation unit 51, generates a distribution image representing the distribution of the monitoring importance in the monitoring image, and outputs the distribution image to the gaze information output unit 31. To do.

  For example, the distribution image generation unit 53 refers to the object region input from the object region estimation unit 52 and the color settings and icons stored in the drawing setting storage unit 44, and each object region is displayed in a color corresponding to the degree of congestion. And an icon corresponding to the monitoring importance level is drawn in each object area.

  At this time, the distribution image generating unit 53 obtains the monitoring importance for each object region by integrating the monitoring importance for each object region. For example, the maximum value of the monitoring importance value for each image section included in each object region is set as the monitoring importance of the object region. Further, the average value or the mode value of the monitoring importance level for each image section included in each object area can be set as the monitoring importance degree of the object area.

  FIG. 5 is a schematic diagram showing an example of a distribution image. In the distribution image 200, for the convenience of expression, the green fill is represented by a horizontal line, the yellow fill is represented by a diagonal line, and the red fill is represented by shading.

  In the distribution image 200, the shape model 201 is displayed in a green color at the estimated position of each of the human images in the area of “low” congestion. In the example of FIG. 5, the monitoring importance of the area is “low” because the monitoring importance is likely to be low in the area of the congestion level “low”, and the position corresponding to the face of the shape model 201 and its vicinity In addition, an expression 211 and an illustration 221 are drawn as icons representing the monitoring importance.

  In addition, the shape model 202 is displayed in yellow at the estimated position of each of the human images in the area of the congestion degree “medium” in the distribution image 200. In addition, an icon representing the monitoring importance calculated for the area is displayed at a position corresponding to the face of the shape model 202 and in the vicinity thereof. In the example of FIG. 5, it is assumed that the monitoring importance is “medium” in the area, and the facial expression 212 and the illustration 222 corresponding to this are drawn.

  In the region of the congestion level “high” in the distribution image 200, the shape model 203 is displayed in red at the estimated position of each human image. In addition, an icon representing the monitoring importance calculated for the area is displayed at a position corresponding to the face of the shape model 203 and in the vicinity thereof. In the example of FIG. 5, it is assumed that the monitoring importance is “high” in the area, and a facial expression 213 and an illustration 223 corresponding to this are drawn.

  In the distribution image 200, the shape of a person is simulated corresponding to each estimated position of the person's image, and a shape model filled with a color corresponding to the degree of congestion is drawn. The correspondence is easy to grasp intuitively. In the example of FIG. 5, the shape model is based on a three-dimensional model that is more complicated than the three-dimensional model composed of the three spheroids shown in FIG.

  Further, in the distribution image 200, since the monitoring importance is displayed using, for example, an icon, the monitoring efficiency by a monitor or the like can be improved.

  Preferably, the distribution image can be generated by transparently synthesizing the image in which the object region represented by the shape model and the icon are drawn as described above with reference to FIG. In addition to the specific situation of the monitoring area based on the monitoring image, the degree of congestion and the importance of monitoring can be grasped, and the monitoring efficiency can be further improved.

  The gaze information output means 31 displays, for each monitoring image, a distribution image of the monitoring importance generated from the monitoring image and information in which the monitoring importance of the monitoring area is associated with the area ID of the monitoring image. Send to 7 address.

  Next, the operation of the image monitoring system 1 will be described. FIG. 6 is a schematic flowchart of processing for calculating the monitoring importance.

  In the image monitoring system 1, the communication unit 3 functions as the image acquisition unit 30 and acquires monitoring images from the plurality of imaging units 2 (2-1, 2-2,...) (Step S100). The image acquisition unit 30 inputs the monitoring image to the image processing unit 5 every time the monitoring image is acquired from any of the imaging units 2-1, 2-2,..., And the image processing unit 5 acquires the monitoring image. Every time the processing of steps S101 to S109 is performed, the monitoring importance is obtained from the monitoring image. Incidentally, the monitoring image is acquired together with the area ID, and the monitoring area captured in the monitoring image is specified from the area ID.

  When the monitoring image is acquired, first, the image processing unit 5 functions as the congestion level estimation unit 50, and the storage unit 4 functions as the density estimator storage unit 40 and the congestion level storage unit 41, respectively, and estimates the congestion level for each image section. And the degree of congestion is stored (step S101). Specifically, the congestion degree estimation unit 50 reads the density estimator from the density estimator storage unit 40, and applies the density estimator to the position of each pixel of the monitoring image at the current time acquired in step S100, for each pixel. The congestion degree is estimated, and the congestion degree for each pixel is additionally stored in the congestion degree storage unit 41.

  Next, the image processing unit 5 functions as the monitoring importance calculation unit 51, and the storage unit 4 functions as the motion amount storage unit 42, and detects the motion amount for each image section and stores the motion amount (step S102). ). Specifically, the monitoring importance degree calculation unit 51 reads the monitoring image of the previous time from the movement amount storage unit 42 and the monitoring image of the previous time that has the same area ID as the monitoring image of the current time. The optical flow for each pixel is detected as a motion amount between the monitoring image and the motion amount for each pixel and the monitoring image at the current time are additionally stored in the motion amount storage unit 42 in association with the area ID. Note that if the area of the monitoring image where the degree of congestion is “low” or higher is not estimated (that is, only the density of the background class), the motion amount of all pixels may be set to zero.

Subsequently, the monitoring importance calculation unit 51 calculates the monitoring importance for each image section (step S103). Specifically, the monitoring importance degree calculation means 51 reads out the congestion degree estimated over the past period τ from the current time from the congestion degree storage means 41, and the motion amount detected over the period τ is the movement amount storage means 42. read from, and calculates the importance S _T monitor at the position of each pixel by expression (1) to (3) using these congestion d _t and the amount of movement v _t for each pixel.

  Subsequently, the image processing unit 5 functions as the object region estimation unit 52 if a region having a congestion level of “low” or higher is estimated in the monitoring image at the current time (“YES” in step S104), The object area is estimated (step S105). At that time, the storage unit 4 functions as the object model storage unit 43. Specifically, the object region estimation unit 52 generates an edge image from the monitoring image at the current time, reads out the stereo model and camera parameters from the object model storage unit 43, and sets a plurality of positions at a plurality of positions. The combination is projected onto the edge image, the degree of coincidence between the projection image of each combination and the edge image is calculated, and the combination having the highest degree of coincidence is specified. And each area | region which projected the stereo model in this specified combination is estimated as an object area | region. The object region estimation unit 52 outputs the estimated object region to the monitoring importance degree calculation unit 51.

  The monitoring importance calculation unit 51 integrates the monitoring importance for each image section obtained in step S103 for each object region input from the object region estimation unit 52, and calculates the monitoring importance for each object region. (Step S106).

  If an area having a congestion level of “low” or higher is not estimated and the entire monitoring image at the current time is estimated as the background (in the case of “NO” in step S104), steps S105 and S106 are skipped. Is done.

  After the processing up to step S106, the image processing unit 5 functions as the distribution image generation unit 53 and the storage unit 4 functions as the drawing setting storage unit 44, and generates a distribution image (step S107). Specifically, when the monitoring importance degree for each object area is calculated, the distribution image generation unit 53 obtains the mode value of the congestion degree in each object area as the congestion degree in the object area, and for each object area. Further, a color code corresponding to the degree of congestion of the object area is read from the drawing setting storage means 44, and each object area is drawn with the color indicated by the color code. Further, for each object region, the level of the monitoring importance of the object region is determined by threshold processing, the level is determined, and an icon corresponding to the level is read from the drawing setting storage unit 44 and drawn in the object region.

  If the monitoring importance level for each object area is not calculated, the distribution image generation unit 53 draws and stores a color code corresponding to the congestion level of each pixel for pixels for which the congestion level of “low” or higher is estimated. Read from the means 44 and draw each pixel in the color indicated by the color code. In addition, for pixels for which the monitoring importance level of “low” or higher is calculated, adjacent pixels having the same monitoring importance level are combined into an integrated region, and the level of monitoring importance is determined for each integrated region to determine the level, An icon corresponding to the level is read from the drawing setting storage means 44 and drawn in the integrated area.

  The distribution image generation unit 53 generates a distribution image by transparently synthesizing the above drawing result with the monitoring image at the current time, and outputs the generated distribution image to the communication unit 3 together with the area ID.

  On the other hand, the monitoring importance calculation means 51 calculates the monitoring importance for each monitoring area (step S108). Specifically, the monitoring importance level calculation unit 51 monitors the monitoring area indicated by the area ID of the monitoring image with the maximum value of the monitoring importance levels calculated for each pixel of the monitoring image at the current time in step S103. Calculated as importance. The monitoring importance level calculation means 51 outputs the calculated monitoring importance level to the communication unit 3 together with the area ID.

  The communication unit 3 to which the distribution image and the area ID are input from the distribution image generation unit 53 and the monitoring importance and area ID of the monitoring area are input from the monitoring importance calculation unit 51 functions as the gaze information output unit 31. The input distribution image, monitoring importance, and area ID are output to the display control unit 7 (step S109).

  When the image monitoring system 1 obtains the monitoring importance for the monitoring image acquired in step S100 and outputs it to the display control unit 7 in step S109, the image monitoring system 1 returns to step S100, and steps S100 to S109 for the next acquired monitoring image. Repeat the process.

  FIG. 7 is a schematic flowchart of processing for displaying the monitoring importance.

  A storage unit included in the display control unit 7 stores a display mode, a display area, and monitoring importance information input in the past.

  The display mode is either a single display mode for displaying one monitoring area or a multiple display mode for displaying a plurality of monitoring areas. The display mode is specified by a mode identifier set in advance, and the mode identifier is stored as a display mode in the storage unit.

  The display area is specified by an area ID in the single display mode, and is specified by a group ID set in advance for a combination of a plurality of area IDs in the multi display mode. The storage unit stores an area ID for a single display mode and a group ID for a multi display mode as display area settings. It is assumed that the input from the operation unit 6 is either the display mode or the display area.

  In normal times, confirmation of the presence or absence of an operation input from the monitor by the operation unit 6 (step S200) and confirmation of the presence or absence of input of monitoring importance level information from the communication unit 3 by the display control unit 7 (step S201). The display control unit 7 is in a standby state in which display processing is not performed when there is no operation input and no monitoring importance level information is input (in the case of “NO” in steps S200 and S201). .

  When an operation input is performed by the monitor (in the case of “YES” in step S200), the display control unit 7 confirms whether the input content is the display mode or the display area (step S202).

  If it is an input of the display mode (in the case of “YES” in step S202), the display control unit 7 updates the mode identifier stored in the storage unit to the input value (step S203), and the process is performed. The process proceeds to S206.

  On the other hand, if the input is a display area (NO in step S202), display control unit 7 updates the area ID or group ID stored in the storage unit to the input value (step S204). ), The process proceeds to step S206.

  When the monitoring importance level information is input from the communication unit 3 (in the case of “YES” in step S201), the display control unit 7 stores the input monitoring importance level information in the storage unit. (Step S205), the process proceeds to Step S206.

  When the standby state is released in this way, the display control unit 7 confirms the setting of the storage unit (step S206). If the display mode is the multi display mode ("YES" in step S206), the display area A distribution image of a plurality of monitoring areas associated with the group ID indicated by the setting is displayed on the display unit 8 together with their monitoring importance (step S207).

  FIG. 8 is a schematic diagram illustrating a display example of the display unit 8 in the multi-display mode. In FIG. 8, the area IDs constituting the group ID are 1 to 4, and on the screen of the display unit 8, icons indicating the monitoring importance of the monitoring boundary are displayed along with the distribution images for each of the four area IDs. For “Area 1” and “Area 3”, an icon of a facial expression whose monitoring importance is “low” is displayed, and for “Area 2”, an icon whose monitoring importance is “medium” is displayed. For “Area 4”, an icon whose monitoring importance is “High” is displayed. From this display, the monitor can quickly grasp that the measures should be taken in the order of “Area 4” and “Area 2”.

  On the other hand, when the display mode is the single display mode (in the case of “NO” in step S206), the distribution image of the monitoring area associated with the area ID indicated by the display area setting is displayed on the display unit 8. (Step S208). The monitoring person can confirm the detailed status of the displayed monitoring area from the distribution image, and can determine the location in the monitoring area where the security guard is to be dispatched, and the like.

  When the display is updated in this way, the process returns to step S200 again, and enters a standby state.

[Modification]
(1) In the above-described embodiment and its modification, an example is shown in which the monitoring importance calculation unit 51 calculates the monitoring importance for each image section of the monitoring image and further calculates the monitoring importance for each monitoring region. However, either one of the monitoring importance levels may be calculated.

  Here, as described above, the congestion degree estimation means 50 estimates the congestion degree indicating the degree of congestion of the object for each attention area set in the target space based on the monitoring image. When only the importance level is calculated, the monitoring area is the attention area.

  When calculating only the monitoring importance level for each monitoring area, the congestion level estimation unit 50 pays attention to the edges of the monitoring image (four sides if the monitoring image is rectangular) instead of estimating the congestion level using the density estimator. The number of people entering the monitoring area and the number of people coming out of the monitoring area can be measured, and the degree of congestion can be estimated from the number obtained by subtracting the number of people who entered the monitoring area.

(2) The monitoring importance calculation means 51 may sequentially calculate D _T and V _T using the following expressions (4) and (5) instead of the expressions (2) and (3). In Expressions (4) and (5), the time and period are defined by the imaging cycle of the monitoring image, time t = T is the time when the current image was captured, and time t = T−1 is the immediately preceding time. Indicates the time when the image was taken. The period τ is represented by the number of times the monitoring image is captured.

D _T = (τ−1) / τ · D _T−1 + 1 / τ · d _T (4)
V _T = (τ−1) / τ · V _T−1 + 1 / τ · v _T (5)

In this case, the congestion degree storage unit 41 only needs to store D _T-1 , and the motion amount storage unit 42 only needs to store V _T-1 and past monitoring images necessary for calculating v _T. Therefore, the capacity of the storage unit 4 can be reduced.

(3) The monitoring importance calculation means 51 detects the time τ in which the high-density class or the medium-density class is continuously estimated from the time T toward the past without setting the time τ as a fixed value, and the detected time the τ equation (2), (3) applying to calculate the _{D T} and _{V T} are, or detected time τ wherein the (4), calculating the _{D T} and _{V T} is applied to (5) You can also.

(4) As a mode in which the process is simplified, the monitoring importance calculation unit 51 obtains a time τ in which the high-density class or the medium-density class is continuously estimated from the time T toward the past. it may calculate the monitoring importance S _T with τ length of the time instead of D _T Te. This aspect corresponds to defining the representative value _dt of the congestion level as a binarized value depending on whether the congestion level is less than the medium density or higher than the medium density. In this aspect, the higher the congestion level, the higher the monitoring importance level. _ST is calculated. Similarly, the length of time τ in which high-density classes are continuously estimated from time T toward the past may be used in place of _DT in equation (1).

  (5) In the above-described embodiment and its modification, the monitoring importance calculation means 51 has shown an example in which the monitoring importance is calculated based on the congestion degree, the congestion duration, and the amount of movement of the object. The importance level calculation means 51 may calculate the monitoring importance level from the congestion level and the duration of the congestion by omitting the amount of movement, or monitor important from the congestion level and the amount of movement of the object by omitting the duration. The degree may be calculated.

  (6) In the above-described embodiment and its modification, the example in which the monitoring importance calculation unit 51 detects a motion vector by optical flow analysis has been described. However, an area where the degree of congestion is greater than or equal to a predetermined level (for example, a high-density class) With respect to the estimated region), it is also possible to detect a motion vector in each segment by performing spatiotemporal segmentation on a time-series image composed of monitoring images sequentially taken over a predetermined time length.

  Here, the spatiotemporal segmentation is a process of generating a spatiotemporal segment that is a group of pixels that are adjacent to each other and have similar pixel values (color or density) in the spatiotemporal space formed by the time series image. Incidentally, a time-series image has coordinates (x, y, and y) on the coordinate axes (x-axis and y-axis) and the time-axis (t-axis) of a two-dimensional image (spatial image) at each time constituting the time-series image. t) can be considered as a three-dimensional image in the defined space-time, and the adjacent / neighbor relationship of the pixels is the spatial pixel positional relationship in the x-axis and y-axis directions and the temporal pixel in the t-axis direction. It is grasped from both sides of the positional relationship.

  Spatio-temporal segmentation requires the overlapping of moving object images between spatial images with continuous time, so it is preferable to apply it to areas where there is little movement of moving objects. It is supposed to be applied to the area.

  The range (analysis section) in the time axis direction of the time-series image for performing spatiotemporal segmentation is basically finite from the viewpoint of processing load, and the time length (analysis section length) is set in advance. For example, when calculating a motion vector in a highly congested area using spatio-temporal segmentation, the analysis interval length can be set to 5 hours. In this case, the monitoring importance level calculation unit 51 stores the past 4 times from the storage unit 4. In the highly congested region of the spatiotemporal image in which the monitoring images at (t = T−4 to T−1) are read and the monitoring images at the current time (t = T) are arranged in time order, the pixel position and the shooting time are adjacent. The pixel value difference degree is calculated between the pixels to be processed, and if the difference degree is small, the process of combining the same segment is repeated to divide the pixel value into a plurality of spatiotemporal segments.

  Then, assuming that a group of pixels having the same time among the pixels included in each spatiotemporal segment is called an intercept, the monitoring importance calculation means 51 obtains, for example, the centroid of each intercept for five times in each spatiotemporal segment. Then, a motion vector is calculated by linearly approximating the temporal change of the gravity center position for each spatiotemporal segment.

  By using spatio-temporal segmentation in which adjacent pixels in the temporal direction are combined in this way, motion vector calculation errors caused by temporal fluctuations in pixel values due to illumination fluctuations can be reduced.

  (7) In the above-described embodiment and its modified example, the distribution image generating unit 53 generates the distribution image by integrating the monitoring importance for each object region. However, the monitoring importance for each image section itself is shown. May be generated. In this case, for example, the image sections having the same monitoring importance can be connected to each other and the marks can be superimposed on the center of gravity of the connection area, or the marks can be overlapped at equal intervals in the connection area.

  (8) In the above-described embodiment and its modification, an example in which the image section is each pixel and the degree of congestion is estimated for each pixel has been described. However, an image section including a plurality of pixels may be used. For example, the monitoring image is divided into a plurality of blocks, and each block is used as an image section. In this case, it is preferable to predetermine that the block is smaller than the estimation extraction window, and the congestion degree estimation means 50 sets the estimation extraction window at the barycentric position of each block to estimate the congestion degree for each image section. . For example, a block of 3 × 3 pixels is set as an image section, an estimation extraction window is set every three pixels, and an estimation feature amount is extracted.

  (9) In each of the above-described embodiments and modifications thereof, an example in which the object to be detected is a person has been shown. However, the present invention is not limited thereto, and the object to be detected is a vehicle, an animal such as a cow or a sheep, or the like. You can also.

  (10) In each of the above-described embodiments and modifications thereof, the density estimator learned by the multi-class SVM method is illustrated. However, instead of the multi-class SVM method, a decision tree-type random forest method, a multi-class Various density estimators such as a density estimator learned by the AdaBoost method or the multi-class logistic regression method can be used.

  Alternatively, a density estimator using a discriminating CNN (Convolutional Neural Network) may be used.

  (11) In each of the above-described embodiments and modifications thereof, the classes of density other than the background estimated by the density estimator are set to three classes, but the classes may be divided more finely.

  (12) In each of the above-described embodiments and modifications thereof, a density estimator that classifies into multiple classes is illustrated, but instead, a regression type density estimation that regresses a density value (estimated density) from a feature value. It can also be a container. That is, a density estimator that has learned the parameters of the regression function for obtaining the estimated density from the features by ridge regression method, support vector regression method, regression tree-type random forest method or Gaussian Process Regression, etc. can do.

  Alternatively, a density estimator using a regression type CNN may be used.

  (13) In each of the above-described embodiments and modifications thereof, the GLCM feature is exemplified as the feature amount learned by the density estimator and the estimation feature amount. However, in place of the GLCM feature, the local binary pattern (Local Binary Pattern (LBP) feature value, Haar-like feature value, HOG feature value, luminance pattern, and other various feature values, or a combination of GLCM features and a plurality of them You can also

  DESCRIPTION OF SYMBOLS 1 Image monitoring system, 2 imaging | photography part, 3 communication part, 4 memory | storage part, 5 image processing part, 6 operation part, 7 display control part, 8 display part, 30 image acquisition means, 31 gaze information output means, 40 density estimation Device storage means, 41 congestion degree storage means, 42 motion amount storage means, 43 object model storage means, 44 drawing setting storage means, 50 congestion degree estimation means, 51 monitoring importance degree calculation means, 52 object area estimation means, 53 distribution image Generation means.

Claims (5)

An image monitoring system that generates information indicating a degree to which the object should be watched from a monitoring image obtained by capturing a space in which a predetermined object may be crowded,
Image acquisition means for sequentially acquiring the monitoring images;
A congestion degree estimating means for estimating a congestion degree representing a degree of congestion of the object in the space from the monitoring image;
Monitoring importance calculation means for calculating a duration of congestion of the object in the attention area set in the space, and calculating a higher monitoring importance as the congestion degree is higher and the duration is longer for the attention area;
An image monitoring system comprising:   The monitoring importance calculation means further calculates a movement amount of the object in the attention area from the monitoring image, and calculates a higher monitoring importance as the movement amount is smaller for the attention area. Item 8. The image monitoring system according to Item 1. An image monitoring system that generates information indicating a degree to which the object should be watched from a monitoring image obtained by capturing a space in which a predetermined object may be crowded,
Image acquisition means for sequentially acquiring the monitoring images;
A congestion degree estimating means for estimating a congestion degree representing a degree of congestion of the object in the space from the monitoring image;
A monitoring importance calculation unit that calculates the amount of motion of the object in the attention area set in the space from the monitoring image, and calculates a higher monitoring importance for the attention area as the congestion degree is higher and the movement amount is smaller. When,
An image monitoring system comprising: The image acquisition means sequentially acquires the monitoring images for each of a plurality of different monitoring areas set in the space,
Making each of the monitoring areas the attention area;
The image monitoring system according to any one of claims 1 to 3, wherein: The space shown in each of a plurality of image sections into which the monitoring image is divided is the attention area,
Furthermore, the image processing apparatus further comprises distribution image generation means for generating a distribution image expressing the distribution on the monitoring image by calculating the monitoring importance for each image section.
The image monitoring system according to any one of claims 1 to 3, wherein:

Images