JP2012205097A - Video processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately give each frame of monitoring video a degree at which it should be watched.SOLUTION: An object tracking unit 33 tracks a moving object appearing in each frame of monitoring video 22, and detects an object image of the moving object in a tracking frame. An object feature extraction unit 34 extracts, from the object image, its feature amount. An importance calculation unit 35 statistically analyzes the feature amounts obtained from respective tracking frames for each moving object with respect to monitoring of singularity of a moving object to calculate an individual representative amount representing the feature amounts, and calculates, for each tracking frame, the importance of the tracking frame corresponding to distances between the feature amounts of respective moving objects and the individual representative amounts of the moving objects.

Description

  The present invention relates to a video processing apparatus that gives a degree to be watched to each frame of a monitoring video.

  Conventionally, the process of detecting a suspicious behavior of a person from a monitoring video has been performed using a common reference for each person.

  In the intruding object monitoring device described in Patent Document 1, an intruding object in a monitoring video is detected and the amount of movement thereof is calculated, and an input image when the amount of movement exceeds a predetermined amount is recorded as a digest video.

  Moreover, in the abnormal action detection apparatus described in Patent Document 2, a comparison reference is created by learning normal action patterns of a plurality of moving objects.

JP 2004-254141 A JP 2010-72782 A

  However, in the related art, if the suspicious movement temporarily taken by a specific person is similar to the movement taken by the majority of persons, it may not be determined as an important scene. For example, a scene in which a specific person who has been running shortly while a large number of people are walking temporarily decreases the speed cannot be determined as an important scene.

  At this time, the deceleration that the specific person temporarily took is an important scene that should be particularly watched as suspicious movements, while the movement that is different from the majority of people that the specific person is running is also watched next. It is desired to determine that it is an important scene to be performed.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a video processing apparatus that can appropriately determine an important scene to be watched for monitoring. Further, based on the fact that the important scene can be discriminated, a summary video shorter than the original monitoring video can be created.

  A video processing apparatus according to the present invention generates information on a frame to be watched in a monitoring video composed of a plurality of frames, and stores a storage unit for storing the monitoring video, and a moving object in the monitoring video by image analysis. A tracking frame in which the moving object is tracked and an object tracking unit for generating correspondence information between the moving object and the object image of the moving object, and features of movement or / and appearance of the moving object from the object image An object feature extraction unit for extracting a quantity; and a statistical analysis of the feature quantity for each moving object to obtain an individual representative quantity representing the distribution of the feature quantity, and the individual representative quantity of each moving object and the moving object A monitoring value calculation unit that calculates a first distance, which is a distance from the feature amount for each tracking frame, and calculates a higher monitoring value for the tracking frame having a larger first distance.

  In another video processing apparatus according to the present invention, the monitoring value calculation unit statistically analyzes the feature amounts of the plurality of moving objects tracked by the object tracking unit to represent the feature amounts. And a second distance that is a distance between the collective representative amount and the individual representative amount of each moving object is calculated, and tracking is performed using the first distance of the moving object and the tracking frame in each tracking frame. The monitoring value is calculated according to the second distance of the moved moving object.

  In another video processing apparatus according to the present invention, the monitoring video is further thinned out by preferentially selecting the tracking frame having a higher monitoring value from the monitoring video, so that the number of the monitoring video is less than the total number of frames of the monitoring video. A summary video creation unit that creates a summary video of the set number of frames is provided.

  In the video processing apparatus according to another aspect of the invention, the summary video creation unit may obtain at least one of the start frame and the end frame in a series of the tracking frames of the moving objects as the monitoring value thereof. Select regardless of.

  In still another video processing apparatus according to the present invention, the summary video creation unit uses dynamic programming to maximize the sum of the monitoring values of the tracking frames constituting the summary video.

  According to the present invention, it is possible to suitably determine an important scene to be watched for monitoring. Further, the present invention makes it possible to preferentially select an important scene from a monitoring video and create a summary video shorter than the original monitoring video.

It is a block block diagram of the video processing apparatus which concerns on embodiment of this invention. It is a schematic diagram which shows the structure of the object information regarding the moving object tracked with the monitoring image. It is a table which shows the structure of an object feature-value. It is a schematic diagram which shows an example of the calculation process of the deviation degree in an object. It is explanatory drawing which shows an example of the deviation degree in an object, the deviation degree between objects, and importance. It is a schematic diagram which shows inclination restriction | limiting when the maximum skip number is set to 2 frames. It is a graph which shows the example of the lattice point which can be selected by dynamic programming, and a local path. It is a general | schematic flowchart explaining operation | movement of an image | video summary apparatus. It is a general | schematic flowchart of the process by the importance calculation part.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The present embodiment is a video processing apparatus that can efficiently check an important scene of a monitoring video obtained by imaging a monitoring space for a long time. For example, the monitoring video is an image of a monitoring space such as a store, and a person who moves as a monitoring target exists in the monitoring space. The video processing apparatus 1 reproduces an important scene at an imaging speed or at a low speed and efficiently reproduces other than the important scene at a high speed in order to efficiently view the long-time monitoring video. In particular, the video processing apparatus 1 detects a scene in which a suspicious human action deviating from a normal action is captured as an important scene, and enables quick viewing.

  FIG. 1 is a block diagram of a video processing apparatus 1 according to the embodiment. The video processing device 1 includes a video recording device 2, a video summarizing device 3, and a display device 4. The video recording device 2 and the display device 4 are connected to the video summarizing device 3.

  The video recording device 2 includes an imaging unit 20 and a recording unit 21.

  The imaging unit 20 is a so-called monitoring camera and is connected to the recording unit 21. The imaging unit 20 images the monitoring space at predetermined time intervals, and sequentially outputs the captured images to the recording unit 21. The imaging time interval is, for example, 1/5 second. Hereinafter, each image captured at this time interval is referred to as a frame, and this time interval is referred to as a frame interval.

  The recording unit 21 is configured by, for example, a digital video recorder (DVR) provided with a large-capacity recording medium such as a hard disk drive (HDD). The recording unit 21 can receive a time-series image captured over a long period of several days, weeks, or months from the imaging unit 20 and record it as a monitoring video 22 on a large-capacity recording medium. The monitoring video 22 is data in which a plurality of frames are arranged in the order of imaging. The recording unit 21 is also connected to the video summarizing device 3 and outputs a monitoring video 22 to the video summarizing device 3.

  The video summarizing apparatus 3 includes a setting input unit 30, a storage unit 31, and a signal processing unit 32, and the setting input unit 30 and the storage unit 31 are connected to the signal processing unit 32. The video summarization device 3 is connected to the recording unit 21 and the display device 4, detects an important scene from the monitoring video 22, creates a summary video in which frames other than the important scene are thinned out, and outputs the created summary video to the display device 4. To do. The video summarizing device 3 can be configured using, for example, a personal computer (PC).

  The setting input unit 30 is a user interface device such as a keyboard and a mouse. The setting input unit 30 is operated by a user such as a store manager or a monitor who checks a suspicious behavior in a monitoring space by watching a summary video and inputs various settings. Used. Here, the setting includes an arbitrary section in the monitoring video 22 designated by the user as the original video of the summary video, the number of frames of the summary video designated by the user (specified number of frames), and the like. Hereinafter, the original video designated as the target of the summary process is simply referred to as a monitoring video 22. The designated number of frames is at least smaller than the total number of frames of the monitoring video 22. Instead of directly inputting the designated number of frames by the user, the playback time of the summary video may be input from the setting input unit 30 and the signal processing unit 32 may divide the playback time by the frame interval and convert it to the number of frames. .

  The storage unit 31 is a storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 31 stores various programs and various data, and inputs / outputs such information to / from the signal processing unit 32. Various data includes object information 310 and importance data 311.

The object information 310 is information in which the region information of each object image in the monitoring video 22 and the feature amount related to at least one of the motion and appearance extracted from each object image are associated with the identifier and the frame number of the moving object. The frame number is a number assigned to each frame constituting the monitoring video 22 in the order of imaging. FIG. 2 is a schematic diagram showing the configuration of the object information 310. “Object ID” is an identifier of a moving object, and is a number that uniquely identifies each moving object. “Number of tracking frames” is the total number of frames (tracking frames) in which an object image of each moving object is detected and tracked. “Tracking frame number” is the frame number of the tracking frame. The tracking frame number associated with each moving object from the beginning to the end represents the appearance section of the moving object in the monitoring video 22. The “object feature amount” is one or both of the motion feature amount and the appearance feature amount extracted from the object image of the moving object in the tracking frame of each moving object, and the feature amount in the tracking frame t of the moving object u. _Is expressed as _{ft, u} . The “object area” is an area where an object image of the moving object is detected in the tracking frame of each moving object, and the object image can be specified from the area. The object region of the moving object u in the tracking frame t is denoted as It _{, u} .

  FIG. 3 is a table showing the structure of the object feature amount. The object feature amount according to the present embodiment is a 27-dimensional vector in which appearance feature amounts and motion feature amount vectors are arranged. “Whole edge feature”, “Upper half edge feature”, and “Lower half edge feature” are appearance feature amounts that have the entire object image, the upper half of the object image divided in the Y direction, and the lower half as extraction ranges, respectively. These are four-dimensional vectors obtained by quantizing the edges extracted from all the pixels in the extraction range in four directions and performing histogram analysis of the edge strength for each edge direction.

  The “whole body color feature”, “upper half color feature”, and “lower half color feature” are appearance feature amounts whose extraction ranges are the entire object image, the upper half, and the lower half of the object image, respectively. In this embodiment, the Luv color system is adopted, and the color features having the whole body, the upper half, and the lower half as the extraction ranges are the L component average value, U component average value, and V component average calculated from all the pixels in the extraction range, respectively. It is represented by a three-dimensional vector in which values are arranged.

  “Object size” is also an appearance feature quantity, and is a two-dimensional vector in which the width and height of a circumscribed rectangle of an object image are arranged.

  “Object position” is one of the motion feature quantities, and is a two-dimensional vector in which the X and Y coordinates of the center of gravity of the object image are arranged. “Object velocity” is also one of the motion feature amounts, and is a two-dimensional vector composed of the X coordinate change amount and the Y coordinate change amount of the object position between the preceding and following tracking frames.

The object feature quantity f _{t, u} in the tracking frame t of each moving object u can be simply a feature quantity for each frame, but can also be a moving average. That is, the object feature quantity f _{t, u} can be a 27-dimensional vector obtained by averaging the feature quantities of the total (2p + 1) frames for each component, including p frames before and after each tracking frame t. A natural number smaller than ½ of the minimum length of the appearance interval estimated in advance is set for p. When the moving average is used, the influence of region extraction errors is reduced by smoothing, and false detection of suspicious behavior is reduced.

Furthermore, the object feature quantity f _{t, u} can be a distribution of small sections including a plurality of frames including the tracking frame t. Specifically, a (2p + 1) frame centered on the frame of interest t is set as a small section, and the feature value distribution of the small section is modeled as a normal distribution. The object feature amount _{ft, u} is a 27-dimensional vector averaged for each component of the feature amount in the small section, and a 27 × 27-dimensional covariance matrix generated using the average vector. Here, p is also set to a natural number smaller than ½ of the shortest length of the appearance section estimated in advance. If the distribution of small sections is used, in addition to the effect of reducing false detection of suspicious behavior by reducing region extraction errors, instantaneous changes in appearance feature quantities can also be expressed by object feature quantities _{ft, u} , so that more suspiciousness is achieved. Actions are easier to detect. In the present embodiment, the distribution of this small section is extracted as the object feature amount f _{t, u} and used for calculating the importance described later for each tracking frame.

  The importance data 311 is a tracking frame and a monitoring value (monitoring value) of the tracking frame. This is information in which importance is associated. Note that the importance level data 311 may include information in which the frames of the monitoring video 22 other than the tracking frame are explicitly associated with the lowest importance level (for example, 0).

  The signal processing unit 32 is configured using an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an MCU (Micro Control Unit), and includes an object tracking unit 33, an object feature extraction unit 34, and an importance calculation. The program in which operations of the unit 35, the summary video creation unit 36, and the like are described is read from the storage unit 31 and executed to operate as each unit. The signal processing unit 32 creates a summary video from the monitoring video 22 input from the recording unit 21 and outputs the created summary video to the display device 4.

  The object tracking unit 33 compares successive frames and tracks the object image of the moving object captured in the monitoring video 22 for each moving object, and stores the tracking result in the object information 310 of the storage unit 31. The tracking result of the object information 310 is referred to by the object feature extraction unit 34. The tracking process is performed by a known method such as extracting an object region from each frame by background difference processing and associating an object region of the same object among the object regions extracted in the preceding and following frames by template matching or a particle filter. As the tracking result, the object ID, the number of tracking frames, the number of tracking frames, the object region, and the object position among the object features are obtained.

  The object feature extraction unit 34 extracts an object feature amount with reference to the tracking result and the monitoring video 22. Specifically, the whole body edge feature, the upper half edge feature, the lower half edge feature, the whole body color feature, the upper half color feature, the lower half color feature, the object speed, and the object size are extracted from the object feature amount.

  The importance calculation unit 35 generates importance data 311 from the monitoring video 22 and stores it in the storage unit 31. The importance calculation unit 35 is a monitoring value calculation unit that calculates the monitoring value of each tracking frame in accordance with the use of this apparatus that monitors the movement of a moving object such as suspicious behavior in the monitoring space and the peculiarity of appearance. It is.

  Specifically, the importance calculation unit 35 statistically analyzes the object feature amount for each moving object, obtains an individual representative amount that represents the object feature amount of the moving object, and determines the object in each tracking frame of the moving object. The distance between each feature quantity and the individual representative quantity (intra-object deviation) is calculated, and the higher importance is calculated for a tracking frame having a larger distance. The importance calculation unit 35 further statistically analyzes the object feature amounts of the plurality of moving objects to obtain a group representative amount representing the object feature amounts of the plurality of moving objects, and each individual representative amount of the moving object and the group representative amount The distance (the degree of deviation between objects) is calculated, and the higher the importance is calculated for a tracking frame of a moving object having a larger distance. Then, the calculated importance is stored in the storage unit 31 in association with the tracking frame number. Incidentally, the individual representative quantity and the collective representative quantity include, for example, an average vector as a representative quantity related to a group of feature quantities to be statistically targeted. In addition, a vector of feature amounts corresponding to a median value, a mode value, and the like can also be used as a representative amount. In addition, individual representative quantities and group representative quantities can include information on variances, and such variances are taken into account as definitions of distances in a multidimensional feature space that is the basis for calculating deviations and importance levels. Can be used.

  The importance of each frame calculated in this way becomes higher as the movement or appearance of the moving object to be monitored shows a singular change or a singular frame, and the frame should be watched in monitoring. It can be used as a scale representing the degree of being a frame.

  The importance calculation unit 35 includes an in-object departure degree calculation unit 350 that calculates an in-object departure degree, an inter-object departure degree calculation unit 351 that calculates an inter-object departure degree, and an in-object departure degree and an between-object departure degree. A departure degree combining unit 352 that combines and calculates importance is included. Hereinafter, each of these parts will be described.

The in-object deviation degree calculation unit 350 statistically analyzes the object feature amount f _{t, u} in the appearance section for each tracked object _u to calculate the individual representative amount A _u, and the object for each object u in each frame t. A distance a _{t, u} from the individual representative amount A _u of the object u of the feature quantity f _{t, u} is calculated as an in-object deviation degree. Although the 27-dimensional object feature quantity can be used as it is for the calculation of the in-object deviation, the 10th to 27th order components shown in FIG. 3 are used in the present embodiment. The object size, the upper half edge feature, the lower half edge feature, the upper half color feature, and the lower half color feature, which are the 12th to 27th order components, raise their hands, bend from standing or bend from standing It is suitable as a feature for detecting suspicious behavior in units of frames because it changes depending on the behavior of people with posture changes such as, or the behavior of people with involvement in goods such as leaving or removing luggage or carts. Yes. In addition, since the object speed, which is the 10th and 11th order components, changes with the action of the moving object such as temporary stop, acceleration, and deceleration, it is suitable as a feature amount for detecting the suspicious action of the moving object in units of frames. .

The individual representative amount A _u of the object u can be a distribution function approximating the distribution of the object feature amount f _{t, u} in the appearance section of the object _u . Specifically, a normal distribution function is used as the distribution function. The average vector μ _u and the covariance matrix Σ _u of the object feature value f _{t, u} in the appearance section of the object _u are calculated by the following equations (1) and (2), and these are applied to the normal distribution function to individually A representative amount A _u = N (μ _u , Σ _u ) is calculated. Incidentally, (1) and (2), T _u is the number tracking frames of the object u, t is 1 to start frame of appearance section of the object u, relative to the end frame and the T _u Frame number.

Object feature amount f _t, in the present embodiment that the _u and frame t and before and after each consisting of p frame sub-interval distribution, objects within the deviance a _{t, u} individual representative amount A _u and the object feature amount by the following formula It can be calculated as a batcha rear distance with ft _{, u} .

In Expression (3), μ _{t, u} is an average of the feature quantities of the object u in a small section (hereinafter referred to as the small section t) corresponding to the frame t, and Σ _{t, u} is the object u in the small section t. Is the covariance of the feature quantity.

It should be noted that various well-known inter-distribution distance scales such as a Cullbacher distance and a Hellinger distance can be used instead of the Batachariya distance. Further, in another embodiment in which the object feature amount f _{t, u} is a moving average or one frame feature amount, the in-object deviation degree at _{, u} is expressed by the following expression using the individual representative amount A _u and the object feature amount f _{t, It} can be calculated as Mahalanobis distance from _u . The distance according to the equation (4) is effective when priority is given to the importance calculation processing speed.

FIG. 4 is a schematic diagram showing an example of the calculation process of the in-object departure degree, and _conceptually shows the process of calculating the in-object departure degree a _58,1 of the object # 1 in the 58th frame. Here, the feature space is simply displayed in two dimensions. In the figure, a white circle is a feature quantity of one frame, a black triangle is an average vector μ ₁ constituting the individual representative quantity A ₁ of the object # ₁ , and a dotted line is represented by a covariance matrix Σ ₁ constituting the individual representative quantity A _1. The resulting average vector μ ₁ to 3 sigma is shown. The white arrow corresponds to the in-object deviation degree a _58,1 . The white triangle is the average vector μ _58,1 of the feature quantity of the object # 1 in the 58th small section when p = 1, and the one-dot chain line is the covariance matrix of the feature quantity of the object # 1 in the 58th small section. The range of the mean vector μ _58,1 to 3 sigma obtained by Σ _58,1 is shown.

FIG. 4A shows an example in which the object feature value f _{t, u} is a feature value of one frame, and the Mahalanobis distance (actually a covariance matrix Σ between N (μ ₁ , Σ ₁ ) and a white circle. Is normalized by ₁ ) to be the in-object deviation a _58,1 .

FIG. 4B shows an example in which the object feature value f _{t, u} is a moving average, and f _58,1 = μ _58,1 . N (μ _{1, Σ 1)} Mahalanobis distance between the _{μ 58,1 (actually} been normalized by the covariance matrix sigma ₁₎ becomes an object within deviance _{a 58,1.}

FIG. 4C shows an example in which the object feature quantity f _{t, u} is a small section distribution, and f _58,1 = N (μ _58,1 , Σ _58,1 ). The virtual distance between N (μ ₁ , Σ ₁ ) and N (μ _58,1 , Σ _58,1 ) (actually normalized by the covariance matrices Σ ₁ and Σ _58,1 ) Deviation a _58,1 .

Further, in another embodiment in which processing speed is prioritized, normalization by dispersion can be omitted. In this case, the individual representative amount A _u of the object u is set as an average value of the object feature amounts f _{t, u} in the appearance section of the object _u, and between each object feature amount f _{t, u of the} object _u and the average value thereof. The Euclidean distance is calculated as the in-object deviation degree at _{, u} . As described above, the mode value or the median value can be used instead of the average value.

In another embodiment, the object feature quantity f _{t, u} is the variance of the feature quantity in the small section, the individual representative quantity A _u is the variance of the feature quantity in the appearance section, and the distance between these variances is the in-object deviation degree. It can also be calculated as at _{, u} .

  Since the in-object deviation calculated in this way is based on the representative feature value calculated for each object, a high in-object deviation is calculated in a frame in which the movement or appearance of each object is temporarily changed. A low degree of deviation in the object is calculated for a frame in which the movement or appearance that the object is exclusively appearing. For example, in a monitoring image 22 in which a specific person who has been trotting while a large number of people are walking temporarily decreases the speed, a high in-object deviation is present in a frame in which the specific person temporarily decreases the speed. A low in-object deviation is calculated for the frame that has been calculated and has been running.

The inter-object deviation degree calculation unit 351 statistically analyzes the object feature amounts f _{t, u} of a plurality of moving objects tracked in the monitoring video 22 to calculate a collective representative amount B, and the individual representative amount A for each object u. the distance b _u from a population representative amount B of _u is calculated as an object between the deviance. Although the 27-dimensional object feature amount can be used as it is for the calculation of the deviation degree between objects, the first to eleventh order components shown in FIG. 3 are used for the calculation of the deviation degree between objects in this embodiment. The whole body edge feature and whole body color feature that analyzed the edge feature and color feature globally are not easily influenced by minute changes in frame units, but it is easy to compare differences between persons such as unique clothes, so the degree of deviation between objects Suitable as original data. In addition, the object position is suitable as original data of the degree of deviation between objects because it is easy to compare differences between persons such as entry / exit from a specific position and movement to a specific area.

The collective representative quantity B can be a distribution function approximating the distribution of the object feature quantities f _{t, u} of all the objects tracked. Specifically, a normal distribution function is used as the distribution function. An average vector μ and a covariance matrix Σ of object feature values f _{t, u} of all objects are calculated, and these are applied to a normal distribution function to determine a collective representative value B as B = N (μ, Σ). The individual representative amount A _u used for calculating the deviation degree between objects is calculated in the same manner as the processing in the deviation deviation calculation unit 350 within the object. However, the point that the calculation target is the first to eleventh order components is different from the case of calculating the in-object deviation.

The inter-object deviation b _u of the object _u can be calculated as a batch rear distance between the collective representative amount B and the individual representative amount A _u of the object. In addition, it is also possible to use various known inter-distribution distance scales such as a Cullback libler distance and a Hellinger distance instead of the Batachariya distance.

In another embodiment in which priority calculation processing speed is prioritized, the individual representative amount A _u is represented by the average vector μ _u , and the inter-object deviation b _u of the object _u is represented by the collective representative amount B and the individual of the object. It is calculated as the Mahalanobis distance between the representative amount a _u.

Further, in another embodiment in which processing speed is prioritized, normalization by dispersion can be omitted. In this case, the population representative amount B all object feature amount f _t, the average vector of _u mu, the individual representative amount A _u object u object feature amount f _t in appearance section of the object _u, the mean vector mu _u of _u , calculated total object feature amount f _t, the mean vector mu and each object feature amount f _t of each object u of _u, the Euclidean distance between the mean vector mu _u of _u as the object between deviance b _u of the object u To do. The mode value or the median value can be used instead of the average value.

In another embodiment, the collective representative quantity B can be calculated as the variance Σ, the individual representative quantity A _u can be the variance Σ _u, and the distance between these variances can be calculated as the inter-object deviation b _u .

  Since the inter-object deviation calculated in this way is based on the collective representative amount calculated from the entire monitoring video 22, a high inter-object deviation is calculated for an object whose movement and appearance are different from the majority of objects. Thus, a low inter-object deviation is calculated for an object that is similar in movement and appearance to the majority of objects. For example, in the above example where a specific person who has trotting while a large number of persons are walking temporarily decreases the speed, a high degree of deviation between objects is calculated for this specific person who has trotting and walking. A low degree of inter-object deviation is calculated for the majority of persons.

The deviation degree synthesis unit 352 calculates the importance of the frame for each frame according to the deviation degree within the object and the deviation degree between the objects. Specifically, based on the following equation, the object in deviance for each frame t a _t, by weighted addition of _u and the object between the deviance b _u calculates the importance c _t of the frame. In the summation calculation of equation (5), the deviation degrees for all moving objects u belonging to the set U (t) of moving objects in the frame t are added together.

  Here, λ is a parameter for adjusting the weight of the in-object deviation and the inter-object deviation, and is set in a range of 0 ≦ λ ≦ 1. For example, it may be set to 0.5.

In another embodiment, the deviation degree compositing unit 352 determines the maximum value of the in-object deviations at and _u and the inter-object deviation b _u of all moving objects in the frame for each frame t. it may be calculated as c _t. The importance c _t in this case is expressed by the following equation. (6) Since the importance c _t it is not affected by the number of moving objects in the frame different from (5), departing degree higher portion of each frame in the object to be tracked even number less scene preferentially select the type Will be.

  FIG. 5 is an explanatory diagram showing an example of the in-object deviation, the inter-object deviation, and the importance, corresponding to the example shown in FIG. There is a tracking frame. Data 601 is an example of the in-object deviation calculated for the objects # 1 to # 3. The in-object deviation for the object # 1 is calculated for the 56th to 61st frames, which are the appearance sections, and The values are 1.0, 1.5, 5.6, 2.0, 1.8 and 1.0. The in-object deviation degree of the object # 2 is calculated in the 58th to 65th frames which are the appearance sections, and the values are 1.2, 3.4, 3.0, 4.5, 5.2, and 2. 6, 2.0, and 1.7. Further, the in-object deviation degree of the object # 3 is calculated for the 112th to 116th frames which are the appearance sections, and the values are 1.1, 1.5, 1.7, 1.5, 1.. 2. Note that the in-object deviation is not calculated for the 66th to 111st frames in which no object is detected.

  Data 602 is an example of the inter-object deviation calculated for the objects # 1 to # 3. The inter-object deviation of the object # 1 is 2.0, the inter-object deviation of the object # 2 is 2.2, and the object # The degree of deviation between 3 objects is 4.9.

  Data 603 is an example of importance obtained by synthesizing the in-object deviation in data 601 and the inter-object deviation in data 602 according to equation (5). The importance for the 56th to 65th frames is 1.5, 1.75, 5.5, 4.8, 4.5, 4.85, 3.7, 2.4, 2.1, 1.95. The importance for the 112th to 116th frames was calculated as 3.0, 3.2, 3.3, 3.2, and 3.05.

For example, for the 58th frame, a _58,1 = 5.6, b ₁ = 2.0, a _58,2 = 1.2, b ₂ = 2.2, and these are set to λ = 0.5. Applying to the equation (5), c ₅₈ = {(1-0.5) × 5.6 + 0.5 × 2.0 + (1-0.5) × 1.2 + 0.5 × 2.2} = 5 .5. Further, when Expression (6) is used instead of Expression (5), c ₅₈ = max {5.6, 2.0, 1.2, 2.2} = 5.6.

  In the example shown in FIG. 5, since the degree of deviation between the objects # 1 and # 2 is low, it cannot be determined that the entire appearance section of these objects is a unique scene. However, the in-object deviation of object # 1 is high in the 58th frame, and the in-object deviation of object # 2 is high in the 59th to 62nd frames. From this, it can be seen that a unique scene can be determined for several frames constituting the appearance section of each object. That is, it can be seen that the deviation degree in the object is effective for determining the important scene for the objects # 1 and # 2.

  On the other hand, the in-object departure degree of the object # 3 is low in any of the 112th to 116th frames, but the inter-object departure degree is high. From this, it can be seen that if the appearance section of the object # 3 is illuminated with the appearance sections of other objects, the entire appearance section of the object # 3 can be determined as a unique scene. That is, it can be understood that the deviation degree between objects is effective for determining an important scene for the object # 3.

  In the importance obtained by synthesizing the in-object deviation and the inter-object deviation, the 58th frame in which the object # 1 is unique, the 59th to 62nd frames in which the object # 2 is unique, and the object # 3 is unique. Higher values are calculated for each of the 112th to 116th frames, and important scene determination with less leakage is performed both from a global viewpoint based on the degree of deviation between objects and a local viewpoint based on the degree of deviation within the object. It becomes possible.

  Data 603 generated by the departure degree synthesis unit 352 and associated with the tracking frame number and the importance is stored in the storage unit 31 as importance data 311.

  The summary video creation unit 36 refers to the importance level data 311 and thins out the monitoring video by selecting frames of the designated number of frames from the monitoring video 22 with priority on the frames with high importance. Create a summary video with a set number of frames less than the total number of frames. Then, the created summary video is output to the display device 4. Here, by sequentially outputting the summary video at the same frame interval as at the time of imaging, important scenes with high importance are played back at the same frame interval as at the time of imaging, and other scenes are thinned out so that they are played back at high speed. The

  Specifically, the summary video creation unit 36 uses dynamic programming (DP) to select a frame from the monitoring video 22 that maximizes the sum of its importance levels with the number of skips within a predetermined range. The summary video creation unit 36 applies the dynamic programming method to a portion including only the tracking frame except for the frame in which no object is detected in the monitoring video 22. For example, if the 56th to 116th frames illustrated in FIG. 5 are the monitoring images to be summarized, the remaining 15 frames excluding the 66th to 111th frames in which no object is detected among these 61 frames are dynamically changed. Planning methods apply.

  In the application, the minimum number of skips can be set to 0, while the maximum number of skips can be set to (total number of tracked frames / designated number of frames × α−1). α is an adjustment value for preventing all selections from being made with the maximum number of skips, and α> 1. For example, α = 2 can be set. For example, when the total number of tracking frames is 15 and the designated number of frames is 10, the maximum number of skips is 15/10 × 2-1 = 2 frames.

  The selection range of the number of skips can be defined as a slope limit in dynamic programming. FIG. 6 is a schematic diagram showing the tilt restriction when the above-mentioned maximum skip number is set to 2 frames. The horizontal axis corresponds to the frame number of the summary video, and the vertical axis corresponds to the frame number of the monitoring video. Yes. In FIG. 6, white circles are grid points of the summary video frame and the surveillance video frame, and a solid line connecting the two grid points represents a local path. Corresponding to the selection range of skip number being 0, 1, 2, and when the summary video frame number is incremented by 1, a local path is set in which the increment number of the frame number of the surveillance video 22 is 1, 2, 3. Is done.

  The importance of the frame of the monitoring video 22 corresponding to the grid points connected by the local path corresponds to the cost when selecting the local path. The summary video creation unit 36 accumulates the cost corresponding to the local path selected by the dynamic programming. The cost accumulated along the path connecting the lattice points of the designated number of frames is the sum of the importance levels, and the summary video creation unit 36 is the accumulated value of the cost among a plurality of paths each combining the local paths. Find the path that maximizes.

  Here, if a moving object appears or disappears from various positions in the image in the summary video, a great stress is applied to the observer who follows this. Therefore, a summary video is created by forcibly selecting the start frame and end frame of the appearance section of each object regardless of their importance. When dynamic programming is used, this control can be realized by prohibiting selection of a local path across the start frame and end frame of the appearance section of each object. By doing so, each moving object appears from the actual appearance position and disappears from the actual exit position in the summary video, so that the stress of the observer can be reduced.

  FIG. 7 is a graph showing lattice points and local paths that can be selected by the dynamic programming based on the example of the monitoring video 22 and the setting example of the dynamic programming described above. The vertical axis corresponds to the tracking frame number in the monitoring video 22, and the horizontal axis corresponds to the frame number of the summary video to be created. White circles and black circles are grid points that can be selected, and black circles are grid points that are forcibly selected. Further, a line connecting the circles is a local path that can be selected under the inclination restriction and the forced selection constraint condition of FIG. Further, by imposing the condition that the cost is maximized, in this example, the path indicated by the thick line in FIG. 7 is selected, and the 56th, 58th, 59th, 60th of the monitoring video 22 is selected as the first to 10th frames of the summary video. , 61, 63, 65, 112, 114, and 116 frames are selected.

  With dynamic programming, a continuous summary video is created within the maximum frame interval, and not only the frames with high importance but also the “connection” frames between them are selected appropriately. It is possible to create a summary video that can reduce the stress on the observer who follows the moving object that appears.

  The display device 4 is a liquid crystal display or a CRT (Cathode Ray Tube) display that displays a summary video input from the signal processing unit 32. The user can quickly view the monitoring video 22 by viewing the summary video reproduced and displayed on the display device 4.

  Next, the operation of the video processing device 1 will be described focusing on the operation of the video summarizing device 3. The imaging unit 20 images the monitoring space at a preset frame interval and outputs a monitoring video 22 to the recording unit 21. The recording unit 21 records the monitoring video 22. FIG. 8 is a schematic flowchart for explaining the operation of the video summarizing apparatus 3. The user designates the monitoring video 22 to be summarized using the setting input unit 30, and inputs the designated number of frames (S1).

  The signal processing unit 32 reads the monitoring video 22 designated in step S1 from the recording unit 21 (S2). The object tracking unit 33 of the signal processing unit 32 processes the read monitoring video 22 to individually track the object imaged in the monitoring video 22 and stores the tracking result in the object information 310 of the storage unit 31. (S3). In addition, the object feature extraction unit 34 of the signal processing unit 32 extracts the feature amount of the object from each tracking frame of each object, and stores the extracted feature amount in the object information 310 of the storage unit 31 (S4). . Then, the importance level calculation unit 35 of the signal processing unit 32 refers to the object information 310 and calculates an importance level indicating the specificity of the action of the object imaged in the frame for each tracking frame (S5). .

  FIG. 9 is a schematic flowchart of processing by the importance degree calculation unit 35. The intra-object deviation degree calculating unit 350 and the inter-object deviation degree calculating unit 351 calculate the individual representative amount of each object by statistically analyzing the object feature amount of all tracking frames for each object, and store the calculated individual representative amount in the storage unit 31 ( S50). Then, for each tracking frame of each object, the in-object departure degree calculation unit 350 calculates the distance between the object feature amount of the object in the tracking frame and the individual representative amount of the object as an in-object departure degree, and stores the storage unit 31 (S51). On the other hand, the inter-object deviation degree calculation unit 351 calculates a collective representative quantity by statistically analyzing all the object feature quantities extracted in step S4 of FIG. 8 (S52). Then, the inter-object deviation calculating unit 351 calculates the distance between the individual representative amount and the collective representative amount of each object calculated in step S50 as the inter-object deviation of the object, and stores it in the storage unit 31 ( S53).

  The departure degree combining unit 352 combines the in-object departure degree calculated in step S51 and the inter-object departure degree calculated in step S53 for each tracking frame to calculate the importance. The importance calculation unit 35 stores the calculated importance in association with the tracking frame number and stores it as importance data 311 in the storage unit 31. At this time, the importance calculation unit 35 may add information in which the frame number of a frame other than the tracking frame is associated with the lowest importance value 0 to the importance data 311.

  When the process by the importance calculation unit 35 shown in FIG. 9 is completed, the process returns to FIG. 8 again, and the summary video creation unit 36 of the signal processing unit 32 uses the dynamic programming to maximize the importance. (S6). Specifically, the summary video creation unit 36 first calculates the total number of tracking frames, obtains the maximum number of skips from the total number of tracking frames and the number of designated frames, and sets the slope limitation of dynamic programming. Next, the summary video creation unit 36 creates a frame number sequence of the tracking frame based on the importance data 311 and uses the same number as the designated number of frames within the range of tilt restriction from the frame in the sequence using dynamic programming. A partial sequence is created, and a partial sequence that maximizes the sum of the importance levels of the tracking frames constituting the sequence is selected. However, when creating a partial sequence, the start and end frames of the appearance section of each object are forcibly incorporated into the partial sequence. In the selected partial series that maximizes the sum of the importance levels, the frames constituting the summary video are expressed by the frame number of the monitoring video 22. The summary video creation unit 36 creates a summary video by extracting the frames indicated by the selected partial series from the monitoring video 22 and arranging them in time series, and stores the summary video in the storage unit 31.

  The summary video creation unit 36 reproduces the summary video by sequentially outputting the summary video stored in the storage unit 31 to the display device 4 at a frame interval at the time of imaging (S7).

  In the above-described embodiment, the importance calculation unit 35 determines the importance from the in-object deviation and the between-object deviation. However, the importance may be calculated only from the in-object deviation. Also, let the user select either the importance calculated by combining the deviation within the object and the deviation between the objects or the importance calculated only from the deviation within the object, and calculate the importance using the selected method. Also good.

  The summary video creation unit 36 sequentially accumulates the importance from the first frame of the monitoring video 22 and selects the frame when the cumulative value reaches the threshold value T2 as the frame of the summary video. When one frame is selected, the cumulative value is selected. Is reduced by the threshold value T2 or the accumulated value is reset to 0, and then the process of starting the same importance accumulation and selecting the next frame is repeated until the end frame of the monitoring video 22 to obtain the summary video. You may create it. The threshold value T2 at this time can be set as (sum of importance for all frames of monitoring video 22 / designated number of frames). According to this processing method, the amount of calculation is reduced compared to the case of using dynamic programming.

  Alternatively, only one of the start frame and the end frame of the appearance section of each object may be forcibly incorporated into the summary video.

  The summary video creation unit 36 may create a summary video by selecting the same number of frames as the designated number of frames in descending order of importance and arranging the selected frames in the order of frame numbers. Alternatively, a summary video can be created by applying a uniform threshold T3 to the importance of each frame and arranging frames whose importance exceeds the threshold T3. In this case, the threshold value T3 can be adjusted to be a summary video of the designated number of frames by sequentially increasing or decreasing. In any of these configurations, the density of important scenes in the summary video increases because no “connecting” is included in the summary video. In this respect, it can be said that it is suitable for skilled users.

  The video processing apparatus 1 can replace the monitoring video stored in the recording unit 21 with a summary video created from the monitoring video 22. By doing so, frames with high monitoring value are left and frames with low monitoring value are deleted, so that the data amount of the recording unit 21 can be reduced while keeping the monitoring value of the monitoring video 22 constant.

  In the above-described embodiment, the importance calculation unit 35 statistically analyzes the entire appearance section of each moving object and calculates the individual feature amount. However, a part of the appearance section may be statistically analyzed. For example, only the section tracked in the area set in advance in the monitoring video 22 can be the target of statistical analysis.

  In the above embodiment, the importance calculation unit 35 statistically analyzes all moving objects and calculates the collective feature amount. However, some moving objects may be statistically analyzed. For example, only the moving object tracked in each time zone by dividing the imaging time zone of the monitoring video 22 can be the target of statistical analysis.

  DESCRIPTION OF SYMBOLS 1 Video processing device, 2 Video recording device, 3 Video summarization device, 4 Display device, 20 Imaging part, 21 Recording part, 30 Setting input part, 31 Storage part, 32 Signal processing part, 33 Object tracking part, 34 Object feature extraction 35, importance calculation unit, 36 summary video creation unit, 310 object information, 311 importance data, 350 in-object deviation calculation unit, 351 inter-object deviation calculation unit, 352 departure degree synthesis unit.

Claims (5)

A video processing device that generates information of a frame to be watched in a monitoring video composed of a plurality of frames,
A storage unit for storing the monitoring video;
An object tracking unit that tracks an object image of a moving object in the monitoring video by image analysis, and generates correspondence information between a tracking frame in which the moving object is tracked and the object image of the moving object;
An object feature extraction unit that extracts a movement or / and appearance feature amount of the moving object from the object image;
The feature amount is statistically analyzed for each moving object to obtain an individual representative amount representing the distribution of the feature amount, and the individual representative amount of each moving object and the feature amount for each tracking frame of the moving object A monitoring value calculation unit that calculates a first monitoring value that is a distance of the tracking frame and calculates a higher monitoring value for the tracking frame having a larger first distance;
A video processing apparatus comprising: The video processing apparatus according to claim 1,
The monitoring value calculation unit statistically analyzes the feature amount of the plurality of moving objects tracked by the object tracking unit to obtain a group representative amount representing the feature amount, and the group representative amount and each moving object And calculating a second distance that is a distance from the individual representative amount of the moving object according to the first distance of the moving object in each tracking frame and the second distance of the moving object tracked in the tracking frame. A video processing apparatus characterized by calculating the monitoring value. The video processing apparatus according to claim 1, further comprising:
Summary video creation for creating a summary video having a set number of frames smaller than the total number of frames of the monitoring video by thinning out the monitoring video by preferentially selecting the tracking frame having a higher monitoring value from the monitoring video A video processing apparatus. The video processing apparatus according to claim 3.
The summary video creation unit selects at least one of a start frame and a termination frame in a series of the tracking frames of each moving object regardless of the monitoring value thereof. Processing equipment. In the video processing device according to claim 3 or 4,
The summary video creation unit uses dynamic programming to maximize the total sum of the monitoring values of the tracking frames constituting the summary video.

Images