JP5496566B2 - Suspicious behavior detection method and suspicious behavior detection device - Google Patents

Description

  The present invention relates to a suspicious behavior detection method and a suspicious behavior detection device that detect suspicious behavior in a monitoring area based on a camera image.

  As this type of monitoring system, there is a system that examines and defines a daily operation to be monitored and detects an abnormal operation by deviation from the daily operation. For example, in an elevator, it is normally stationary, but there are rarely people who exercise, and that exercise is a periodic movement that moves the hands and feet periodically. If there is no operation, there is a system that determines that the operation is abnormal (see Patent Document 1). Also, it analyzes the general behavior of people such as walking and bending, and based on the time-series data about this general behavior, identifies whether or not a specific dedicated behavior (for example, changing from walking to → bending, After that, there is an image monitoring system that determines that the patient is sick if the bending action continues for N seconds or longer (see Patent Document 2).

  On the other hand, there is a monitoring system that defines abnormal behavior patterns (locations and actions) and detects behaviors similar to the defined behaviors as abnormal behaviors. For example, there is a walking monitoring system that detects movements such as staggering walking and staggered walking in a dangerous area (such as a sidewalk or a platform end of a station) as abnormal walking (see Patent Document 3). In addition, the monitoring object is analyzed in units of monitoring elements (operations of human body parts, appearance identification, number of visual recognition actions), and monitoring items for the monitoring objects are determined based on predetermined determination rules for each subdivided monitoring element. There is a system that determines and outputs an alarm based on the comprehensive determination (see, for example, Patent Document 4).

Japanese Patent No. 3994416 JP 2007-272488 A JP 2007-89105 A JP 2006-28784 A

  In a system that detects an operation deviating from daily behavior as an abnormal behavior, if the behavior pattern of the abnormal behavior desired to be detected is similar to the daily behavior pattern, a detection failure is likely to occur. For example, in a monitoring area such as a parking lot, there are a plurality of types of objects such as people and vehicles, the degree of freedom of entering and exiting the monitoring target, and the actions are high, and possible daily behavior patterns are varied and there are individual differences. For this reason, behaviors similar to abnormal behaviors are included in the range of daily behaviors, and abnormal behaviors (such as criminal acts) similar to daily behaviors are likely to be missed.

  On the other hand, criminal acts such as vandalism do not occur frequently, so the number of surveillance video samples obtained is only a small amount. In addition, behavioral patterns related to criminal acts are complex and varied, with individual differences. For this reason, in a surveillance system that detects behavior similar to the defined abnormal behavior pattern as abnormal behavior, even if the surveillance camera captures the actual criminal behavior, the behavior extracted from the image and criminal behavior sample data are used. The degree of detachment when compared is increased, and detection omission is likely to occur.

  It should be noted that in a system that supports the video surveillance of a monitor, it is desirable to detect abnormal behavior without omission even if there is a rare case where daily behavior is erroneously detected as abnormal behavior.

  The present invention is intended to solve the above-mentioned problems, and is suspicious even in a wide area where it is difficult to define suspicious (abnormal) behavior depending on the place, time, and action that occurs, even in the monitoring targets of various complex behavior patterns. It is an object of the present invention to provide a suspicious behavior detection method and a suspicious behavior detection device that can prevent omission of behavior detection.

  The gist of the present invention for achieving the object lies in the inventions of the following items.

[1] A processing unit tracks a person in an image input from a camera, first determination as to whether or not the action of the tracked person is an extraordinary action deviating from a daily action pattern, A suspicious behavior detection method that performs a second determination as to whether or not the behavior is a suspicious unique behavior, and detects the suspicious behavior of the person based on the determination results of the first determination and the second determination ,
Register feature time series data, which is time series of feature data indicating the feature information of the person being tracked, in the database for multiple people performing daily actions, and at the time of feature quantities of persons performing specific actions Register the knowledge-based reasoning function found from the series data in the reasoning function storage unit,
In the tracking of a person in an image input from the camera, the feature amount time series data of the person is collected,
The first determination is performed by a statistical method based on feature amount time-series data relating to a determination target person and feature amount time-series data for a plurality of persons who perform daily activities that have been tracked before registered in the database. Done
The suspicious behavior detection method , wherein the second determination is performed by applying a knowledge-based inference function registered in the inference function storage unit to feature amount time-series data relating to a determination target person .

  In the invention of [1] above and [5] described later, the first determination as to whether or not the behavior of the tracked person is an extraordinary action deviating from the daily action pattern, and whether or not the action of the person is a suspicious unique action Since the suspicious behavior of the person is detected based on both determination results of the second determination, detection with higher reliability is possible compared to the case where the suspicious behavior is detected only by one determination.

In addition, since many feature amount time-series data can be collected for daily behavior, in the first determination, it is determined whether or not the behavior is extraordinary by a statistical method based on them. On the other hand, since abnormal behavior such as criminal acts has a small number of feature time series data sampled, in the second determination, a knowledge-based reasoning function derived from the small feature amount time series data is applied to perform unique behavior. It is determined whether or not.

[ 2 ] If at least one of the determination result of the first determination for the person to be determined is an extraordinary action or the determination result of the second determination for the person is a specific action is true, The suspicious behavior detection method according to [1] , wherein the suspicious behavior is detected for the person and the second determination is performed only when the determination result of the first determination is a daily behavior. .

In the invention of [ 2 ] and [ 6 ] to be described later, since it is determined that the person has suspicious behavior when it is determined to be an extraordinary behavior or a peculiar behavior, the detection failure of the suspicious behavior can be reduced.
[3] When the determination result of the first determination is an unusual action and the determination result of the second determination is a suspicious specific action, the suspicious level is set to the first level,
When the determination result of the first determination is unusual behavior and the determination result of the second determination is not suspicious unique behavior, and when the determination result of the first determination is everyday behavior and the determination result of the second determination is suspicious In the case of unusual behavior, the suspicious level is set to a second level lower than the first level.
[1] The suspicious behavior detection method according to [1].
[4] In the suspicious behavior detection method according to any one of [1] to [3], tracking of a person in an image input from a camera is performed by detecting an object in the processing target image. The object tracking method of
Detecting an object in the image to be processed;
When a new tracking target object is detected, a plurality of search blocks having a predetermined size for the tracking target are generated in the area of the object, and each search block has a feature of the object in the search block existing area. Storing information in association with each other;
For each search block, the object and the object to be tracked by the search block based on the comparison between the feature information of the object existing in the search block existence area and the feature information stored in association with the search block Calculating the fitness with
A determination is made as to whether or not an object existing in the search block existence area is a tracking target object of the search block based on the fitness of the search block, and if the determination result is true, the search block Determining a tracking target object as a tracking success object;
For each tracking success object, probabilistically replacing a search block with a low fitness with a search block with a high fitness between a plurality of search blocks targeting the object,
Updating the feature information stored in association with the search block for the search block whose tracking target is the tracking target;
Moving each search block based on a combined motion of the object to be tracked in the search block and a random motion;
Repeat
The process of probabilistically replacing a search block having a low fitness with a search block with a high fitness between a plurality of search blocks of the same object as the tracking target is the sum of the fitness of all search blocks having the object as a tracking target. A roulette is generated with the ratio of the fitness of each search block to the appearance probability of the search block, a random number lottery based on the roulette is performed for each search block whose tracking target is the object, and the search block is selected by the lottery This is done by replacing the search block corresponding to the result.

[5] A tracking unit that tracks a person in an image input from the camera;
A first determination unit that determines whether the behavior of the person tracked by the tracking unit is an extraordinary behavior deviating from the daily behavior pattern;
A second determination unit that determines whether the action of the person is a suspicious unique action;
A suspicious determination unit that detects the suspicious behavior of the person based on the determination results of the first determination unit and the second determination unit;
A database in which feature amount time series data obtained by chronologically storing feature amount data indicating the feature information of the person being tracked is registered for a plurality of people who perform daily actions,
A reasoning function storage unit in which a knowledge-based reasoning function found from feature amount time-series data in which feature amount data indicating the feature information of a person performing a specific action is time-sequentially registered;
Have
The tracking unit collects feature amount time-series data of time-series feature amount data indicating the feature information of the person for the tracking target person,
The first determination unit determines whether or not the action of the person is an extraordinary action by a statistical method based on the feature amount time-series data related to the determination target person and the feature amount time-series data registered in the database. Determine whether
The second determination unit applies the knowledge-based inference function stored in the inference function storage unit to the feature amount time-series data related to the determination target person to determine whether or not the person's action is a suspicious unique action A suspicious behavior detecting device characterized by determining

[6] The suspicious determination unit may determine that the determination result of the first determination unit with respect to the determination target person is an extraordinary action, or that the determination result of the second determination unit with respect to the person is a specific action. If at least one is true, the person is determined to have detected suspicious behavior, and the determination by the second determination unit is performed only when the determination result of the first determination unit is a daily action. The suspicious behavior detection device according to [5].
[7] The suspicious determination unit
If the determination result of the first determination unit is an unusual behavior and the determination result of the second determination unit is a suspicious unique behavior, the suspicious level is the first level,
When the determination result of the first determination unit is an unusual action and the determination result of the second determination part is not a suspicious unique action, and the determination result of the first determination part is an everyday action and the determination of the second determination part When the result is a suspicious unique action, the suspicious level is set to a second level lower than the first level. The suspicious behavior detecting device according to [5] or [6].
[8] In the suspicious behavior detection apparatus according to any one of [5] to [7], the tracking unit detects the object in the image to be processed and tracks the following object: By tracking method,
Detecting an object in the image to be processed;
When a new tracking target object is detected, a plurality of search blocks having a predetermined size for the tracking target are generated in the area of the object, and each search block has a feature of the object in the search block existing area. Storing information in association with each other;
For each search block, the object and the object to be tracked by the search block based on the comparison between the feature information of the object existing in the search block existence area and the feature information stored in association with the search block Calculating the fitness with
A determination is made as to whether or not an object existing in the search block existence area is a tracking target object of the search block based on the fitness of the search block, and if the determination result is true, the search block Determining a tracking target object as a tracking success object;
For each tracking success object, probabilistically replacing a search block with a low fitness with a search block with a high fitness between a plurality of search blocks targeting the object,
Updating the feature information stored in association with the search block for the search block whose tracking target is the tracking target;
Moving each search block based on a combined motion of the object to be tracked in the search block and a random motion;
The process of probabilistically replacing a search block having a low fitness with a search block with a high fitness between the search blocks having the same object as the tracking target is the adaptation of all search blocks that have the object as the tracking target. Generating a roulette having the search block appearance probability as a ratio of the fitness of each search block to the sum of degrees, and performing random number lottery based on the roulette for each search block whose tracking target is the object; Is replaced with a search block corresponding to the outcome of the lottery result.

  According to the suspicious behavior detection method and the suspicious behavior detection device according to the present invention, even in the monitoring target of various complex behavior patterns in a wide area where it is difficult to define the suspicious (abnormal) behavior due to the location, time, and action that occurs, It is possible to prevent omission of detection of suspicious behavior.

It is a block diagram which shows schematic structure of the suspicious action detection apparatus which concerns on embodiment of this invention. It is a flowchart which shows the operation | movement which a suspicious action detection apparatus performs. It is explanatory drawing which shows an object data table. It is explanatory drawing which shows the outline | summary of action unusual degree derivation | leading-out process (step S13 of FIG. 2). It is explanatory drawing which shows an example of the concept of an action unusual degree derivation process. It is explanatory drawing which illustrated the accumulation | storage state of the feature-value time series data. Input to the extraordinary degree deriving unit is an explanatory view illustrating how extraction when extracting the feature quantity at stores series data area of the time length T ₁ minute. It is a flowchart which shows the detail of action unusual degree derivation | leading-out process (step S13 of FIG. 2). It is a flowchart which shows the detail of action specificity derivation | leading-out process (step S16 of FIG. 2). It is explanatory drawing which illustrated the list | wrist of the parameter | index (inference function argument) of the attention action. It is explanatory drawing which shows the arithmetic expression which calculates | requires the knowledge base reasoning function and the maximum value of the specificity calculated | required with these knowledge base reasoning functions. It is a binary tree figure at the time of determining whether it is suspicious action from unusualness and specificity. It is a 2nd flowchart which shows the operation | movement which a suspicious action detection apparatus performs. It is explanatory drawing which shows the determination logic table corresponding to the process of FIG. It is a state transition diagram of the object which the tracking part of the processing part of a suspicious action detection device tracks. It is explanatory drawing which shows each process phase of the tracking determination process of the object which the process part of a suspicious action detection apparatus performs. It is a flowchart which shows a tracking discrimination | determination process. It is explanatory drawing which shows the outline of the search by a search block. It is explanatory drawing which shows an example of the state which divided | segmented the image (image for 1 frame) of a process target into the some division block. 18 is a flowchart showing object tracking / search processing (FIG. 17; step S105). It is a flowchart which shows search block generation processing (FIG. 20; step S122). It is explanatory drawing which shows the relationship between the origin (px0, py0) and center position (px, py) of a search block. It is explanatory drawing which shows a search block data table. It is explanatory drawing which shows an example of an HSV color space accumulation histogram. It is a flowchart which shows the detail of a fitness update process (FIG. 20: step S123). It is explanatory drawing which shows the relationship between the distance between search block center and the object gravity center of tracking object, and a fitness. It is a flowchart which shows the detail of a division | segmentation block area ID allocation process (FIG. 20: step S124). It is a flowchart which shows the detail of search block erasure | elimination process (FIG. 20: step S125). It is a flowchart which shows the detail of a search block resampling process (FIG. 20: step S128). It is explanatory drawing which shows an example of the roulette produced | generated by the search block resampling process. It is explanatory drawing which shows an example of the replacement condition of the search block by search block resampling processing. It is a flowchart which shows the detail of a search block update process (FIG. 20: step S129). It is explanatory drawing which shows an example of the tracking and search condition of the object by the search block before and behind an intersection. It is a flowchart which shows the detail of a person and vehicle discrimination | determination process (FIG. 17: step S111).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a schematic configuration of a suspicious behavior detection apparatus 10 according to an embodiment of the present invention. The suspicious behavior detection apparatus 10 of the present embodiment is a surveillance camera system that detects a suspicious person who acts in a wide area where a plurality of objects and persons exist. In this embodiment, a surveillance camera system that monitors a suspicious person in a large parking lot will be described as an example. That is, if the moving object detected from within the camera image is a person or a vehicle, if it is determined that it is a person, the behavior is analyzed to determine whether it is a suspicious action, and when a suspicious action is detected Performs functions such as alerting observers. The camera captures a wide range, and the “view” of a person passing through the imaging area can change variously.

  The suspicious behavior detection apparatus 10 detects a target object by analyzing the image captured by the camera unit 11 and capturing the image by capturing the monitoring target area, tracking the object, and searching for the object. The processing unit 12 that performs processing such as type discrimination and human behavior analysis, the output unit 14 that outputs the processing result of the processing unit 12, and inputs of various settings and instructions regarding the suspicious behavior detection device 10 are received. The input unit 13 is configured to include a database unit 15 in which reference data for discriminating object types and analyzing human behavior is stored. The camera unit 11, the output unit 14, the input unit 13, and the database unit 15 are each connected to the processing unit 12. Hereinafter, an object recognized in the image data is referred to as an object.

  The camera unit 11 captures a moving image. The camera unit 11 functions to capture a moving image by capturing several tens of frames of images per frame (a still image for one screen) per second. The processing unit 12 includes a CPU (Central Processing Unit), a ROM (Read Only Memory) in which programs and data executed by the CPU are stored, a RAM (Random Access Memory), and the like as main parts.

  The processing unit 12 extracts a feature of the object during tracking of the object by the tracking unit 21, the storage unit 22, and the tracking unit 21 that tracks and searches for an object in the image input from the camera unit 11, The accumulation control unit 23 that repeatedly stores the feature amount data indicating the feature in the storage unit 22, and the type of the object is determined based on the feature amount data of the object for a predetermined time length or longer stored in the storage unit 22 And a behavior analysis unit 25 that analyzes the behavior of the person based on the feature amount data of the object that has been identified as a person.

  The feature amount data is, for example, the shape, size, movement, aspect ratio, etc. of the object. The feature amount data is stored in the storage unit 22 in time series. The accumulation control unit 23 stores the feature data in the storage unit 22 at predetermined time intervals during tracking of the object. Here, it is performed for each frame. Note that the time-series feature amount data is referred to as feature amount time-series data.

  The tracking unit 21 has a function of tracking the same object as the same object before and after the occurrence of a phenomenon such as the intersection of the tracking target objects or the temporary hiding of the tracking target object behind the obstacle. I have.

  The person / vehicle discrimination unit 24 has a function of discriminating whether the tracking target object is a person or a vehicle. What is necessary is just to perform discrimination | determination of a person and a vehicle by well-known arbitrary methods.

  The behavior analysis unit 25 includes an extraordinary degree deriving unit 25a (first determination unit) for deriving a degree (unusual degree) that a human action is an extraordinary action, and the human action is a specific action (abnormal behavior). Specificity deriving unit 25b (second determination unit) for deriving the degree (specificity), and function as suspicious determination unit 25c for determining whether or not the person's behavior is suspicious behavior based on the extraordinary degree and specificity Fulfill.

  The database unit 15 includes a human feature DB unit 15a in which feature data such as a shape related to a person is stored, a vehicle feature DB unit 15b in which feature data such as a shape related to a vehicle is stored, and a feature amount time series related to daily activities of a person. The daily behavior feature amount time series DB unit 15c in which data and the like are accumulated, and the unique behavior feature amount time series DB unit 15d in which feature amount time series data and the like relating to human specific behavior are accumulated are configured. For example, criminal acts such as vandalism and vehicle theft are regarded as specific actions (suspicious actions) to be detected.

  The person / vehicle discriminating unit 24 compares the data stored in the human feature DB unit 15a with reference to determine whether the object is a person, and compares the data stored in the vehicle feature DB unit 15b with reference comparison. To determine whether the object is a vehicle. The extraordinary degree deriving unit 25a derives the extraordinary degree of the behavior of the person using a statistical method from the movement of the object determined to be a person and the data accumulated in the daily behavior feature time series DB unit 15c. To do. The specificity deriving unit 25b applies the knowledge-based reasoning function found from the data accumulated in the unique action feature time series DB unit 15d to the movement of the object determined to be a person, and the person Deriving the specificity of behavior.

  The output unit 14 includes an interface circuit for outputting a processing result data signal to the outside, various setting screens and operation screens, a display device for visually displaying processing results, and the like. The input unit 13 includes various operation switches in addition to an interface circuit for inputting signals relating to various settings and instructions regarding operations.

  Next, the operation of the suspicious behavior detection device 10 will be described.

  FIG. 2 shows a flow of operations performed by the suspicious behavior detection apparatus 10. Detailed examples of operations related to the tracking unit 21, the storage unit 22, the accumulation control unit 23, and the person / vehicle determination unit 24 will be described later. Here, it is assumed that the object tracking is performed by analyzing the image input from the camera unit 11 and it is determined whether each object is a person or a vehicle, and the operation of the behavior analysis unit 25 is mainly performed. explain.

  An object data table 30 shown in FIG. 3 is created for each object being tracked. The object data table 30 is created every time a new object is detected, and is updated every time one frame of image is processed until the object disappears.

The object data table 30 includes an object ID (obj_id), an object type (obj_type ^obj_id ), an object state (obj_state ^obj_id ), an object suspicious determination flag (suspicious_flag ^obj_id ), an object existence duration (obj_time ^obj_id ), Object area left edge position (obj_lx ^obj_id ), object area right edge position (obj_rx ^obj_id ), object area top edge position (obj_ty ^obj_id ), object area bottom edge position (obj_by ^obj_id ), object area gravity center position (horizontal direction) (obj_gx ^obj_id ), object area centroid position (vertical direction) (obj_gy ^obj_id ), object size (obj_size ^obj_id ), object aspect ratio (obj_aspect ^obj_id ), and object motion vector (horizontal direction) (obj_dx ^obj_id ) And the motion vector of the object (vertical And direction) (obj_dy ^obj_id), Object - nearest vehicle distance and (obj_car_dist ^obj_id) is registered.

The object ID (obj_id) is an identifier uniquely assigned to the object. Whether the object type (obj_type ^obj_id ) is an object that has been determined to be a person (hereinafter also referred to as a human object) or an object that has been determined to be a vehicle (hereinafter also referred to as a vehicle object) is not determined. Indicates whether it is an object. The object state (obj_state ^obj_id ) indicates whether the state of the object is being tracked (ACTIVE) or temporarily lost (LOST). The object suspicious determination flag (suspicious_flag ^obj_id ) is set when the object is a person, and indicates whether the object is a human object determined as a suspicious person (person who has performed suspicious behavior). The distance between the object and the nearest vehicle (obj_car_dist ^obj_id ) is set when the object is a person, and indicates a distance to the nearest vehicle.

  Returning to FIG. 2, the description will be continued. After initializing the frame time t to 0 (step S1), the processing unit 12 checks whether or not an object (person object) that is determined to be a person exists in the tracked object (step S2). . If there is no tracking person object (tracking person) (step S3; No), the frame time t is incremented by 1 (step S22), the process proceeds to step S2, and the process is continued. Note that the processing unit 12 executes the processes in steps S2 to S22 every time one frame image (or a predetermined plurality of frames) is captured from the camera unit 11.

  If there is a tracking human object (tracking person) (step S3; Yes), it is checked whether there is a disappearing human object (disappearing person) (step S4). If there is no disappeared person object (step S4; No), the process proceeds to step S9. If there is a disappeared person object (step S4; Yes), the object data table 30 of all the disappeared person objects is cleared (steps S5 to S8), and the process proceeds to step S9.

In step S9, the variable i indicating the number of the human object to be processed is cleared to 0, and the processes of steps S10 to S21 are performed for each of the tracking human objects. That is, it is checked whether or not the object suspicious determination flag (suspicious_flag ^{obj_id (i)} ) is in an ON state (ON indicates a suspicious person) (step S10). The quantity time-series data is updated (step S11). Here, the contents of the object data table 30 for each frame (all or a part (top / bottom / left / right end position, center of gravity position, motion vector, size, aspect ratio, object-nearest vehicle distance, etc.)) The feature amount data of the object is stored in the storage unit 22 in time series (as feature amount time series data).

Next, it is checked whether or not the feature amount time-series data is accumulated for T ₁ hour or more (step S12). If the accumulated amount is less than T ₁ hour (step S12; No), the process proceeds to step S15. To do. When accumulated for T ₁ hour or longer (step S12; Yes), based on the feature amount time-series data accumulated for T ₁ hour or longer, behavioral extraordinary for deriving the extraordinary degree related to the behavior of the human object Degree derivation processing is performed (step S13).

If the calculated extraordinary degree is equal to or greater than a predetermined threshold Th ₁ (step S14; Yes), it is determined that the human object is performing suspicious behavior, and the object suspicious determination flag (suspicious_flag ^{obj_id (i)} ) is ^{turned on.} (Step S18), an alarm is displayed (step S19), and the process proceeds to step S20. Note that the determination based on the extraordinary degree is the first determination. If the calculated extraordinary degree is less than the predetermined threshold Th ₁ (step S14; No), the process proceeds to step S15.

In step S15, the feature amount time series data is checked whether it is accumulated over _two hours T, if the accumulation amount is less than ₂ hours T (step S15; No), the process proceeds to step S20. When accumulated for T ₂ hours or more (step S15; Yes), behavior specificity derivation for deriving specificity regarding the behavior of the human object based on the feature time series data accumulated for T ₂ hours or more. Processing is performed (step S16).

If the determined specificity is equal to or greater than a predetermined threshold Th ₂ (step S17; Yes), it is determined that this person object is performing suspicious behavior, and the object suspicious determination flag (suspicious_flag ^{obj_id (i)} ) is ^turned ON. (Step S18), an alarm is displayed (Step S19), and the process proceeds to Step S20. The determination based on the specificity is the second determination.

When the object suspicious determination flag (suspicious_flag ^{obj_id (i)} ) is already ON (step S10; Yes), an alarm is displayed (step S19), and the process proceeds to step S20.

  In step S20, the variable i is incremented by 1 and it is checked whether or not the variable i after addition is equal to the number of all-person objects being tracked (step S21). If they are not equal (step S21; No), the process proceeds to step S10. Return and process the next person object. If equal (step S21; Yes), the frame time t is incremented by 1 (step S22), and the process returns to step S2 to process the next frame image.

FIG. 4 shows an outline of the behavior unusual degree derivation process (step S13 in FIG. 2), and FIG. 5 shows an example of the concept of the behavior unusual degree derivation process. In FIG. 4, the extraordinary degree deriving unit 25a performs a predetermined statistical process on feature amount time-series data relating to a large number of daily behaviors stored in the daily behavior feature amount time-series DB unit 15c, and represents a typical daily behavior pattern. Is compared with the action pattern indicated by the feature time series data in the time series section j for T ₁ hour or more, and the degree of deviation from the representative daily action pattern (vector (matrix)) is Derivation as dailyness is performed.

FIG. 5 shows an example of a statistical method performed by the extraordinary degree deriving unit 25a, based on feature amount time-series data relating to a large number of daily behaviors stored in the daily behavior feature amount time-series DB unit 15c. Cluster analysis of the data distribution, and the distance between the center of the cluster and the action pattern (feature time series vector (matrix) x _{j for} time series interval j) indicated by the feature time series data in time series interval j for the human object An example of obtaining the extraordinary degree ND _j is shown. This is an example of a method for deriving the extraordinary degree, and the present invention is not limited to this.

FIG. 6 illustrates the accumulation state of feature amount time-series data. The feature amount time series data storage area F stores feature amount data up to Lmax time before the current time in time series. Note that Lmax> T ₁ and T ₂ . The storage area F is provided in the storage unit 22, for example. FIG (a) shows a state where the feature time-series data of the time T ₁ length necessary for derivation of the extraordinary degree is accumulated. The storage area F is not clogged and has a free area. FIG (b) shows a state in which time-series data, wherein the amount of T ₂ hours length necessary for derivation of specificity has been accumulated. The storage area F is not yet filled and there is a free area. FIG. 5C shows a state where the storage area F is filled with feature amount time-series data. Thereafter, the data is shifted one by one to the left, the oldest data is discarded, and the latest data is stored in a free area at the right end. That is, the stored contents are updated in a FIFO (First in First Out) format. In this example, T ₂ > T ₁ , but T ₁ > T ₂ or T ₁ = T ₂ may be used. T _1, T _{2 is} the target behavior pattern (short behavior or gestures, prolonged continuous operation, etc.) of interest, respectively in accordance with a flexibly adaptable.

FIG. 7 exemplifies how to extract feature amount time-series data of time length T ₁ input to the extraordinary degree deriving unit 25a from the storage area F of FIG. FIG. 4A shows a case where the time length T ₁ is divided into time series sections and input, and FIG. 4B shows a case where the time length T ₁ time series section (window) is slid and input. Show.

FIG. 8 shows the flow of behavior extraordinary degree derivation processing (step S13 in FIG. 2). j = 0 (step S41) and the maximum unusual value ND _max = 0 (step S42), respectively. Subsequently, the feature amount time-series data of the time-series section j is read from the storage area F and input to the extraordinary degree derivation unit 25a to obtain the extraordinary degree ND _j (step S43), and the extraordinary degree ND _j is extraordinary. If it is larger than the maximum degree value ND _max (step S44; Yes), the extraordinary degree maximum value ND _max is updated with the extraordinary degree ND _j obtained this time (step S45). The processing from step S43 to S45 is repeated while shifting the time series section j as shown in FIG. 7 (step S46) (step S47; No), and when the above processing is completed for all feature quantity time series sections (step S47). ; Yes), the maximum value ND _max of the extraordinary degree is set as an output value, and the process is terminated.

  FIG. 9 shows details of the action specificity deriving process (step S16 in FIG. 2). In the behavior specificity degree derivation process, an arbitrary number (in this case, S) of knowledge base inference functions discovered from the feature amount time series data related to the person who performs abnormal behavior stored in the unique behavior feature amount time series DB unit 15d. These are defined and registered, and these are applied to the behavior of the person to be determined to derive the specificity of the behavior (the degree of suspicious behavior). The knowledge base inference function is stored in, for example, the unique action feature amount time series DB unit 15d or stored in another storage area (for example, the storage unit 22 on the processing unit 12 side).

  In this example, as shown in FIG. 10, the distance between a person and a parked vehicle, the parked vehicle periphery bending, the parked vehicle peep, the intense movement around the parked vehicle, etc. are defined as the index of action to be focused on (inference function argument). In addition, the value of each index is obtained from the feature amount time-series data regarding the human object to be determined.

  Then, the values of these indexes are substituted into S knowledge base reasoning functions, respectively, to determine the specificity for each knowledge base reasoning function, and the maximum value among them is derived as the action specificity of the human object. Yes.

Specifically, as shown in FIG. 9, the variable j indicating the time series interval is initialized to 0 (step S51), and the maximum value AB _max of specificity is initialized to 0 (step S52). Subsequently, the specificity AB _j is initialized to 0 (step S53), and 1 is set to the variable i (step S54). Then, an argument (index value) relating to the i-th knowledge base inference function is obtained from the feature amount time series data relating to the determination target human object and input (step S55). By assigning these index values to the i-th knowledge base inference function, the specificity AB ⁱ _j by the i-th knowledge base inference function is obtained (step S56).

The specificity AB ⁱ _j and the specificity AB _j are compared in magnitude (step S57). If specificity AB ⁱ _j > specificity AB _j (step S57; Yes), AB _j is determined by the specificity AB ⁱ _j. Update (step S58), if specificity AB ⁱ _j > specificity AB _{j is} not satisfied (step S57; No), specificity AB _j is left as it is. Subsequently, i is incremented by 1 (step S59). If i> S is not satisfied (step S60; No), the process returns to step S55 and the process is continued.

If i> S (step S60; Yes), the specificity AB _j is compared with the maximum specificity AB _max (step S61), and if specificity AB _j > specificity AB _max (step S61; Yes). ), The specificity AB _max is updated with the specificity AB _j (step S62). If the specificity AB _{j is} not greater than the specificity AB _max (step S61; No), the specificity AB _max is left as it is. Subsequently, j is incremented by +1 (this corresponds to, for example, shifting the time series section by one as shown in FIG. 7) (step S63). If the confirmation (steps S53 to S62) for all the feature amount time series sections has not been completed (step S64; No), the process returns to step S53 to continue the processing. If the processing has been completed (step S64). Yes), the maximum value AB _max of specificity is set as an output value, and the process is terminated.

FIG. 11 shows S knowledge base reasoning functions 51 and a formula 52 indicating that the maximum value of the specificity obtained by each of these S knowledge base reasoning functions is the specificity AB _j .

FIG. 12 shows a binary tree diagram 54 showing the flow when determining whether or not the behavior is suspicious from the extraordinary degree ND and the specificity AB. If the extraordinary degree ND is equal to or greater than the threshold Th ₁ , it is determined that the behavior is suspicious regardless of the specificity AB, and if the unusual degree ND is less than the threshold Th ₁ , the suspicious is determined if the specificity AB is equal to or greater than the threshold Th _2. It is determined to be normal, and when the extraordinary degree ND is less than the threshold Th ₁ and the specificity AB is less than the threshold Th ₂ , it is determined to be normal. In the flowchart shown in FIG. 2, the processing is in accordance with the binary tree diagram 54.

  As shown in FIG. 2 and FIG. 12, in the method of determining in two steps, if “unusuality” is equal to or greater than a threshold value, processing regarding “specificity” can be omitted at all, so that the overall processing speed is improved. There is an advantage of doing. In FIG. 2 and FIG. 12, attention is first focused on “unusuality”, and then suspicious behavior detection processing based on “specificity” is performed. Thus, the suspicious behavior detection process based on the “unusual level” may be performed.

Note that the processing may be performed according to the flow shown in FIG. 13 instead of FIG. In FIG. 13, the same processes as those in FIG. 2 are denoted by the same step numbers, and description thereof is omitted. In FIG. 13, when the extraordinary degree is greater than or equal to the threshold Th ₁ or the specificity is greater than or equal to the threshold Th ₂ (step S17B; Yes), it is determined that the action is suspicious (step S18). FIG. 14 shows a decision logic table 55 corresponding to FIG. When at least one of the conditions of the extraordinary degree is equal to or greater than the threshold Th ₁ and the specificity is equal to or greater than the threshold Th ₂ is satisfied, the logical sum is determined to be suspicious behavior.

  As shown in FIGS. 13 and 14, in the logical sum determination method, for example, if both “unusual degree” and “specificity” are equal to or higher than the threshold value, the alarm level is high, and if only one is the threshold value, the alarm level is: Thus, alarm level setting desired by the user (monitor) and alerting according to the alarm level can be made.

  As described above, with regard to daily behavior that can collect a large amount of data (feature amount time series data), statistical methods are applied to the daily behavior data accumulated in the daily behavior feature amount time series DB unit 15c. Since a typical daily behavior pattern is derived and the extraordinary degree is obtained as a deviation from the typical daily behavior pattern, the extraordinary degree is obtained by objective judgment based on a large number of collected daily behavior data. be able to.

  Also, since there is little data (features time-series data) collected for suspicious behavior such as criminal acts, it is difficult to make an appropriate judgment using statistical methods, but the knowledge-based reasoning function obtained from the collected suspicious behavior data is used. Thus, the specificity can be derived with higher reliability even from a small amount of data regarding suspicious behavior.

And, since it is determined that the extraordinary degree is greater than or equal to the threshold Th ₁ and the specificity is greater than or equal to the threshold Th ₂ as suspicious behavior, only one of the conditions of extraordinary degree and specificity is used. Compared to the above, it is possible to reduce detection failure of suspicious behavior.

  Next, an object tracking method by the tracking unit 21, the storage unit 22, and the accumulation control unit 23 will be described.

  In this tracking method, even if a phenomenon such as the intersection of the tracking target objects or the temporary hiding of the tracking target object behind the obstacle occurs, the same object can be tracked as the same object before and after that. It can be done. Note that the following tracking method is an example, and the tracking method is not limited to this.

  FIG. 15 illustrates a state transition of an object tracked by the tracking unit 21 of the suspicious behavior detection apparatus 10. The object status is undetected UNDETECTED, PENDING determining whether the object detected in the UNDETECTED state is an object that can be tracked, ACTIVE being tracked and successfully tracked, and tracking It is managed in four states: LOST, which is the target object, but tracking has failed (disappearing or searching).

  While the object is not detected, the UNDETECTED state continues (E1 in the figure). When an object is detected in the UNDETECTED state, the state shifts to the PENDING state (E2). In the PENDING state, the duration is counted (E4), and when a predetermined time (PENDING_TIME) elapses in the PENDING state, the object transitions to the ACTIVE state (E5). An object that disappears from the image before transitioning from the PENDING state to the ACTIVE state transitions to the UNDETECTED state (E3), and is excluded from the tracking target.

  In the ACTIVE state, the duration is counted (E6). If tracking of an object in the ACTIVE state fails (the object disappears), the object transitions to the LOST state, and the duration of the ACTIVE state is reset (E7). Even in the LOST state, the duration is measured (E9), and when the predetermined duration (LOST_TIME) elapses, the object is removed from the tracking target and becomes the UNDETECTED state (E10). An object found (detected) by the search by the search block within a predetermined time (LOST_TIME) after entering the LOST state returns to the ACTIVE state (E8).

  FIG. 16 shows each processing phase of the object tracking determination process performed by the processing unit 12 of the suspicious behavior detection apparatus 10. FIG. 17 shows the flow of the tracking determination process. In the tracking determination process, object tracking and whether each object is a vehicle or a person are determined. As shown in FIG. 16, the tracking determination process includes an object detection phase P1, an object tracking / search phase P2, a person / vehicle determination phase P3, and a human object extraction phase P4. The object detection phase P1 corresponds to steps S102 to S104 in the flowchart of FIG. 17, the object tracking / search phase P2 corresponds to steps S105 to S110, the person / vehicle discrimination phase P3 corresponds to step S111, and the human object extraction phase P4 corresponds to steps S112 and S113. Note that in the process SA indicated by the broken line in FIG. 17, if the processes of steps S2 to S21 in FIG. 2 or FIG. 13 are performed, the tracking and the detection of the suspicious behavior are performed for each frame.

<Object detection phase P1>
First, after initializing the frame time t to 0 (FIG. 17; step S101), an object candidate area is detected from the processing target image captured by the camera unit 11 (step S102). The object candidate area is detected by a known method such as a difference process between frames or a binarization process.

  Next, it is confirmed whether or not one or more object candidate areas have been detected in the processing target image (step S103). When one or more object candidate areas are detected (step S103; Yes), the process proceeds to step S105. If no object candidate area is detected (step S103; No), it is confirmed whether or not there is an object whose state is LOST (hereinafter referred to as a LOST object) (step S104). The LOST object is an object in which an object detected as a tracking target in the past disappears (tracking failure), and the duration of the disappearance state is less than a predetermined time (LOST_TIME), and is an object being searched by a search block described later.

  If the LOST object does not exist (step S104; No), the frame time t = t + 1 is set (step S114), the process returns to step S102, and the same process is performed on the next frame image captured by the camera unit 11. As described above, the object detection phase P1 is a state in which it is confirmed whether or not one or more object candidate areas exist, and when there is no object candidate area, the detection of the object candidate area is awaited.

<Object Tracking / Search Phase P2>
In step S105, a tracking / search process is performed on the object candidate area detected in the object detection phase P1. In the object tracking / search process, a tracking process for associating an object in the image at the previous time with an object in the image at the current time and a search process for searching for an object that disappears due to hiding or crossing are performed. The tracking process is performed by a known method such as a motion vector.

  The search process for searching for the lost object is to generate a plurality of search blocks around the center of gravity of the newly detected tracking target object, and then cause each search block to follow the tracking target object. The tracking is performed by focusing on the local area (here, the area of the search block unit) in the object area. The search block is a unit area for tracking and searching for an object. If the search for the lost object is not successful within a predetermined time (LOST_TIME), the searched object is regarded as having been lost and is set to the UNDETECTED state, and is excluded from the tracking target. Details of the object tracking / search process will be described later.

  For objects that have been successfully tracked / searched in the object tracking / search process (objects in the active state), feature information such as shape, motion, texture, etc. is extracted (calculated) (step S106). Extraction of object feature information such as shape and movement may be performed by any known method. The feature amount data indicating the extracted feature information of the object is sequentially accumulated in the storage unit 22, and updated or newly generated as time series data (step S107). The feature amount data accumulated in time series for each object is used to determine whether the object is a person or a vehicle.

  Thereafter, if the object to be tracked is an object that has already been identified as a vehicle (step S108; Yes), the frame time t = t + 1 is set (step S114), the process returns to step S102, and the next frame captured by the camera unit 11 is obtained. Similar processing is performed on the image. When the object to be tracked is already determined to be a person and the object exists in the image (tracking / searching has succeeded) (step S109; Yes), the human object extraction process in step S113 Migrate to

  If neither the vehicle nor the person is identified as the object being tracked (step S109; No), the state (status) of each object is updated according to the state transition of FIG. obj_time is incremented and updated (step S110), and the process proceeds to a person / vehicle discrimination process (step S111).

<Person / vehicle discrimination phase P3>
In the person / vehicle discrimination process, based on the time-series feature amount data (feature amount time-series data) of the tracking target object accumulated and stored in the object tracking / search phase P2, whether the tracking target object is a person, It is determined whether the vehicle is a vehicle (step S111). The person / vehicle discrimination process is performed by an appropriate known method.

<Human object extraction phase P4>
If it is determined that the object to be tracked is a person (step S112; Yes), the process proceeds to a human object extraction process (step S113). If the object is not determined to be a person (step S112; No), the frame time t (Step S114), the process returns to step S102, and the same processing is performed on the next frame image captured by the camera unit 11.

  In the human object extraction process, the location of the person is communicated to the monitoring person in an easy-to-understand format, and monitoring attention is urged (step S113). For example, when an image from the camera unit 11 is displayed on the display device of the output unit 14, the detected person's location is displayed in a red frame. Thereafter, the frame time t is incremented and updated (step S114), and the process returns to step S102 and continues. Suspicious behavior detection (the processing of steps S2 to S21 in FIG. 2 or FIG. 13) is performed on the extracted human object.

  Next, details of the object tracking / searching process in step S105 of FIG. 17 will be described.

  In the object tracking / search process, tracking of an object and searching for an object (LOST object) that has failed (disappeared) in tracking are performed by a known method. The search is performed by a plurality of search blocks that move so as to track the disappeared tracking target object.

  FIG. 18 shows an outline of the search by the search block. For a newly detected object A to be tracked (person A in the example of FIG. 18), a plurality of search blocks D having a predetermined size for which the object A is tracked are generated and each of these search blocks D is The characteristic information of the tracking target object A is associated and stored (t1). In FIG. 18, each rectangular area shaded is a search block D. Each object to be tracked is assigned a unique object ID as identification information for uniquely identifying the object, and the object ID of the object to be tracked is registered by registering the object ID of the object to be tracked in each search block. Is associated with the search block.

  After that, the object is tracked by an existing method such as matching of the image of the frame at the previous time and the image of the frame at the current time, and each search block D is tracked to the object to be tracked (the object's object). Move so as to follow the motion vector (FIG. 18; t2).

  If tracking of an object fails (disappears) due to crossing or hiding (FIG. 18; t3), the movement of the search block D whose tracking target is the object is continued according to a predetermined movement rule. For example, the movement is continued at random, or the movement is continued by combining the movement of the object before disappearance with the random movement. In FIG. 18, the object A is hidden behind the obstacle 61 at time t3, and each search block D moves with a motion that combines the motion of the object A up to the current time and a random motion. The influence of the motion before the disappearance of the object A gradually decreases, and the random motion becomes dominant, and the search block gradually moves so as to be spread over a wide range (t4). In FIG. 18, the character “t3” is attached to each search block at time t3, and the character “t4” is attached to each search block at time t4.

  As described above, when an object exists at the position (existing region) of the search block continuously moved so as to search for the lost object, the feature information of the object is stored in association with each search block. The feature information of the tracking target object is compared, and based on the comparison result, it is determined whether or not the object existing at the position of the search block is the tracking target object of the search block. If the object is a tracking target object, the object ID of the tracking target object registered in the search block is assigned to the object. Thereby, the same object is tracked as the same object before and after disappearance.

  In the example of FIG. 18, the object A is found (detected) by the search block D1 at time t4. After the object to be tracked disappears, each search block moves and spreads with the random movement dominant, so if the object A appears at the position A ′ indicated by the broken line, It will be discovered by the search block D2 which has moved to the position. In this way, even if an intersection or a temporary hiding behind an obstacle occurs, the same object is tracked as the same object (with the same object ID) before and after that.

  In the present embodiment, as shown in FIG. 19, an image to be processed (image for one frame) is divided into a plurality of small regions (divided blocks Bk), and an object is captured in units of divided blocks. Thereby, the amount of calculation related to tracking is reduced. Here, the image to be processed is divided by a grid-like boundary line so that each divided block Bk is rectangular.

  In the example of FIG. 19, the object A is divided into 33 rectangular divided blocks Bk that are shaded. Tracking / searching of the object A is performed for each divided block Bk. The object A is recognized as an aggregate of the 33 divided blocks Bk, and the feature information such as the shape, the center of gravity position, and the movement of the object A as a whole is grasped. Here, when the area of the object occupying the divided block Bk is equal to or greater than a predetermined occupancy ratio (for example, 50% or more), the divided block is regarded as a divided block where the object exists.

  FIG. 20 shows the flow of the object tracking / search process (FIG. 17; step S105). In object tracking (FIG. 20; step S121), an object ID is assigned to a newly detected object and an object data table 30 as shown in FIG. 3 is generated.

  For objects that have been detected at the past time and have already been assigned an object ID, tracking is performed by a method such as associating an object in the previous time image with an object in the current time image by a known method such as a motion vector. To do.

  Next, search block generation processing for generating a search block for the object that is newly tracked is performed (step S122). Details of the search block generation process are shown in FIG. A state (status) that is present in the processing target image is ACTIVE (step S142; Yes), and there is no search block for which the object is a tracking target (step S143; Yes). A plurality of search blocks are generated around the center of gravity (steps S144 to S155).

  A search block is a unit area for tracking / searching an object to be tracked. Whether or not an object existing in the search block's existing area is the object to be tracked by the search block, with the search block as a unit. To be judged. A predetermined number (BLOCKFACTOR) of search blocks is generated for each divided block in which an object to be tracked exists. That is, the number of search blocks corresponding to the size of the tracking target object is generated. The number of BLOCKFACTOR is one or more and is a number that can be arbitrarily changed, and may be set to an appropriate number based on the ratio of the area size of the divided block and the area size of the search block. For example, the number of BLOCKFACTOR is set to 2 to 5. FIG. 22 shows the relationship between the origin (px0, py0) and the center position (px, py) of the search block. The search block is a pixel area of BLOCK_SIZE × BLOCK_SIZE (5 × 5 pixel area in the example of FIG. 22). Here, although the search block is rectangular, the shape of the search block is not limited to this, and may be another shape such as a circle.

  Further, a search block data table 70 as shown in FIG. 23 is generated for each generated search block. In the search block data table 70, information indicating the attribute of the corresponding search block is registered. The following values are registered in the search block data table 70 as initial values when they occur.

-Object ID: obj_id → Object ID to be tracked
Search block state (status): state ^obj_id _{n_block} → ON
Search block position (x coordinate): bx ^obj_id _{n_block} → x
^-Search block position (y coordinate): by ^obj_id _{n_block} → y
・ Search block motion vector (horizontal direction): dx ^obj_id _{n_block} → dx (gx ^obj_id ,
gy ^obj_id )
Search block motion vector (vertical direction): dy ^obj_id _{n_block} → dy (gx ^obj_id ,
gy ^obj_id )
・^Fitness of search block: cor ^obj_id _{n_block} → 1.0
Also, an HSV color space cumulative histogram as shown in FIG. 24 is generated from the HSV values (pixel values expressed in the HSV color space) of each pixel in the search block existence region. This cumulative histogram is derived by scanning the search block area in units of pixels. For pixel value i, if j ≧ i,
hist _{H, S, V} (t, j) ^obj_id _{n_block} = hist _{H, S, V} (t, j) ^obj_id _{n_block} + 1
And That is, the possible values of the pixel value are divided into a predetermined number of classes, and if the pixel value of the pixel is i, “1” is added to each of the accumulated values of the classes to which i belongs. This process is performed for all the pixels in the search block.

  The HSV color space is a color space composed of three components: hue, saturation (saturation / chroma), and brightness (brightness / lightness / value), and the HSV color space cumulative histogram is the H (hue) component. , S (saturation) component and V (lightness) component. Each value related to the HSV color space cumulative histogram is stored in association with the search block as feature information of the object (image) at the position (existing region) of the search block.

Next, the fitness for all search blocks is updated (FIG. 20: step S123, details are shown in FIG. 25). In the process of updating the fitness level, the fitness level is updated (FIG. 25; steps S164 to S167) for all search blocks whose search block status (status) is ON (FIG. 25; step S163; Yes). Do. The goodness of fit is an index indicating the degree of coincidence between the object existing at the position (existing area) of the search block and the tracking target object of the search block. The degree of fitness is the feature information of the object existing in the existence area of the search block (feature information as an image such as color and the feature information related to the center of gravity position) and the feature information of the object stored in association with the search block (the relevant It is calculated based on a comparison with the feature information of the object existing at the previous time position of the search block. In this embodiment, a case where an HSV color space cumulative histogram is used will be described. Specifically, by the product of the time difference of the HSV color space cumulative histogram in the search block existence area and the distance coefficient based on the distance between the center of gravity of the object to be tracked by the search block and the center of the search block. To derive. The goodness of fit ranges from 1 to 0. Specifically, it is as the following formula.
The time difference total_diff of the HSV color space cumulative histogram is expressed by the following equation, for example.
For example, the distance coefficient is expressed by the following equation.
Therefore, the closer the distance between the center of the search block and the center of gravity of the object area is, the higher the fitness of the search block is. For the disappearing object (LOST object), the distance coefficient is obtained by using the center of gravity position at the frame time immediately before the LOST state as the center of gravity of the object.

  FIG. 26 illustrates the relationship between the distance between the center of the search block and the center of gravity of the object to be tracked and the degree of fitness. Since the search blocks D1 and D2 are not on the tracking target object, the distance from the object center of gravity is large (distance factor (dist_factor) → small), and the degree of coincidence of the object (image) feature information (HSV color space cumulative histogram) ) Is low (total_diff → small), so the fitness is low (approaching 0). Since the search blocks D3 and D4 are on the object to be tracked, the distance from the object center of gravity is small (distance factor (dist_factor) → large), and the degree of coincidence of object (image) feature information (HSV color space cumulative histogram) Since the degree of matching is also high (total_diff → large), the degree of matching is high (close to 1).

  Next, divided block region ID assignment processing is performed (FIG. 20: step S124, details are shown in FIG. 27). In this process, the divided block Bk in which some object exists in the divided block area and one or more search blocks exist has the highest priority (having the highest fitness) among those search blocks. By assigning the tracking target object ID registered in the search block, the object ID is assigned to the divided block Bk, and the object of the object ID is set to the ACTIVE state. The object that was originally in the ACTIVE state continues to be in the ACTIVE state.

  When the object ID registered in the search block whose tracking target is the LOST object is assigned to any one of the divided blocks Bk by the above processing, the tracking target object in the LOST state is found (detection) Was). Note that an object that is a tracking target object but has not been assigned an object ID to any of the divided blocks Bk is determined to be a LOST object in the object state update process (S110) of FIG. 17, and the state is maintained or updated. .

  Next, search block erasure processing is performed (FIG. 20: step S125, details are shown in FIG. 28). If the object state is UNDETECTED, or if the state is LOST and the object duration obj_time is longer than LOST_TIME (LOST state lasts longer than LOST_TIME), it belongs to the object (track the object) Delete all search blocks. Deletion is performed by clearing all search block data tables belonging to the object.

  Next, search block resampling processing is performed (FIG. 20: step S128, details are shown in FIG. 29). The search block resampling process is performed for all search blocks belonging to the object in the ACTIVE state (FIG. 20; step S127; Yes). In the search block resampling process, a search block with a low fitness is replaced with a search block with a high fitness between a plurality of search blocks with the same tracking target object.

  Here, a roulette is generated in which the ratio of the fitness of each search block to the sum of the fitness of all search blocks that track the same object (for example, object A) is the appearance probability of the search block, A random number lottery based on the roulette is performed for each search block whose object is to be tracked, and the search block is replaced with a search block corresponding to the result of the lottery result. That is, the randomness lottery is performed the same number of times as the number of all search blocks for which the object is tracked based on the roulette, and replaced by the search block corresponding to the winning point for each lottery, The search block having a low match is easily replaced with a search block having a high fitness.

More specifically, the following processing is performed for each object whose state (status) is ACTIVE among the objects existing in the processing target image. First, for the ACTIVE object obj_id of interest, a roulette is generated as follows based on the fitness cor ^obj_id _{n_block} of the search block n_block belonging to the object.

Search block n_block state (status) If state ^obj_id _{n_block} is ON,
roulette ^obj_id _{n_block} = roulette ^obj_id _{n_block-1} + cor ^obj_id _{n_block}
Search block n_block state (status) If state ^obj_id _{n_block} is OFF,
roulette ^obj_id _{n_block} = roulette ^obj_id _{n_block-1}
Next, after generating random number data for each search block belonging to the active object obj_id of interest, and determining the roulette output for the search block based on the random number data, Replace with the corresponding search block.

  FIG. 30 shows an example of a roulette generated for an arbitrary object. There are a total of M search blocks that use this object as a tracking target. On the roulette where the total fitness of all search blocks from search blocks 1 to M is 360 degrees, the output of each search block is arranged at an occupation angle corresponding to the ratio of the fitness. For example, the occupancy angle of the search block 1 is ((the suitability of search block 1) / (the sum of the suitability of all search blocks)) × 360 degrees.

  Since the search block with a high degree of matching has a large occupancy angle, the probability of winning in the random number lottery is high. On the other hand, a search block with a low degree of fitness has a small occupancy angle and a low probability of winning. Therefore, a search block with a low fitness is easily replaced with a search block with a high fitness.

  Here, to replace the search block is to update the contents of the search block data table 70 with the search block data table corresponding to the roulette for items other than the search block ID.

  As described above, the suitability of a search block is high when the search block exists on the tracking target object, and is low when the search block is located away from the tracking target object. Therefore, as shown in FIG. 31A, when search block replacement (search block resampling) is performed for the ACTIVE object, search blocks D5 to D7 on the tracking target object A and search blocks not on the object A There is a large difference in fitness between D1 and D4. Therefore, the roulette generated by the search block resampling is as shown in FIG. 31B (the occupied angle is small for D1 to D4 and the occupied angle is large for D5 to D7).

  When random number lottery for the number of search blocks (in this case, seven times) is performed using this roulette, the result is as shown in FIG. 31 (c), for example. Probabilistically replaced with a high search block. As shown in FIG. 31D, many of the search blocks that were not on the tracking target object are replaced with search blocks on the tracking target object, and the search blocks are easily aggregated on the tracking target object.

  Next, search block update processing is performed (FIG. 20: step S129, details are shown in FIG. 32). In the search block update process, the movement of each search block and the update of the feature information regarding the tracking target object stored in association with each search block are performed. The position information and motion vector information are updated for all search blocks existing in the image (status is ON) (steps S248 and S249). Further, a search block belonging to an object whose status (status) is ACTIVE (with the object as a tracking target), and an object assigned to the divided block of the tracking target object ID and the position where the search block exists For the search block whose ID matches, the HSV color space cumulative histogram is also updated (step S247).

The location information is updated as follows.
Here, rnd_x and rnd_y indicate random numbers in the x and y directions, respectively. By adding a random number, a random motion is added to the search block.

The update of the motion vector information is as follows, and is a linear load sum of the motion vector of the search block at the previous time and the motion vector in the divided block at the current search block location.
Here, k represents an update rate (inertia constant). The value of k can be changed. If tracking has succeeded, the motion vector of the object becomes dominant, and if tracking has failed (the tracking target object has disappeared (LOST state)), movement by random numbers becomes dominant. .

  By tracking and searching for an object as described above, even if the object to be tracked is temporarily hidden by the obstacle 61 as shown in FIG. That is, the same object ID can be assigned to the same object before and after being hidden by the obstacle 61 for tracking.

  Similarly, even when tracking target objects intersect, tracking can be performed by assigning the same object ID to the same object before and after the intersection (FIGS. 33A to 33D). In the intersection area, a search block having a high degree of matching has an advantage. In FIG. 33B, since the object A passes behind the object B, the search block tracking the object B on the front side is dominant. As shown in FIG. 5C, when the objects A and B are separated from each other from the intersection state, the search block belonging to the object A follows the object A, and the search block belonging to the object B follows the object B. (Fig. 4D).

  As described above, in the object tracking / search method according to the present embodiment, when a new object to be tracked is detected, a plurality of search blocks are generated near the center of gravity of the object, and each search block has its search block. Is stored in association with the feature information of the tracking target object in the existing region, and after that, the search block is moved so as to follow the tracking target object and the feature information of the tracking target object stored in the search block is stored. After updating and tracking fails (disappears), the search block continues to move so that random motion becomes dominant, and the tracking target that disappears by determining the fitness based on the local area called the search block By searching for the same object, the same object as the same object before and after crossing and hiding To tail.

  Note that the feature quantity data of the object such as shape and movement extracted from the tracked object in step S106 in FIG. 17 is stored in the storage area F in the FIFO (First in First Out) format as shown in FIG. It is stored as series data (FIG. 17, step S107). The stored feature amount time-series data is used when determining whether each tracking target object is a person or a vehicle in the person / vehicle determination process (step S111) in FIG.

  Next, the person / vehicle discrimination process (S111 in FIG. 17) will be described in detail. FIG. 34 shows the flow of the person / vehicle discrimination process. The person / vehicle discrimination process is performed for each object to be tracked. First, it is confirmed whether or not the state of the object to be tracked is ACTIVE (tracking is in progress). If the object is not ACTIVE (step S301; No), the person / vehicle discrimination process for the object to be tracked is terminated.

  In the case of ACTIVE (step S301; Yes), the time length (L time length) in which the feature amount data related to the tracking target object is stored in time series in the storage area F is a predetermined time length (T time). It is confirmed whether or not it is longer (step S302). The T time length can be set to an arbitrary time length equal to or less than the Lmax time length that can be stored in the storage unit 22. For example, T is set to one half of Lmax.

  If the L time length is less than the T time length (step S302; No), the person / vehicle discrimination process for the tracking target object is terminated.

  If the L time length is equal to or longer than the T time length (step S302; Yes), the stored feature amount time-series data for the L time length is input to the person / vehicle determination unit 24 (step S303). In the determination unit 24, a person / vehicle is determined.

  In this way, the object is tracked, and the feature amount data indicating the feature information of the object is sampled and stored for a certain time length in time series during tracking, and the data is stored for a predetermined period or longer (T time length or longer). Since it is determined whether the object is a person or a vehicle based on the feature amount time series data, the person / vehicle can be identified with higher reliability compared to the case where the person / vehicle is detected immediately after the object is detected. can do. That is, since a person has a high degree of freedom of movement and the “appearance” varies, an error is likely to occur in a determination based only on an image at a certain time, but the suspicious behavior detection device 10 of the present embodiment has a predetermined period. Since the determination is made based on the feature data for a plurality of times collected in time series during the object tracking, it is possible to determine the person / vehicle with high reliability.

  In addition, since the same object can be tracked as the same object even if there is a crossing or hiding, even if there is a crossing or hiding during tracking, the time series feature amount data of the same object can be collected.

  Further, since the type (person / vehicle) of the tracking target object is not discriminated when the feature amount data is less than the predetermined time length (T time length), it is prevented from being erroneously discriminated by the feature amount data having a short sampling time length. Thus, the reliability of determination is ensured.

  The embodiment of the present invention has been described with reference to the drawings. However, the specific configuration is not limited to that shown in the embodiment, and there are changes and additions within the scope of the present invention. Are also included in the present invention.

  In the embodiment, the daily behavior feature amount time series DB unit 15c may be configured to sequentially update the storage contents or additionally store the data during the monitoring operation, or accumulate a sufficient amount of data in advance. It may be configured. The process of deriving a representative daily behavior pattern by a statistical method may be performed during the monitoring operation, but is derived and stored in advance based on the storage content of the daily behavior feature time series DB unit 15c. The stored daily representative behavior pattern may be used during the monitoring operation.

In the embodiment, the binary tree diagram 54 of FIG. 12 and the decision logic table 55 of FIG. 14 are used to prevent detection omissions. However, for example, when detecting suspicious behavior with high certainty, Only when the degree is greater than or equal to the threshold Th ₁ and the specificity is greater than or equal to the threshold Th ₂ may be detected as suspicious behavior.

  The feature amount time-series data is not limited to the items exemplified in the embodiment such as the shape and size, and may be any data as long as it indicates the type of object to be determined and the feature amount of various human movements. Can be set.

  In the embodiment, as the object tracking method, a method of tracking / searching a lost object using a search block is exemplified. However, the present invention is not limited to this, and tracking may be performed by any method. A method capable of tracking the same object as the same object before and after crossing or hiding is preferable, but a tracking method without such a function may be used.

  In the tracking / search shown in the embodiment, the object is tracked / searched in divided block units, but tracking may be performed in object units. In the embodiment, the search block is moved by synthesizing the movement of the object to be tracked and the random movement (position change based on the random number, Equation 4), and the search block targeted for the search block belonging to the ACTIVE object Combined with resampling (probabilistic replacement of search block with low fitness with search block with high fitness), the search block moves to track the tracking target object while tracking is successful The search block is moved so that the random motion becomes dominant during the disappearance, but the method of moving the search block is not limited to the method exemplified in the embodiment. That is, the tracking target object may be tracked during tracking success, and it may be moved according to the movement rule that random motion becomes dominant during disappearance.For example, according to the tracking target motion vector without random elements during tracking success. It may be configured to move and move only with random movement while eliminating the inertial movement during disappearance.

  As the rule of movement during disappearance, it is preferable to add a random motion to the motion of the object immediately before the disappearance, and it is preferable that the random motion becomes more dominant as time passes.

  In the embodiment, the fitness of each search block is determined based on the time difference of the HSV color space cumulative histogram and the distance coefficient based on the distance between the object and the search block. May be determined. The degree of matching is preferably determined using the degree of coincidence as an image and the positional relationship between the object and the search block as elements.

  In the embodiment, as a person / vehicle discrimination method, after considering an object's “appearance” variation and tracking an object of a predetermined time length (feature amount time series data of a predetermined time length or more is accumulated for the object to be tracked) Later, based on the feature amount time-series data, it is determined whether the object is a person or a vehicle. A highly reliable person / vehicle discrimination method is preferable, but such a person / vehicle discrimination method is not necessary.

  In the embodiment, the case where the person and the vehicle are discriminated is described as an example, but the type of the object to be discriminated is not limited to these.

DESCRIPTION OF SYMBOLS 10 ... Suspicious behavior detection apparatus 11 ... Camera part 12 ... Processing part 13 ... Input part 14 ... Output part 15 ... Database part 15a ... Human feature DB part 15b ... Vehicle feature DB part 15c ... Daily action feature amount time series DB part 15d ... Peculiar behavior feature amount time series DB unit 21 ... tracking unit 22 ... storage unit 23 ... accumulation control unit 24 ... person / vehicle discriminating unit 25 ... behavior analysis unit 25a ... unusual degree deriving unit 25b ... specificity deriving unit 25c ... suspicious judgment Part 30 ... Object data table 51 ... Knowledge base reasoning function 52 ... Singularity (knowledge base reasoning function maximum value)
54 ... Binary tree diagram 55 ... Decision logic table 61 ... Obstacle 70 ... Search block data table F ... Storage area P1 ... Object detection phase P2 ... Object tracking / search phase P3 ... Human / vehicle discrimination phase P4 ... Human object extraction phase

Claims (8)

A processing unit tracks a person in an image input from a camera, and first determination as to whether or not the action of the tracked person is an extraordinary action deviating from the daily action pattern, and the action of the person is suspicious. A suspicious behavior detection method for performing a second determination as to whether or not the singular behavior is present, and detecting the suspicious behavior of the person based on the determination results of the first determination and the second determination ,
Register feature time series data, which is time series of feature data indicating the feature information of the person being tracked, in the database for multiple people performing daily actions, and at the time of feature quantities of persons performing specific actions Register the knowledge-based reasoning function found from the series data in the reasoning function storage unit,
In the tracking of a person in an image input from the camera, the feature amount time series data of the person is collected,
The first determination is performed by a statistical method based on feature amount time-series data relating to a determination target person and feature amount time-series data for a plurality of persons who perform daily activities that have been tracked before registered in the database. Done
The suspicious behavior detection method , wherein the second determination is performed by applying a knowledge-based inference function registered in the inference function storage unit to feature amount time-series data relating to a determination target person . If at least one of the determination result of the first determination for the person to be determined is an extraordinary action or the determination result of the second determination for the person is a peculiar action is true, the person The suspicious behavior detection method according to claim 1 , wherein the second determination is performed only when the suspicious behavior is detected and the determination result of the first determination is a daily behavior .   If the determination result of the first determination is an extraordinary behavior and the determination result of the second determination is a suspicious specific behavior, the suspicious level is set to the first level
  When the determination result of the first determination is unusual behavior and the determination result of the second determination is not suspicious unique behavior, and when the determination result of the first determination is everyday behavior and the determination result of the second determination is suspicious In the case of unusual behavior, the suspicious level is set to a second level lower than the first level.
  The suspicious behavior detection method according to claim 1.   The suspicious behavior detection method according to any one of claims 1 to 3, wherein the following object tracking method for tracking a person in an image input from a camera by detecting an object in a processing target image: Do by
  Detecting an object in the image to be processed;
  When a new tracking target object is detected, a plurality of search blocks having a predetermined size for the tracking target are generated in the area of the object, and each search block has a feature of the object in the search block existing area. Storing information in association with each other;
  For each search block, the object and the object to be tracked by the search block based on the comparison between the feature information of the object existing in the search block existence area and the feature information stored in association with the search block Calculating the fitness with
  A determination is made as to whether or not an object existing in the search block existence area is a tracking target object of the search block based on the fitness of the search block, and if the determination result is true, the search block Determining a tracking target object as a tracking success object;
  For each tracking success object, probabilistically replacing a search block with a low fitness with a search block with a high fitness between a plurality of search blocks targeting the object,
  Updating the feature information stored in association with the search block for the search block whose tracking target is the tracking target;
  Moving each search block based on a combined motion of the object to be tracked in the search block and a random motion;
  Repeat
  The process of probabilistically replacing a search block having a low fitness with a search block with a high fitness between a plurality of search blocks of the same object as the tracking target is the sum of the fitness of all search blocks having the object as a tracking target. A roulette with the ratio of the fitness of each search block to the appearance probability of the search block is generated, a random number lottery based on the roulette is performed for each search block whose tracking target is the object, and the search block is selected by the lottery This is done by replacing the search block corresponding to the result. A tracking unit for tracking a person in an image input from the camera;
A first determination unit that determines whether the behavior of the person tracked by the tracking unit is an extraordinary behavior deviating from the daily behavior pattern;
A second determination unit that determines whether the action of the person is a suspicious unique action;
A suspicious determination unit that detects the suspicious behavior of the person based on the determination results of the first determination unit and the second determination unit;
A database in which feature amount time series data obtained by chronologically storing feature amount data indicating the feature information of the person being tracked is registered for a plurality of people who perform daily actions,
A reasoning function storage unit in which a knowledge-based reasoning function found from feature amount time-series data in which feature amount data indicating the feature information of a person performing a specific action is time-sequentially registered;
Have
The tracking unit collects feature amount time-series data of time-series feature amount data indicating the feature information of the person for the tracking target person,
The first determination unit determines whether or not the action of the person is an extraordinary action by a statistical method based on the feature amount time-series data related to the determination target person and the feature amount time-series data registered in the database. Determine whether
The second determination unit applies the knowledge-based inference function stored in the inference function storage unit to the feature amount time-series data related to the determination target person to determine whether or not the person's action is a suspicious unique action A suspicious behavior detecting device characterized by determining In the suspicious determination unit, at least one of the determination result of the first determination unit with respect to the determination target person being an unusual action or the determination result of the second determination unit with respect to the person being a specific action is If true, it is determined that suspicious behavior is detected for the person, and the determination by the second determination unit is performed only when the determination result of the first determination unit is daily activity. Item 6. The suspicious behavior detection device according to Item 5. The suspicious determination unit
If the determination result of the first determination unit is an unusual behavior and the determination result of the second determination unit is a suspicious unique behavior, the suspicious level is the first level,
When the determination result of the first determination unit is an unusual action and the determination result of the second determination part is not a suspicious unique action, and the determination result of the first determination part is an everyday action and the determination of the second determination part The suspicious behavior detection device according to claim 5 or 6, wherein when the result is suspicious unique behavior, the suspicious level is set to a second level lower than the first level. The suspicious behavior detection apparatus according to any one of claims 5 to 7, wherein the tracking unit performs tracking of a person by the following object tracking method of detecting and tracking an object in an image to be processed.
  Detecting an object in the image to be processed;
  When a new tracking target object is detected, a plurality of search blocks having a predetermined size for the tracking target are generated in the area of the object, and each search block has a feature of the object in the search block existing area. Storing information in association with each other;
  For each search block, the object and the object to be tracked by the search block based on the comparison between the feature information of the object existing in the search block existence area and the feature information stored in association with the search block Calculating the fitness with
  A determination is made as to whether or not an object existing in the search block existence area is a tracking target object of the search block based on the fitness of the search block, and if the determination result is true, the search block Determining a tracking target object as a tracking success object;
  For each tracking success object, probabilistically replacing a search block with a low fitness with a search block with a high fitness between a plurality of search blocks targeting the object,
  Updating the feature information stored in association with the search block for the search block whose tracking target is the tracking target;
  Moving each search block based on a combined motion of the object to be tracked in the search block and a random motion;
  Repeat
  The process of probabilistically replacing a search block having a low fitness with a search block with a high fitness between a plurality of search blocks of the same object as the tracking target is the sum of the fitness of all search blocks having the object as a tracking target. A roulette with the ratio of the fitness of each search block to the appearance probability of the search block is generated, a random number lottery based on the roulette is performed for each search block whose tracking target is the object, and the search block is selected by the lottery This is done by replacing the search block corresponding to the result.