JP2010128594A - Motion detection device and motion detection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To highly precisely detect the motion of an object by using information on a plurality of objects. <P>SOLUTION: This motion detection device is provided with: a state information DB 151 for storing state information representing the motion state of an object and expressed in a case frame including a verb representing a type of motion of the object and a case element as a noun related to the motion of the object, and indicating a semantic relation between the verb and the case element; a state acquisition part 139 for acquiring a plurality of pieces of state information representing the motion state of each of the plurality of objects from the state information DB151; and a second detection part 140 for, on the basis of a rule predefining the motion of the object corresponding to the combination of verbs included in each of the plurality of pieces of acquired state information, detecting the motion of the object corresponding to the combination. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a motion detection apparatus and a motion detection method for detecting the motion of an object such as a person from image information captured by an imaging means such as a camera.

  2. Description of the Related Art Conventionally, a recognition device that analyzes an image captured by a monitoring camera or the like and recognizes an action of a monitoring target such as a person is known. For example, in Patent Document 1, it is suspicious from a plurality of monitoring targets by determining whether the pattern is a normal pattern or an abnormal pattern using a pattern recognition technique such as SVM (support vector machine) from video captured by a stereo camera. Techniques have been proposed for detecting an action target to be monitored.

JP 2008-217602 A

  However, the method of Patent Document 1 detects abnormal behavior (walking trajectory) by pattern recognition from the walking trajectory information of a plurality of persons, for example, by using learned walking trajectory data. That is, it is only determined whether each of a plurality of walking trajectories is normal or abnormal. For this reason, for example, when a plurality of people conspire to perform a fraud, it is not possible to detect an action with high accuracy only by pattern recognition of the walking trajectory of each person, resulting in an abnormal action. In some cases, it could not be determined.

  The present invention has been made in view of the above, and an object of the present invention is to provide a motion detection device and a motion detection method that can detect the motion of an object with high accuracy by using information on a plurality of objects. And

  In order to solve the above-described problems and achieve the object, the invention according to claim 1 is information representing the motion state of an object, and is related to a verb representing the motion type of the object and the motion of the object. A state storage means for storing state information represented by a case frame indicating a semantic relationship between the verb and the case element, and a plurality of the objects from the state storage means. Based on a rule that predetermines the motion of the object in accordance with a combination of the verbs included in each of the acquired plurality of state information, state acquisition means for acquiring a plurality of the state information representing the operation state of Detecting means for detecting the motion of the object corresponding to the combination.

  According to a second aspect of the present invention, in the motion detection device according to the first aspect, the state information includes a time when the motion of the object has occurred as the case element, and the state acquisition unit is configured to store the state memory. A plurality of the state information having substantially the same time are acquired from the means.

  According to a third aspect of the present invention, in the motion detection device according to the first aspect, the state information includes a main character representing an object that is a subject of the motion, and another object that is a target of the motion based on the main character. A combination of the verb that represents the type of the action by the main case and the verb that represents the type of the action mainly based on the target case, and The detection unit includes the operation determined according to the combination, and the detection means includes the target case of the first state information as one of the state information and the state information of the other of the plurality of state information acquired. Is matched when the main case of the second state information is the same, and the combination of the verb of the first state information and the verb of the second state information matches the combination included in the rule Detecting the operation corresponding to the combined, characterized by.

  According to a fourth aspect of the present invention, in the motion detection device according to the first aspect, the state information includes a main case that represents an object that is the subject of the motion, and a location case that represents a position where the motion has occurred. The rule is determined according to the combination of the verb representing the type of action of the first object and the verb representing the type of action of the second object, and the combination The detection means includes the location case of the first state information that is one of the state information and the second state information that is the state information of the other of the plurality of state information acquired. The action corresponding to the matched combination when the place case matches and the combination of the verb of the first state information and the verb of the second state information matches the combination included in the rule Detecting, characterized.

  According to a fifth aspect of the present invention, in the motion detection device according to the first aspect, the state information includes a target case representing another object to be the target of the motion as the case element, and the detection means Further, among the plurality of acquired state information, the verb included in the state information matches a predetermined specific verb, and the target cases included in the state information match each other. The number of state information is calculated, and when the number is larger than a predetermined threshold, it is detected that an abnormality has occurred.

  The invention according to claim 6 is the motion detection device according to claim 1, wherein the information is predetermined information as a motion state of the suspicious person, the verb representing the kind of motion of the suspicious person, and the suspicious person. A suspicious person information storage means for storing suspicious person information including a condition related to a case element that is a noun related to a person's action, and a verb representing the detected action matches a verb included in the suspicious person information, And a determination unit that determines that the object that is the subject of the action that satisfies the condition included in the suspicious person information is a suspicious person, as a case element related to the detected action is further included. It is characterized by.

  In the invention according to claim 7, the state acquisition means includes information indicating the motion state of the object, a verb indicating the motion type of the object, and a case element that is a noun related to the motion of the object. A plurality of pieces of state information representing respective motion states of the plurality of objects are obtained from state storage means for storing state information represented by a case frame indicating a semantic relationship between the verb and the case element. The state acquisition step and the detection means detect the motion of the object corresponding to the combination based on a rule that predetermines the motion of the object according to the combination of the verbs included in each of the acquired plurality of state information And a detecting step.

  According to the present invention, based on a predetermined rule, a motion of an object corresponding to a combination of verbs representing the motion type of each object included in a plurality of state information representing the motion state of each of the plurality of objects is detected. Therefore, there is an effect that the motion of the object can be detected with high accuracy.

  Exemplary embodiments of an operation detection apparatus and an operation detection method according to the present invention will be explained below in detail with reference to the accompanying drawings.

  The motion detection apparatus according to the present embodiment analyzes the behavior (motion state) of a person who is a monitoring target from image information captured by a surveillance camera, and stores the analysis result in a deep case frame structure. To do. Then, the person's action is further detected by comparing the deep case frames stored in the storage unit with each other. Specifically, when a combination of actions of a plurality of persons generated at the same time matches a predetermined combination, it is detected that a predetermined action corresponding to the combination has occurred.

  In the following, an example in which the motion detection device is realized as a security device that guards a predetermined area using information from a monitoring camera and various sensors will be described. Applicable devices are not limited to security devices, and can be applied to any device that has a function of analyzing image information and detecting the motion of an object.

  The present embodiment will be described with reference to the accompanying drawings. First, a configuration example of a monitoring system to which this exemplary embodiment is applied will be described. FIG. 1 is a block diagram showing a configuration of a monitoring system 10 according to the present embodiment. As shown in FIG. 1, the monitoring system 10 includes a security device 100 and a monitoring center 200. The security device 100 and the monitoring center 200 are connected via a network 300 such as a telephone line, a wireless network, or the Internet.

  The monitoring center 200 receives a report from the security device 100 that has detected an abnormality in the monitoring area, and instructs the standby security guard to go to the monitoring area (not shown) where the abnormality has been detected. Report to relevant agencies such as the police and fire departments.

  The security device 100 includes a sensor 110, a monitoring camera 120, and a security operation unit 130.

  The sensor 110 is a human sensor that is installed in a monitoring area that is a security target and is mainly installed for the purpose of detecting an abnormality such as an intruder. The sensor 110 outputs a detection signal when the presence of a person or the opening / closing of a door is detected. The sensor 110 is, for example, an infrared sensor that detects the presence of a person based on a change in the amount of received infrared light, a blocking sensor that detects the presence of a person by blocking reception of infrared light, and the presence of a person due to disturbance of electromagnetic waves. Can be configured by various sensors that detect abnormalities in the monitoring area, such as a radio wave sensor that detects the opening and closing of the door with a magnet.

  The monitoring camera 120 images a monitoring area that is a security target. The monitoring camera 120 is installed mainly in the vicinity of the entrance / exit of a residence, which is a monitoring area, or in the vicinity of the entrance / exit to the site.

  Next, the security operation unit 130 will be described. The security operation unit 130 includes a detection information reception unit 131, an image information acquisition unit 132, a security state switching unit 133, an operation panel control unit 134, an operation panel 135, a transmission / reception unit 136, and a security state storage unit 137. , A first detection unit 138, a state acquisition unit 139, a second detection unit 140, a suspicious person determination unit 141, and an HDD (Hard Disk Drive) 150.

  The detection information receiving unit 131 receives detection information that is sent when an abnormality is detected by the sensor 110. The detection information is a detection signal sent from the sensor 110. When a plurality of sensors 110 are installed, the detection information includes a detection signal and a sensor ID that identifies the sensor 110. The image information acquisition unit 132 acquires image information captured by the monitoring camera 120.

  The guard state switching unit 133 stores the guard state received by the operation panel control unit 134 in the guard state storage unit 137.

  The security state storage unit 137 stores either a security mode or a security release mode that is a security state of the current security device 100. The security state storage unit 137 can be configured by any storage medium that is generally used, such as a hard disk, an optical disk, a memory card, and a RAM (Random Access Memory).

  Here, the security mode refers to a state in which alarm information notifying abnormality in the security area is transmitted to the monitoring center 200 connected via the network 300 when the sensor 110 detects abnormality and acquires detection information. . The security release mode refers to a state in which alarm information notifying abnormality is not transmitted to the monitoring center 200 even when the detection information is acquired.

  The operation panel 135 displays information related to the security device 100, and information is input to the security device 100. The operation panel 135 inputs the security state of either the security mode or the security release mode by pressing a button or the like. The operation panel control unit 134 controls the operation panel 135 such as outputting display information to the operation panel 135 and acquiring input information input from the operation panel 135.

  The transmission / reception unit 136 transmits / receives various information to / from the monitoring center 200 via the network 300. For example, the transmission / reception unit 136 transmits image information captured by the monitoring camera 120 to the monitoring center 200. In addition, the transmission / reception unit 136 transmits alarm information to the monitoring center 200 when the detection information is received from the sensor 110 and the security mode is stored in the current security state stored in the security state storage unit 137. Send.

  The HDD 150 stores various types of data related to object motion detection. Specifically, the HDD 150 stores a state information DB (database) 151, a rule DB 152, and a suspicious person information DB 153. Each DB may be configured to be stored in a storage medium other than the HDD 150 such as an optical disk or a memory card.

  The first detection unit 138, the second detection unit 140, the HDD 150, and the like are provided in the monitoring center 200, information obtained by the sensor 110 and the monitoring camera 120 is sent to the monitoring center 200, and operation detection is performed at the monitoring center. It may be configured.

  The state information DB 151 stores state information representing the motion state of the object in a deep case frame structure. A deep case frame (hereinafter also simply referred to as a case frame) refers to a language structure that expresses the semantic relationship of a noun to a verb as a deep case, based on the verb.

  FIG. 2 is a diagram illustrating an example of a data structure of state information stored in the state information DB 151. As shown in FIG. 2, the state information includes a case frame ID that identifies a case frame, an action name that represents the type of action (action), and a case element (deep case) that is a noun related to the action. Yes. The deep case includes a main case, an attribute case, a place case, a time case, a source case, a target case, a target case, and a tool case.

  The main character represents the subject that causes the action. The attribute case refers to information on the main case. For example, the attribute case includes height, body type, upper body color, lower body color, head color, face information, gait information, and the like.

  Face information and gait information are further divided into detailed elements. For example, the face information includes elements such as a facial feature representing a facial feature, a line-of-sight direction, the presence / absence of a mask, the presence / absence of sunglasses, gender, and age. The gait information includes elements such as stride length, posture, leg length, and walking speed.

  A place case represents a place where an action or a state occurs. For example, the place case includes a head position, a body position, a hand position, a face position, a leg position, and the like.

  The time case represents the time at which an action or condition occurs. For example, the time case represents the time representing the current time, the operation start time representing the time when the operation was started, the operation end time representing the time when the operation was completed, and the date when the operation occurred.

  The source case represents the starting point of movement or state change. For example, the source case includes location information indicating the position of the starting point where the leading character moves or moves, and attribute information indicating the attribute of the leading character at the starting point.

  The target case represents an end point such as movement or state change. For example, the target case includes location information indicating the position of the end point where the main character moves or moves, and attribute information indicating the attribute of the main character at the end point.

  A target case represents an object that is the target of an action or state. The target case includes a target ID that is unique identification information and attribute information that is unique information.

  The tool scale represents an auxiliary means of movement. The tool case includes a tool ID that is unique identification information and attribute information that is unique information.

  In addition, the deep case which can be utilized is not restricted to these, For example, all other deep cases, such as the experience person's case showing the experience person of a mental event, can be used.

  FIG. 3 is a diagram illustrating a specific example of the state information stored in the state information DB 151. FIG. 3 shows an example of state information generated when the person A (main character) detects the action of “squatting”. In this case, specific values such as the height and body shape of the person A are set in the attribute case. In the place case and the time case, specific values relating to the position and time when the person A squats down are set.

  Returning to FIG. 1, the rule DB 152 stores an action detection rule that is referred to when the second detection unit 140 described later detects an action by comparing the case frames stored in the state information DB 151. FIG. 4 is a diagram illustrating an example of the data structure of the action detection rule stored in the rule DB 152.

  As shown in FIG. 4, the behavior detection rule is information that is a comparison source, that is, a temporal frame that is state information that is temporally previous, and information that is a comparison destination that is temporally subsequent. This is a format in which the post-grade frame that is information, other conditions, and detected state information are associated with each other.

  Each of the front case frame and the back case frame includes an action name and case information. In the case information, at least one deep case to be compared is set.

  The front case frame and the back case frame correspond to information (difference information) representing a pattern in which two case frames have a difference. That is, when two case frames are matched with the action name and case information set in the front case frame and the back case frame, it is determined that there is a difference between the two case frames. For example, in FIG. 4, the action name of a case frame (case frame 1) is “exists”, the body position included in the place case is “A”, and the action names of other case frames (case frame 2) are When the body position included in the place case is “B”, it is determined that there is a difference between the case frame 1 and the case frame 2. When it is determined that there is a difference, an action with the action name described in the “detected state information” column is detected as an action corresponding to the difference.

  In other conditions, conditions to be satisfied by the state information to be compared are set as necessary. For example, in the behavior detection rule for detecting the behavior with the behavior name “squatting” in the same figure, the relationship B> C between the head position B included in the front case frame and the head position C included in the back case frame. Satisfying is set as a condition.

  The detected state information includes the action name of the detected action and case information representing at least one deep case to be saved. In the figure, for example, when an action with the action name “move” is detected, the body position A included in the front case frame and the body position B included in the back case frame are respectively corresponding to the detected actions. An example of saving as the body position A of the source case of the frame and the body position B of the target case is shown.

  The behavior detection rule is not limited to the format shown in the figure, and any format rule can be applied as long as a predetermined behavior can be determined according to a difference between a plurality of state information.

  FIG. 5 is a diagram illustrating a transition relationship between detected actions. Specifically, FIG. 5 shows the relationship between the behavior detected from the image information and the behavior detected by the behavior detection rule by comparing the status information stored in the status information DB 151. The ellipse in the figure represents an example of the action name to be detected. Moreover, the character string attached | subjected to the arrow between ellipses represents the reference | standard of the difference between the state information for detecting another action from a certain action.

  This criterion corresponds to the action detection rule stored in the rule DB 152 of FIG. For example, the criterion “variation in position coordinates” corresponds to the first action detection rule for detecting the action name “move” in FIG. That is, for example, when a difference “positional mutation” is detected by comparing a plurality of state information corresponding to the action name 501 (exists) in FIG. Is newly detected.

  Note that action names 501 to 504 in FIG. 5 represent action names detected from the image information by the first detection unit 138 described later. The other behavior names represent the behavior names detected by the second detection unit 140 described later using the behavior detection rules. Also, action names 511 to 513 in the same figure represent action names that are not derived from other actions. That is, when this state is reached, it means leaving from the transition of FIG.

  In the present embodiment, the rule DB 152 further stores an action detection rule that is referred to when detecting an operation corresponding to a combination of state information of a plurality of objects (persons). Hereinafter, the action detection rule for detecting an action from the states of a plurality of objects is referred to as a situation grasping rule. FIG. 6 is a diagram illustrating an example of a data structure of the situation grasp rule.

  As shown in FIG. 6, the situation grasp rule has a format in which information on a certain person A, information on another person B, other conditions, and detected state information are associated with each other.

  The information about the person A and the information about the person B include action names and case information, respectively. In the case information, a deep case condition to be satisfied by each person is set.

  When the two case frames corresponding to each of two persons match the action name and case information set by the information about person A and the information about person B, they are listed in the “detected state information” column. The action with the action name is detected.

  In other conditions, conditions to be satisfied by the state information to be compared are set as necessary. For example, in the situation grasping rule for detecting the action with the action name “threat” in the figure, there is a case frame with the action name “see”, and the main case and the target case of the case frame are person A and person B, respectively. The condition is set.

  The detected state information includes the action name of the detected action and case information representing at least one deep case to be saved. In the same figure, for example, when an action with an action name “threat” is detected, an example is shown in which it is stored as a target case of a case frame corresponding to the action in which the person B is detected.

  The situation grasping rules are not limited to the form shown in the figure, and any form of rules can be applied as long as predetermined actions can be determined according to a combination of state information representing the actions of multiple persons. it can. Moreover, although the rule which detects an action from the combination of the status information of two persons is illustrated in the same figure, you may use the rule which detects an action from the combination of the status information of three or more persons.

  As a combination of actions of a plurality of persons, for example, a combination of actions of a victim and a perpetrator such as a criminal act or a surrounding person when an abnormal situation occurs is assumed. Then, a situation grasp rule is set in which the behavior predicted when such a combination of behaviors of a plurality of people occurs is set in the detected state information.

  For example, the first and second lines in the figure represent the situation grasp rules corresponding to the situation when the person A is the perpetrator and the person B is the victim. Also, the third line in the figure represents a situation grasp rule for detecting suspicious behavior from a combination of behaviors of a plurality of people. By using such rules, for example, even if an individual person's action seems to be suspicious but cannot be determined as suspicious, It can be determined that there is.

  Returning to FIG. 1, suspicious person information DB153 memorize | stores suspicious person information predetermined as an action pattern showing the operating state of a suspicious person. FIG. 7 is a diagram illustrating an example of a data structure of suspicious person information stored in the suspicious person information DB 153 according to the present embodiment. As shown in FIG. 7, the suspicious person information includes an action name indicating the type of action of the suspicious person, a deep case (case information) related to the action of the suspicious person, and a condition to be satisfied by the deep case. Yes.

  The shaded area in the figure shows a suspicious person or a suspicious behavior when the action name is "Korokoro" and the body position of the place character is "in front of the entrance", that is, when the action is in front of the entrance. An example stored as is shown.

  Note that the action name, the type of deep case, and the like stored in the suspicious person information DB 153 are stored in advance by the user.

  Returning to FIG. 1, the first detection unit 138 detects the motion state of the object from the image information by analyzing the image information acquired by the image information acquisition unit 132. Further, the first detection unit 138 generates state information representing the detected operation state and stores it in the state information DB 151.

  The first detection unit 138 detects an operation state in which a person “exists”, for example, by extracting a person region from image information. In addition, the first detection unit 138 detects an operation state of “seeing” the person by detecting the detected gaze direction of the person. Further, the first detection unit 138 extracts an object other than a person from the image information, and when the extracted object newly appears, an operation state of “take out” is present, and the extracted object has existed before. When it overlaps with the position of a person's hand, an operation state of “touch” is detected.

  In addition, as a method for the first detection unit 138 to extract a person or an object, a variation area extracted by comparing backgrounds of continuous image information is used as a person area, or a person pattern or an object pattern is collated. Any method conventionally used, such as a method of extracting a person or an object by the above, can be applied. In addition, as a method for the first detection unit 138 to detect the gaze direction of a person, any conventionally used method such as a method of extracting a face area by collating with a face area pattern and detecting the gaze direction is used. Applicable.

  The state acquisition unit 139 acquires a plurality of state information from the state information DB 151. For example, the state acquisition unit 139 acquires two pieces of state information having consecutive times among the state information detected by the first detection unit 138 and stored in the state information DB 151. Further, the state acquisition unit 139 acquires two pieces of state information having different main characters at the same time among the state information stored in the state information DB 151. The former is used when detecting a behavior by the behavior detection rule of FIG. The latter is used when an action is detected by the situation grasping rule of FIG. Note that the time does not need to be exactly the same, and may be configured to be determined to be substantially the same if the error is within a predetermined time range.

  The second detection unit 140 detects a new action by determining whether or not two pieces of state information with successive times are compatible with the action detection rule stored in the rule DB 152.

  Specifically, the second detection unit 140 acquires, from the rule DB 152, an action detection rule in which the action name and the case information of the two pieces of state information having the same time acquired by the state acquisition unit 139 match. Then, when the behavior detection rule can be acquired, the second detection unit 140 detects the behavior defined by the behavior detection rule as a new behavior.

  Moreover, the 2nd detection part 140 detects a new action by determining whether two state information from which a main character differs at the same time corresponds to the condition grasping | storing rule preserve | saved in rule DB152.

  Specifically, the second detection unit 140 acquires, from the rule DB 152, a situation grasp rule in which action names and case information of two state information having different main characters at the same time acquired by the state acquisition unit 139 match. And the 2nd detection part 140 detects the action defined by the situation grasping rule as a new action, when the situation grasping rule can be acquired.

  The suspicious person determination unit 141 determines whether or not there is a suspicious person by searching the state information DB 151 for state information that matches the suspicious person information stored in the suspicious person information DB 153. Specifically, the suspicious person determination unit 141 matches the action name of the state information with the action name of the suspicious person information among the state information stored in the state information DB 151, and the suspicious person among the case elements of the state information. Search for state information in which the case element described in the case information of the person information satisfies the conditions of the suspicious person information. Then, the suspicious person determination unit 141 determines that the person who is the main character of the state information that satisfies the condition is a suspicious person.

  Next, operation detection processing from an image by the security device 100 according to the present embodiment configured as described above will be described with reference to FIG. The operation detection process from an image refers to a process for detecting an operation (behavior) of a person or the like from image information captured by the monitoring camera 120. FIG. 8 is a flowchart showing an overall flow of the operation detection process from an image according to the present embodiment.

  First, the image information acquisition unit 132 inputs image information captured by the monitoring camera 120 (step S601). Next, the first detection unit 138 detects a person area from the image information (step S602). Next, the first detection unit 138 determines whether a person area has been detected (step S603). If no person area is detected (step S603: NO), the next image information is input and the process is repeated (step S601).

  When the person area is detected (step S603: YES), the first detection unit 138 creates a case frame with action name = “exists” (step S604). A value that does not overlap with other case frames is assigned to the case frame ID of the created case frame.

  Next, the 1st detection part 138 stores the detection position of each site | part of the detected person area in a place case (step S605). For example, the first detection unit 138 calculates the coordinates of the head, body, hand, face, and leg included in the detected person region, and the head position, body position, hand position, face position, and leg, respectively. Stored in a place case as a part position. It should be noted that each part of the person region can be detected by any conventionally used method such as a method of detecting by matching with a matching pattern for each part.

  In parallel with step S604 to step S605, the first detection unit 138 performs person identification processing (step S606 to step S607), face detection processing (step S608 to step S610), body type determination processing (step S611 to step S612), And each process of a gait detection process (step S613-step S614) is performed. Note that these processes are not necessarily performed in parallel, and can be configured to be executed in an arbitrary order.

  In the person identification process, the first detection unit 138 identifies the extracted person by a method of collating with the extracted person region (step S606). Then, the first detection unit 138 stores the agent ID already assigned to the identified person (AgtID in FIG. 2) in the main case of the created case frame (step S607).

  In the face detection process, the first detection unit 138 detects a face area by a method of matching with a face area pattern (step S608). Further, the first detection unit 138 stores face information such as the line-of-sight direction, the presence / absence of a mask, and the presence / absence of sunglasses in the attribute case according to the collation result (step S609). When the line-of-sight direction is detected, the first detection unit 138 creates a case frame of action name = “see” (step S610).

  In the body type determination process, the first detection unit 138 determines the body type of the detected person by a method of obtaining a body type given in advance to the collated person pattern (step S611). Then, the first detection unit 138 stores the determined body type in the attribute case of the created case frame (step S612).

  In the gait detection process, the first detection unit 138 detects gait information such as a gait speed, a stride length, a leg length, and a posture mainly by analyzing a leg portion of the detected human area (step S613). The walking speed can be calculated, for example, by comparing the positions of the leg portions of successive image information. Then, the first detection unit 138 stores the detected gait information in the attribute case of the created case frame (step S614).

  Next, the first detection unit 138 detects an object from the image information (step S615). The object detection process may be executed together with the person area detection process (step S602).

  Next, the first detection unit 138 determines whether or not an object has been detected (step S616). When an object is detected (step S616: YES), the first detection unit 138 determines whether the object has been detected based on the previous image information and the positional relationship between the object and the person area, action name = “ A case frame of “touch” or “take out” is created (step S617).

  Specifically, the first detection unit 138 can determine that the object has already been detected, for example, by collating with the object detected in the previous image information, and can also determine the position of the person's hand and the position of the object. If it can be determined that they overlap, a case frame of action name = “touch” is newly created. Further, if the first detection unit 138 can determine that an undetected object is newly detected and can determine that the position of the person's hand and the position of the object overlap, the action name = “take out” case. Create a new frame.

  When it is determined in step S616 that no object has been detected (step S616: NO), or after creating a new case frame in step S617, the first detection unit 138 uses the case frame created so far to the state information DB. (Step S618), and the motion detection process from the image is terminated.

  Next, operation detection processing from state information by the security device 100 according to the present embodiment will be described with reference to FIG. The operation detection process from the state information refers to a process of detecting an operation (behavior) of a person or the like from the state information stored in the state information DB 151. FIG. 9 is a flowchart showing an overall flow of the operation detection process from the state information in the present embodiment.

  First, the state acquisition unit 139 acquires an unprocessed case frame (hereinafter referred to as a first case frame) among the case frames that are state information stored in the state information DB 151 (step S701). Next, the state acquisition unit 139 acquires a case frame (hereinafter referred to as a second case frame) that is temporally continuous with the acquired first case frame from the state information DB 151 (step S702).

  Next, the second detection unit 140 searches the rule DB 152 for action detection rules that are satisfied by the acquired two case frames (step S703). Specifically, the second detection unit 140 matches the action names of the first case frame and the second case frame with the action names of the front case frame and the back case frame of the action detection rule, respectively, and the case information detects the action. A behavior detection rule that satisfies the condition represented by the rule case information is searched.

  For example, the action names of the first case frame and the second case frame are both “exist”, and the body position of the first case frame (referred to as body position A) and the body position of the second case frame (body position B) The first action detection rule in the rule DB 152 as shown in FIG. 4 is searched.

  Next, the second detection unit 140 determines whether or not a behavior detection rule has been searched (step S704). If the behavior detection rule has been searched (step S704: YES), the “detected state” of the searched behavior detection rule is determined. A case frame of the action name set in “information” is created (step S705).

  If no action detection rule is found (step S704: NO), or after creating a case frame, the second detection unit 140 determines whether there is an unprocessed case frame (step S706).

  When there is an unprocessed case frame (step S706: YES), the state acquisition unit 139 acquires the next case frame and repeats the process (step S701). If there is no unprocessed case frame (step S706: NO), the operation detection process from the state information is terminated.

  By such processing, it is possible to detect the behavior of an object such as a person not only from image information but also from state information stored as text information. For this reason, it is possible to reduce the processing load of the motion detection process as compared with the conventional method of recognizing the motion of the monitoring object only by analyzing the image.

  Next, the situation grasping process by the security device 100 according to the present embodiment will be described with reference to FIG. The situation grasping process refers to a process of grasping the situation of the operation from the state information of a plurality of persons (objects) stored in the state information DB 151 and detecting the action (behavior) of a new person or the like. FIG. 10 is a flowchart showing the overall flow of the situation grasping process in the present embodiment.

  First, the state acquisition unit 139 acquires an unprocessed case frame (hereinafter referred to as a first case frame) among the case frames that are state information stored in the state information DB 151 (step S1001). Next, the state acquisition unit 139 acquires another case frame (hereinafter referred to as a second case frame) at the same time as the acquired first case frame from the state information DB 151 (step S1002).

  Next, the second detection unit 140 determines whether or not case frames of a plurality of persons have been acquired, that is, whether or not the main case of the first case frame is different from the main case of the second case frame (step S1003). ). When the case frames of a plurality of persons are acquired (step S1003: YES), the second detection unit 140 acquires the action name of each person's case frame (step S1004).

  Next, the second detection unit 140 determines whether or not a case frame of action name = “see” exists (step S1005). If it exists (step S1005: YES), the second detection unit 140 calculates the number of other persons with the same object (person) as the action name = “see”, and the number is larger than a predetermined threshold. Whether or not (step S1006). For example, the second detection unit 140 sets the same object (person) as the object (person) set as the target case of the first case frame or the second case frame to the target case of action name = “see”, The number of other case frames with different main cases is calculated and compared with a threshold value.

  If the number of persons is greater than the threshold (step S1006: YES), the second detection unit 140 determines that an abnormal situation has occurred because a plurality of persons are paying attention to the specific person, and an abnormality has occurred. Information indicating that this has been done is output (step S1007).

  As described above, according to the present embodiment, not only the abnormal behavior of a specific person is detected from the status information of a plurality of people, but also that an abnormal situation has occurred from the status information of the plurality of people. You can also In the above example, a specific target is judged to be abnormal when an action of “seeing” occurs by a plurality of people. However, the action is not limited to “seeing”, and a predetermined specific action is determined. You may comprise so that it may judge as object. In this case, the number of state information including a specific verb representing a specific action may be calculated to determine whether the number is greater than a predetermined threshold.

  When the number of persons is not greater than the threshold (step S1006: NO), the second detection unit 140 sets the main case of the action frame with the action name = “see” as the person A and the target among the first case frame and the second case frame. The case is acquired as person B (step S1008). Then, the second detection unit 140 searches the rule DB 152 for a situation grasp rule that is satisfied by the case frame of the person A and the case frame of the person B (step S1009). Specifically, the second detection unit 140 matches the action names of the case frame of the person A and the case frame of the person B with the action names of the information about the person A and the information about the person B in the situation grasp rule, respectively. A situation grasp rule that satisfies the condition represented by the case information of the situation grasp rule is searched.

  For example, when the action name of the person A's case frame is “push out a hand”, the action name of the person B ’s case frame is “raise his hand”, and the target case of the person A ’s case frame is the person B The first rule of the situation grasping rule as shown in FIG. 6 is searched.

  If it is determined in step S1005 that there is no case frame of action name = “see” (step S1005: NO), the second detection unit 140 grasps the situation that the first case frame and the second case frame satisfy. A rule is searched from the rule DB 152 (step S1010). Specifically, the second detection unit 140 matches the action names of the first case frame and the second case frame with the action names of the information related to the person A and the information related to the person B in the situation grasp rule, and the case information is A situation grasp rule satisfying the condition indicated by the case information of the situation grasp rule is searched.

  For example, the action name of the first case frame is “move”, the action name of the second case frame is “look around”, and the points indicated by the location case of the first case frame and the second case frame match ( For example, in the case of point A), the rule in the third line of the situation grasp rule as shown in FIG. 6 is searched.

  Next, the second detection unit 140 determines whether or not a situation grasp rule has been searched (step S1011), and if found (step S1011: YES), the “detected state” of the searched situation grasp rule. A case frame of the action name set in “information” is created (step S1012).

  If a case frame of a plurality of persons is not acquired in step S1003 (step S1003: NO), a case grasping rule is not searched in step S1011 (step S1011: NO), or after creating a case frame, the second The detection unit 140 determines whether there is an unprocessed case frame (step S1013).

  When there is an unprocessed case frame (step S1013: YES), the state acquisition unit 139 acquires the next case frame and repeats the process (step S1001). If there is no unprocessed case frame (step S1013: NO), the situation grasping process is terminated.

  By such processing, not only the state information of each object (person) is detected from the difference (mutation) of the state information of each object (person), but also the object (from the state information of a plurality of objects (persons)) It is possible to grasp the situation of the action of the person and detect the action (motion) with higher accuracy.

  Next, the suspicious person determination process by the security device 100 according to the present embodiment configured as described above will be described with reference to FIG. FIG. 11 is a flowchart showing the overall flow of the suspicious person determination process in the present embodiment.

  In each process shown in the figure, a suspicious person information DB 153 as shown in FIG. 7 is stored in advance by the user or the monitoring center 200, and further, an instruction for making a suspicious person determination is given by the user from the operation panel 135 or the like. And

  First, the suspicious person determination unit 141 reads the stored suspicious person information with reference to the suspicious person information DB 153 (step S1101). Then, using the read suspicious person information as a key, a deep case frame including the suspicious person information is searched from the state information DB 151 (step S1102).

  Then, it is determined whether or not there is a deep case frame including suspicious person information in the state information DB 151 (step S1103), and when it is determined that there is no deep case frame including suspicious person information (step S1103: NO). ), Proceed to the next step without doing anything.

  On the other hand, when it is determined that there is a deep case frame including suspicious person information (step S1103: YES), the deep case frame including the suspicious person information is displayed on the operation panel 135 (step S1104). Thereafter, it is determined whether or not all the suspicious person information rules stored in the suspicious person information DB 153 are referred to (step S1105), and when it is determined that all the suspicious person information rules are not referred to (step S1105). (S1105: NO), the process returns to step S1101, and the subsequent processing is repeated. On the other hand, when it is determined that all the suspicious person information rules have been referred to (step S1105: YES), the suspicious person determination process ends.

  Next, a specific example of the suspicious person determination process based on the analysis of the status information of a plurality of persons will be described with reference to FIGS. FIG. 12 is a diagram illustrating an example of an image photographed in this example. In the drawing, an example of a situation in which a person on the right side (referred to as a person A) sticks out a hand against the person on the left side (referred to as a person B) and the person B raises both hands is shown.

  FIGS. 13 and 14 are diagrams showing examples of case frames representing the operating states of the person A and the person B detected from the image of FIG. 12, respectively.

  In the example of FIG. 12, it is described that the person A is in an operation state in which a hand is sticking out against the person B. In the example of FIG. 14, it is described that the person B is in an operating state in which his hand is raised. When such state information is detected, the second detection unit 140 applies the rule in the first line of the situation grasping rule as shown in FIG. 6 so that the person A threatens the person B. Can be detected.

  FIG. 15 is a diagram illustrating an example when the case frame of FIG. 12 is updated with the detected action name. As shown in FIG. 15, the action name that was “push out a hand” in FIG. 13 has changed to “threat”. Instead of updating the action name, a case frame of a new detected action name may be created and stored in the state information DB 151.

  Further, when such state information is detected, the suspicious person determination unit 141 applies the suspicious person information on the sixth line of the suspicious person information DB 153 as illustrated in FIG. Person A who is the main character can be detected as a suspicious person.

  As described above, in the motion detection device according to the present embodiment, abnormal behavior and suspicious behavior that cannot be determined from the status information of one individual's behavior can be detected more accurately by using the status information of a plurality of people. .

  It should be noted that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

It is a block diagram which shows the structure of the monitoring system concerning this Embodiment. It is a figure which shows an example of the data structure of the status information memorize | stored in status information DB. It is a figure which shows the specific example of the status information memorize | stored in status information DB. It is a figure which shows an example of the data structure of the action detection rule memorize | stored in rule DB. It is a figure showing the transition relation between the detected actions. It is a figure which shows an example of the data structure of a situation grasp rule. It is a figure which shows an example of the data structure of suspicious person information memorize | stored in suspicious person information DB. It is a flowchart which shows the whole flow of the operation | movement detection process from the image in this Embodiment. It is a flowchart which shows the whole flow of the operation | movement detection process from the status information in this Embodiment. It is a flowchart which shows the flow of the whole situation grasping | ascertainment process in this Embodiment. It is a flowchart which shows the flow of the whole suspicious person determination process in this Embodiment. It is a figure showing an example of a photoed image. It is a figure which shows an example of the case frame showing the operation state of the person A detected from the image of FIG. It is a figure which shows an example of the case frame showing the operation state of the person B detected from the image of FIG. It is a figure which shows the example at the time of updating the case frame of FIG. 12 with the detected action name.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Monitoring system 100 Security apparatus 110 Sensor 120 Monitoring camera 130 Security operation part 131 Detection information reception part 132 Image information acquisition part 133 Security state switching part 134 Operation panel control part 135 Operation panel 136 Transmission / reception part 137 Security state storage part 138 1st detection Unit 139 Status acquisition unit 140 Second detection unit 141 Suspicious person determination unit 150 HDD
151 Status information DB
152 Rule DB
153 Suspicious person information DB
200 Monitoring Center 300 Network 501-504, 511-513 Action Name

Claims (7)

Information representing the motion state of an object, including a verb indicating the type of motion of the object and a case element that is a noun related to the motion of the object, and a semantic relationship between the verb and the case element State storage means for storing state information represented by the case frame shown;
State acquisition means for acquiring a plurality of the state information representing the operation state of each of the plurality of objects from the state storage means;
Detection means for detecting an action of an object corresponding to the combination based on a rule that predetermines an action of the object according to a combination of the verbs included in each of the acquired plurality of state information;
A motion detection apparatus comprising: The state information includes a time when the movement of the object occurs as the case element,
The status acquisition means acquires a plurality of the status information having substantially the same time from the status storage means;
The motion detection apparatus according to claim 1. The state information includes, as the case elements, a main case that represents an object that is the subject of the action and a target case that represents another object that is the target of the action by the main case,
The rule includes a combination of the verb representing the type of the action based on the main case and the verb representing the type of the action mainly based on the target case, and the action determined according to the combination,
The detection means matches the target case of the first state information, which is one of the state information, with the main case of the second state information, which is the other state information, among the plurality of state information acquired. Detecting the action corresponding to the matched combination when the combination of the verb of the first state information and the verb of the second state information matches the combination included in the rule;
The motion detection apparatus according to claim 1. The state information includes, as the case elements, a main case that represents an object that is the subject of the action, and a place case that represents a position where the action has occurred,
The rule includes a combination of the verb representing the type of action of the first object and the verb representing the type of action of the second object, and the action defined according to the combination,
The detection means is configured such that, out of the plurality of acquired state information, the location case of the first state information which is one of the state information and the location case of the second state information which is the other state information are one. If the combination of the verb of the first state information and the verb of the second state information matches the combination included in the rule, detecting the action corresponding to the matched combination;
The motion detection apparatus according to claim 1. The state information includes, as the case element, a target case representing another object to be the target of the action,
The detection means further includes that the verb included in the state information matches a predetermined specific verb among the plurality of acquired state information, and the target cases included in the state information are mutually Detecting the occurrence of an abnormality when the number of the state information that matches the number is greater than a predetermined threshold;
The motion detection apparatus according to claim 1. Suspicious person information stored in advance as a suspicious person's action state, including a verb representing the kind of action of the suspicious person and a condition regarding a case element that is a noun related to the action of the suspicious person Suspicious person information storage means,
A verb representing the detected action matches a verb included in the suspicious person information, and a case element related to the detected action is a subject of the action satisfying the condition included in the suspicious person information. Determination means for determining that the object is a suspicious person,
The motion detection apparatus according to claim 1. The state acquisition means includes information indicating the motion state of the object, and includes a verb indicating the motion type of the object and a case element that is a noun related to the motion of the object, and the verb and the case element A state acquisition step of acquiring a plurality of pieces of state information representing operation states of the plurality of objects, from a state storage unit that stores state information represented by a case frame indicating a semantic relationship between;
A detecting step for detecting an action of an object corresponding to the combination based on a rule in which an action of the object corresponding to the combination of the verbs included in each of the acquired plurality of state information is determined in advance;
An operation detection method comprising:

Images