JP5167188B2 - Operating state detecting device and operating state detecting method - Google Patents

Description

  The present invention relates to an operation state detection apparatus and an operation state detection method for detecting the movement of an object such as a person or an article from image information captured by an imaging unit such as a camera and read information from a wireless tag attached to the article.

  2. Description of the Related Art Conventionally, a recognition device that analyzes an image captured by a monitoring camera or the like and recognizes an action of a monitoring target such as a person is known. For example, in Patent Document 1, it is suspicious from a plurality of monitoring targets by determining whether the pattern is a normal pattern or an abnormal pattern using a pattern recognition technique such as SVM (support vector machine) from video captured by a stereo camera. Techniques have been proposed for detecting an action target to be monitored.

JP 2008-217602 A

  However, the method of Patent Document 1 detects abnormal behavior (walking trajectory) by pattern recognition from the walking trajectory information of a plurality of persons, for example, by using learned walking trajectory data. That is, it is only determined whether each of a plurality of walking trajectories is normal or abnormal. For this reason, for example, when a plurality of people conspire to perform a fraud, it is not possible to detect an action with high accuracy only by pattern recognition of the walking trajectory of each person, resulting in an abnormal action. In some cases, it could not be determined. In addition, an action such as moving an important object, which is an important article, can be performed. However, it is impossible to manage an important object by associating a person with the important object to be moved only by detecting the action of the person.

  The present invention has been made in view of the above, and by using a plurality of information obtained from a person and an article (important object), the movement of the object is detected with high accuracy, and security for the important object is enhanced. An object of the present invention is to provide an operation state detection apparatus and an operation state detection method capable of simultaneously achieving the management.

  In order to solve the above-described problems and achieve the object, the operation state detection device of the present invention is information representing an operation state of an object generated based on captured image information and read information read from a wireless tag. A state represented by a case frame including a verb representing a type of action of the object and a case element that is a noun related to the action of the object, and indicating a semantic relationship between the verb and the case element State information construction means for constructing information, storage means for storing the state information constructed by the state information construction means, and a plurality of the state information representing the operation states of the plurality of objects are acquired from the storage means. A state information acquisition means that performs the action state of the object according to the combination of the verbs included in each of the acquired plurality of state information based on a predetermined rule. Detecting means for detecting the operating state of the object response, characterized in that and an abnormality judging means for judging whether the abnormality based on the detection result by the detection means.

  The operation state detection apparatus of the present invention is information representing an operation state of an object generated based on captured image information and read information read from a wireless tag, the verb representing the type of the object operation, State information constructing means for constructing state information represented by a case frame including a case element that is a noun related to the motion of an object, and indicating a semantic relationship between the verb and the case element, and the state information construction First storage means for storing state information constructed by the means, second storage means for storing management information including at least a storage state, a final handler, and a position of the object of the object, and the storage The state information acquisition means for acquiring a plurality of state information representing the operation state of each of the plurality of objects, and the state information for the object that is the target article are acquired from the first storage means. Fixed asset management means for detecting the presence / absence of movement of the location from the state information with respect to the object, and updating the management information stored in the second storage means when the movement of the location is further detected. And.

  The operation state detection method of the present invention is information representing the operation state of an object generated by the state information construction unit based on the captured image information and the read information read from the wireless tag. A state that includes a verb representing a type and a case element that is a noun related to the action of the object, and that is constructed by constructing state information represented by a case frame indicating a semantic relationship between the verb and the case element A state information construction step for storing information in the storage means; a state information acquisition means for acquiring a plurality of the state information representing the operation states of the plurality of objects from the storage means; and a detection means. Is an object corresponding to the combination based on a rule that predetermines the motion state of the object according to the combination of the verbs included in each of the acquired plurality of state information A detection step of detecting an operating state, the abnormality determination means, characterized in that it comprises a, and an abnormality determination step of determining whether or not an abnormality based on the detection result of the detecting step.

  The operation state detection method of the present invention is information representing the operation state of an object generated by the state information construction unit based on the captured image information and the read information read from the wireless tag. A state that includes a verb representing a type and a case element that is a noun related to the action of the object, and that is constructed by constructing state information represented by a case frame indicating a semantic relationship between the verb and the case element State information construction step for storing information in the first storage means, and state information acquisition means for acquiring the plurality of state information representing the operation states of the plurality of objects from the first storage means. When the acquisition step and the fixed asset management means can acquire the state information for the object that is the target article from the first storage means, the presence / absence of movement of the place from the state information for the object Management information stored in second storage means for storing management information including at least a storage state, a final handler, and a position of the object when the object is detected and further movement of the place is detected. And a fixed asset management processing step to be updated.

  According to the present invention, based on a predetermined rule, an object corresponding to a combination of verbs representing the type of motion of each object included in the plurality of state information representing the motion state of each of the plurality of objects (people and articles) It is possible to detect the movement of an object with high accuracy, and to detect the movement state by associating an article with a person using information read from a wireless tag. There is an effect that it is possible to simultaneously achieve security enhancement and management for articles such as goods.

FIG. 1 is a block diagram showing a configuration of a monitoring system in the present embodiment. FIG. 2 is a diagram illustrating an example of a data structure of state information stored in the state information DB. FIG. 3 is a diagram illustrating a specific example of the state information stored in the state information DB. FIG. 4 is a conceptual diagram of a deep case frame structured in time series. FIG. 5 is a diagram illustrating an example of a data structure of behavior detection rules stored in the behavior derivation rule DB. FIG. 6 is a diagram illustrating an example of the data structure of the article registration DB. FIG. 7 is a diagram illustrating a specific example of access restriction. FIG. 8 is a diagram illustrating an example of the data structure of the environment information DB. FIG. 9 is a diagram illustrating an example of the data structure of the person DB. FIG. 10 is a diagram illustrating an example of the data structure of the article management DB. FIG. 11 is a diagram illustrating a specific example when abnormality determination is performed from state information. FIG. 12 is a diagram illustrating an example of information obtained by image processing and information obtained from the RFID reader in the situation illustrated in FIG. FIG. 13 is a flowchart illustrating an abnormality determination process performed by the abnormality determination unit. FIG. 14 is a diagram illustrating a specific example when processing for fixed asset management is performed from state information. FIG. 15 is a diagram illustrating an example of information obtained by image processing and information obtained from the RFID reader in the situation of FIG. FIG. 16 is a diagram illustrating an update process of the article management DB. FIG. 17 is a flowchart showing fixed asset management processing by the fixed asset management unit.

  Hereinafter, an embodiment of a security device according to the present invention will be described in detail with reference to the accompanying drawings.

  The present embodiment is an example in which the operation state detection device of the present invention is realized as a security device that guards a predetermined area using information from an imaging device such as a monitoring camera and an RFID reader. The device to which the present invention can be applied is not limited to the security device, and the present invention analyzes the information on the object (person and article) from the captured image information and RFID tag information, and detects the operation state of each object. Note that the present invention can be applied to any device having a function to

  The security device according to the present embodiment analyzes the behavior (operation state) of a person who is a monitoring target from image information captured by an imaging device such as a monitoring camera, and represents the analysis result in a deep case frame structure. Save to. In addition, information (RFID tag information) recorded on the RFID tag from an RFID tag (wireless tag) attached to the article is read by the RFID reader, and based on the read RFID tag information and the analysis result of the image information, Information about the article is expressed in a deep case frame structure and stored in the storage unit. Then, by comparing the deep case frames stored in the storage unit with each other, the action of the person and the operation state (particularly, movement) of the article are further detected. Specifically, when the combination of the actions of a plurality of persons and the movement of an article that occurred at the same time matches a predetermined combination, it is detected that a predetermined operation state corresponding to the combination has occurred.

First, a configuration example of a monitoring system including a security device according to the present embodiment will be described. FIG. 1 is a block diagram showing a configuration of a monitoring system 10 in the present embodiment. As shown in FIG. 1, the monitoring system 10 includes a security device 100, a management PC 200, imaging devices (cameras) 300 _{1 to} 300 _n connected to the security device 100, and RFID readers (RFID readers) 400 ₁ to 400 ₁ . 400 _n . Further, as shown in the figure, another PC 200 ′ may be provided. The security device 100 and the management PC 200 are connected via a network such as a telephone line, a wireless or wired network, and the Internet.

  When the security device 100 detects an abnormality, the management PC 200 receives a notification from the security device 100 that has detected an abnormality in the monitoring area, and receives a notification from the monitoring device that has detected an abnormality in the standby security officer The instruction to go to the figure) is given and, if necessary, reports are made to related organizations such as the police and fire department. The management PC 200 and other PCs 200 'are also used for monitoring the current state and position of an article to which an RFID tag is attached.

The imaging devices 300 _{1 to} 300 _n are cameras that image a monitoring area that is a security target, and are installed in each monitoring area.

The RFID readers 400 _{1 to} 400 _n are installed in each monitoring area to be guarded, and read RFID tag information (ID or the like) from RFID tags attached to articles that are mainly important objects.

  Next, the security device 100 will be described. The security device 100 includes an image input unit 111, an image processing unit 112, a state information construction unit 113, a data input unit 114, an abnormality determination unit 115, a fixed asset management unit 116, and an access control unit 117, and a state And a storage unit 120 including an information DB (database) 121, an action derivation rule DB 122, an article registration DB 123, an environment information DB 124, a person DB 125, and an article management DB 126. The control unit 110 includes a CPU (processor) and a memory, and the function of each unit is realized by the CPU executing a program corresponding to each unit loaded in the memory. In addition, the storage unit 120 including each DB can be configured by any storage medium that is generally used, such as a hard disk, an optical disk, a memory card, and a RAM (Random Access Memory). The storage unit 120 also stores the program.

Image information captured by the imaging devices 300 _{1 to} 300 _n is input to the image input unit 111.

  The image processing unit 112 analyzes the image information input to the image input unit 111 to detect an operation state and a feature of the object from the image information (details will be described later). Then, the state information construction unit 113 generates state information representing the operation state detected by the image processing unit 112 and stores it in the state information DB 121.

  For example, the image processing unit 112 extracts the person region from the image information, thereby detecting an operation state in which the person “exists”. In addition, the image processing unit 112 detects an operation state of “seeing” the person by detecting the detected gaze direction of the person. Further, the image processing unit 112 extracts an object other than a person from the image information, and when the extracted object newly appears, the image processing unit 112 sets an operation state of “removal”, and the extracted object has existed before. When the position of the hand overlaps, an operation state of “touch” is detected.

  In addition, as a method for the image processing unit 112 to extract a person or an article, a variation area extracted by comparing backgrounds of continuous image information is used as a person area, or by comparing with a person pattern or an article pattern. Any conventionally used method such as a method of extracting a person or an article can be used. In addition, as a method for the image processing unit 112 to detect the gaze direction of the person, any conventional method for extracting a face area and detecting the gaze direction by matching with a face area pattern may be used. it can.

The RFID tag information read by the RFID readers 400 _{1 to} 400 _n is input to the data input unit 114. The state information construction unit 113 constructs the operation state of the object (person or article) as state information by a deep case frame described later based on the analysis result by the image processing unit 112 and the RFID tag information input to the data input unit 114. To do. A plurality of imaging devices 300 _{1 to} 300 _n and RFID reading devices 400 _{1 to} 400 _n are identified by their IDs, and input information is used together with the IDs. Since these IDs can associate each device with the area in which they are installed, the IDs identify the areas where the objects detected by the imaging devices 300 _{1 to} 300 _n and the RFID readers 400 _{1 to} 400 _n are located. Can be used for

  The access control unit 117 performs access control when transmitting / receiving various types of information to / from the management PC 200 and other PCs 200 'via the network. Details of the abnormality determination unit 115 that detects an abnormal operation state and the fixed asset management unit 116 that performs processing for management of articles such as important items will be described in detail later.

  The state information DB 121 stores the state information, which is constructed by the state information construction unit 113 and expresses the motion state of the object in a deep case frame structure (described later) in time series (see FIG. 4). The deep case frame (hereinafter also simply referred to as a case frame) refers to a language structure that expresses the semantic relationship of a noun to a verb as a deep case with a verb as a reference.

  FIG. 2 is a diagram illustrating an example of a data structure of state information stored in the state information DB 121. As shown in FIG. 2, the status information includes a case frame ID (CFID) for identifying a case frame, an action name (ActNm) indicating the type of action (action) such as “squatting”, and a noun related to the action. It contains a case element (deep case). The deep case includes a main case (Agent), an attribute case (Feature), a place case (PosCase), a time case (TimeCase), a source case (SrcCase), a target case (DstCase), a target case (ObjCase), and a tool case ( ToolCase).

  FIG. 3 shows a specific example of the state information stored in the state information DB 151. The main character represents a subject that causes an action, and specifically includes a tracking number (ID) and an authentication name (Name) as shown in FIG. The attribute case refers to information on the main case. For example, the attribute case is divided into a facial feature (Face), a body feature (Body), and a walking feature (Gate). The face feature (Face) includes face detection reliability (FConf), face feature (Ident), mouth opening / closing degree (Mouth), gaze direction (PanTilt), presence / absence of mask (Mask), presence / absence of sunglasses (SG), gender ( Sex), age (Age), head color (HCColor) and the like. The body features include height, upper body color (BColor), lower body color (LCColor), and the like. The walking features include a step width, a body shape, a leg length, a walking speed, and the like.

  A place case represents a place where an action or a state occurs. For example, the location case includes a location name (PName), head coordinates (Head), body coordinates (Body), foot coordinates (Leg), right-hand coordinates (RHnad), LHand (left-hand coordinates), barycentric coordinates (WB), and the like. Including.

  The time case represents the time when the movement or state change occurred. For example, the time case includes a date and time (time) when an operation or a new state occurs, an elapsed time (PassTime) until the operation or state change ends, and the like.

  The source case represents the starting point of movement or state change. For example, the source case includes location information (PosCase) indicating the position of the starting point where the leading character moves, moves, etc., attribute information (Feature) indicating the attribute of the leading character at the starting point, and time information (TimeCase) at that time .

  The target case represents an end point such as movement or state change. For example, the target case includes location information (PosCase) indicating the position of the end point where the main character moves or moves, attribute information (Feature) indicating the attribute of the main character at the end point, and time information (TimeCase) at that time .

  A target case represents an object that is the target of an action or state. The target case includes a tracking number (ID), which is unique identification information, and an authentication name (Name).

  The tool scale represents an auxiliary means of movement. The instrument includes a tracking number (ID), which is unique identification information, and an authentication name (Name).

  The state information constructing unit 113 described above sets the specific value in the case frame configured as described above, thereby analyzing the analysis result by the image processing unit 112 and the RFID tag information input to the data input unit 114 as an object. It is constructed as state information representing the operation state of The available deep cases are not limited to those described above, and may be further subdivided. For example, any other deep case such as an experienced person representing an experienced person of a mental event can be used.

  Returning to FIG. 1, the behavior derivation rule DB 122 stores a behavior derivation rule that is referred to when the abnormality determination unit 115 described later compares the case frames stored in the state information DB 111 to detect an operation state. This action derivation rule compares the case frames stored in the state information DB 121, and if there is a change between the previous and next case frames, a rule for generating and reconstructing a case frame representing the change is stored. It is. FIG. 5 shows an example of the data structure of the action detection rule stored in the action derivation rule DB 122.

  As shown in FIG. 5, the action derivation rule is information that is a comparison source, and is a previous frame that is state information that is temporally previous, and information that is a comparison destination that is temporally subsequent. This is a format in which the post-grade frame that is information and other conditions are associated with the derived state information.

  Each of the front case frame and the back case frame includes an action name and case information. In the case information, at least one deep case to be compared is set.

  The front case frame and the back case frame correspond to information (difference information) representing a pattern in which two case frames have a difference. That is, when two action frames and the action information set in the case frame and the case frame match each other, it is determined that there is a difference between the two case frames. For example, in FIG. 5, the action name of a certain front case frame (case frame 1) is “exists”, the body position included in the place case is “A”, and the action name of the back case frame (case frame 2) is When the body position included in the place case is “B”, it is determined that there is a difference between the front case frame and the back case frame. When it is determined that there is a difference, an action having the action name described in the “state information that can be derived” column (“moving” in the above example) is detected as an action corresponding to the difference.

  In other conditions, conditions to be satisfied by the state information to be compared are set as necessary. For example, in the behavior derivation rule for deriving the behavior with the behavior name “Squam” in the same figure, the relationship B> C between the head position B included in the front case frame and the head position C included in the back case frame. Satisfying is set as a condition.

  The state information that can be derived includes the action name of the derived action and case information that represents at least one deep case to be saved. In the figure, for example, when an action with the action name “move” is detected, the body position A included in the front case frame and the body position B included in the back case frame are respectively corresponding to the derived actions. An example of saving as the body position A of the source case of the frame and the body position B of the target case is shown.

  Note that the behavior derivation rules are not limited to the format shown in the figure, and rules of any format can be applied as long as predetermined behaviors can be determined according to differences between a plurality of state information. Can do.

  FIG. 6 shows an example of the data structure of the article registration DB 123. As shown in the figure, the article registration DB 123 includes an ID of an article (particularly an important object) to be registered and managed (this ID corresponds to an ID read from an RFID tag attached to the article), an article name (safe Key, customer list, notebook PC, projector, etc.), access restriction, type, and storage position are associated with each other and stored as article registration information. The “access restriction” is a use condition set for each article. A plurality of access restrictions can be set for each article, and when a state deviating from each condition is detected by the abnormality determination unit 115, it is determined to be abnormal. As this access condition, for example, a value of 0 to 99 is assigned as a user condition, a value of 100 to 199 is assigned as a location condition, a value of 200 to 299 is assigned as a time condition, and a value of 300 to 399 is assigned as an application condition. The “type” indicates the property of each article. For example, 0 is a key, 1 is a CD / DVD, 2 is a book / document, 3 is a PC device, and 4 is a tool. The properties (types) of the articles are indicated by numerical values. In the example of FIG. 6, in the case of a safe key with ID = 0, three usage conditions of user conditions, access conditions of 0, 100, and 200, a location condition, and a time condition are set, and the type of this item is “0”. That is, it is registered that the storage location is a key storage box with a key.

  A specific example of access restriction is shown in FIG. As described above, the values of 100 to 199 are assigned as place conditions, the values of 200 to 299 are assigned as time conditions, and the values of 300 to 399 are assigned as application conditions. Values are assigned such that 0 is set to “only the user”, 1 is set to “requires the presence of a plurality of people at the time of use”, and 2 is set to “only the person who is permitted to use”. As the place condition, the value is set to 100 when “move between areas is prohibited”, 101 when “check is necessary when moving between areas”, 102 when “pickout out of floor” is set, etc. Assigned. Also, as the time condition, 200 is assigned when “within 30 minutes”, 201 is assigned with “within 1 day”, 202 is assigned with “within 1 week”, and the usage conditions are assigned. Is assigned a value such as “300 when copying is prohibited” and 301 when “requires the presence of multiple people during copying”.

FIG. 8 shows an example of the data structure of the environment information DB 124. As shown in the figure, the environment information DB 124 includes IDs, names, start point coordinates, end point coordinates of installation objects such as RFID readers (RFID readers 400 _{1 to} 400 _n ) provided in each monitoring area and each monitoring area, And area / installation type are registered. The range of the monitoring area is defined by the starting point coordinates and the ending point coordinates of the monitoring area. The starting point coordinates and the ending point coordinates of the installation object such as the RFID reader in each monitoring area indicate the coordinates of the installation position of the installation object.

  FIG. 9 shows an example of the data structure of the person DB 125. As shown in the figure, the person DB 125 is obtained by associating and registering a person and information related to the authority for the person's access to the important object. Specifically, a person's ID, name, management authority (administrator / user / guest, etc.), department (general affairs department / development department / operation department, etc.), and use permission deadline are registered for each person. For example, information such as entry / exit management can be used as the information related to the authority of a person to access important items.

  FIG. 10 shows an example of the data structure of the article management DB 126. The article management DB 126 is a DB for managing the current state of the article. As shown in the figure, the article ID, name, state (storage / take-out, etc.), final handler, latest update position, latest update Management information including time is registered for each article. Each DB in the storage unit 120 is updated regularly, as necessary, or in response to a predetermined event. For example, the state information DB 121 is preferably updated when the movement of the object is detected by the image processing unit 112 in addition to the periodic update when the time interval of the regular update is long. .

  The abnormality determination unit 115 acquires a plurality of state information from the state information DB 121. For example, the abnormality determination unit 115 acquires two pieces of state information having consecutive times among the state information generated by the state information construction unit 113 and stored in the state information DB 111. Moreover, the abnormality determination part 115 acquires two state information from which the main character differs at the same time among the state information preserve | saved in state information DB121. The acquired state information is used when an action is detected by the action derivation rule of FIG. The times need not be exactly the same, and may be configured to be determined to be substantially the same if the error is within a predetermined time range.

  In addition, the abnormality determination unit 115 detects a new action by determining whether or not two pieces of state information with successive times are compatible with the action derivation rule stored in the action derivation rule DB 122.

  Specifically, the abnormality determination unit 115 acquires from the behavior derivation rule DB 122 an action derivation rule in which the action name and the case information of the two pieces of state information having the previously acquired times match. Then, when an action derivation rule that matches the action name and the case information of the two state information can be acquired, the action determined by the action detection rule is detected as a new action. And the abnormality determination process mentioned later is performed.

(Abnormality judgment)
The abnormality determination unit 115 further performs an abnormality determination from the state information for the person registered in time series in the state information DB 121 and the state information for the article. A specific example is shown in FIG. FIG. 11 is a diagram illustrating a specific example when abnormality determination is performed from state information. In the figure, it is assumed that an RFID tag is attached to the article. FIG. 12 shows an example of information obtained by image processing and information obtained from the RFID reader in the situation shown in FIG.

  As shown in FIG. 11, a camera and an RFID reader are installed in each area, and in each area, a person's behavior is analyzed and movement between areas of the article is detected. Information shown in FIG. 12 is obtained as information obtained in the states (1), (2), and (3) of FIG. That is, the information obtained by the image processing by the image processing unit 112 in (1) is [behavior name: exists], [main character: person A, person B], [location case (location name): area A]. Similarly, the information obtained from the RFID reader in (1) is [behavior name: exists], [main character: article], and [location character: area A]. Also, information obtained by image processing by the image processing unit 112 in (2) of FIG. 11 is [action name: exists], [main character: person A, person B], [location character: area C], Similarly, the information obtained from the RFID reader in (2) is [action name: exists], [master: article], [location case: area C]. Also, the information obtained by the image processing by the image processing unit 112 in (3) of FIG. 11 is [action name: exists], [main character: person A], [location character: area B]. ), The information obtained from the RFID reader is [action name: exists], [master: article], and [location case: area B]. In this example, the detection of the operation state of the article is based on information obtained from the RFID reader. Of course, information obtained by analyzing image information can also be used. By using the information obtained from the RFID reader as in this example, it is possible to obtain information for detecting the operation state even when the article cannot be captured by the camera.

  In this situation, when “100” is set as the access restriction shown in FIG. 7, that is, when the condition “movement between areas is prohibited” is set, (2) in FIG. When information is acquired from the RFID reader for updating the state information DB 121, “movement from area A to area B” of the article is registered in the state information DB 121 as time-series state information (generated at this time). The case frame has a different location case from the previously registered case frame (state information) (area A → area B)), and can be determined to be abnormal based on this state information.

  Further, in this situation, when “1” is set as the access restriction shown in FIG. 7, that is, when the condition “requires the presence of multiple people when used” is set, In the situations (1) and (2), the person A and the person B exist in the area where the object is present, so that the access restriction condition is not met and no abnormality occurs. However, as in the situation (3), the object When it is detected from the state information that only the person A exists, the abnormality determination unit 115 determines that there is an abnormality based on this state information.

(Abnormality judgment processing)
Here, the abnormality determination process by the abnormality determination unit 115 of the security device 100 according to the present embodiment will be described with reference to FIG. FIG. 13 is a flowchart showing the abnormality determination process by the abnormality determination unit 115. Note that an RFID tag is attached to the article.

  First, it is determined whether a case frame for the target article is registered in the state information DB 121 (step S1301). Here, the status information DB 121 is searched based on the ID (tracking number) which is the main character of the article, and it is determined whether or not the corresponding case frame is registered.

  When the case frame of the target article is registered in the state information DB 121 (Yes in step S1301), the process proceeds to step S1302, and in this step, information on the article is acquired from the article registration DB 123. Here, the article registration DB 123 is searched based on the ID of the article, and the corresponding information such as the article name, access restriction (user condition, place condition, time condition, usage condition, etc.), type, storage position, etc. is acquired. To do.

  Next, the status information of the person who is closest to the target article is acquired from the status information DB 121 (step S1303). Usually, a general article is moved by a person except for a self-propelled object. Therefore, when an article is moved, it is considered that a person exists in the closest space and time. Therefore, here, based on the place case and time case obtained from the case frame (state information) of the article registered in the state information DB 121, the state for the person who is closest to the article in space and time Get information.

  In step S1304, user conditions, place conditions, time conditions, and usage conditions are checked. Here, when any or all of the above-mentioned user conditions, place conditions, time conditions, and usage conditions are set as access restrictions in the information acquired in step S1302, the state information for the person acquired in step S1303. Checks whether these conditions are met. Regarding the user condition, the person DB 125 is referred to for the person specified from the state information for the person, and it is also checked whether the management authority, the use permission deadline, and the like match the user condition. In addition, regarding the location conditions and application conditions (for example, in the case of copy prohibition, the prohibition condition is that an article exists in a certain position / space of the copying machine) In addition, since the coordinates are specified, the environmental information DB 124 is referred to, the positional relationship where the person or the article exists is specified, and it is checked whether the state information for the person or the article matches the location condition or the like.

  If the status information for the person deviates from any of the conditions (Yes in step S1305), an abnormality output for notifying the management PC 200 of the abnormality is performed (step S1306).

  Through the above-described abnormality determination process for monitoring an article and a person in association with each other, security for the article (with an RFID tag) can be further strengthened.

  Next, processing for fixed asset management by the fixed asset management unit 116 will be described. The fixed asset management unit 116 acquires a plurality of pieces of state information from the state information DB 121. For example, two pieces of state information with continuous time are acquired. The operation state (movement) is detected on the basis of the main character (ID or the like) and the place character of the state information (in the case of an article, there is no information such as head coordinates for the person shown in FIG. 3). And the fixed asset management process mentioned later is performed. The times need not be exactly the same, and may be configured to be determined to be substantially the same if the error is within a predetermined time range.

(Fixed assets management)
Below, the method of the fixed asset using the monitoring system 10 of this embodiment is demonstrated. Specific examples thereof are shown in FIGS. Here, FIG. 14 is a diagram illustrating a specific example when processing for fixed asset management is performed from the state information. FIG. 15 is a diagram illustrating an example of information obtained by image processing and information obtained from the RFID reader in the situation of FIG. FIG. 16 is a diagram illustrating an update process of the article management DB. In FIG. 14, it is assumed that an RFID tag is attached to the article.

  As shown in FIG. 14, a camera and an RFID reader are installed in each area. In each area, a person's behavior is analyzed, and movement between the areas of the article is detected as an operation state of the article. Information shown in FIG. 15 is obtained as information obtained in the states (1), (2), and (3) of FIG. That is, the information obtained by the image processing by the image processing unit 112 in (1) is [behavior name: exists], [main character: person A], [location case (location name): area A], [time case: Similarly, the information obtained from the RFID reader in (1) is [Action Name: Exists], [Main Character: Article], [Location Case: Area A], [Time Case: T1]. Also, the information obtained by the image processing by the image processing unit 112 in (2) of FIG. 14 is [behavior name: exists], [main character: person A, person B], [location character: area C], [time] Similarly, the information obtained from the RFID reader in (2) is [Action Name: Exists], [Primary: Goods], [Location Case: Area C], [Time Case: T3]. . Also, the information obtained by the image processing by the image processing unit 112 in (3) of FIG. 14 is [behavior name: exists], [main character: person B], [location character: area B], [time character: T3. Similarly, the information obtained from the RFID reader in (3) is [action name: exists], [master: article], [location case: area B], [time case: T3].

  Here, it is assumed that the initial state of the article management DB 126 is in the state of FIG. In the state of (1) shown in FIG. 14, the person A and the article are in the area A, and the person A is about to take out the article. When the article management DB 126 is updated in this state, the article management DB 126 is updated as shown in FIG. That is, among the items in the article update DB, “state”, “last handler”, “latest update position”, and “last update time” are “take-out”, “person A”, “area A”, “T1”, respectively. (Where T1 is the time at the time of this update).

  Next, when the state shown in (2) of FIG. 14 is reached (at this time, the person A joins the person B in the area C), the article management DB 126 is updated as shown in (C) of FIG. That is, among the items in the article update DB, “latest update position” and “last update time” are updated to “area C” and “T2” (where T2 is the time at the time of this update), respectively. .

  Next, when the state shown in (3) of FIG. 14 is reached (at this time, the article is handed over from person A to person B, and the article and person B exist in area B), the article management DB 126 (see FIG. It is updated as in D). That is, among the items in the article update DB, “last handler”, “latest update position”, and “last update time” are “person B”, “area B”, and “T3” (however, T3 is this update). Time).

  As described above, the management state of the RFID-tagged article is registered and managed in the article management DB 126.

(Fixed asset management processing)
Here, the fixed asset management processing by the fixed asset management unit 116 of the security device 100 of the present embodiment will be described with reference to FIG. FIG. 17 is a flowchart showing fixed asset management processing by the fixed asset management unit 116. It is assumed that an RFID tag is attached to the article, the RFID tag information is read by the RFID reader, and the ID of the article is acquired. Moreover, this process is performed regularly, for example.

  First, it is determined whether a case frame for the target article is registered in the state information DB 121 (can be acquired from the state information DB 121) (step S1701). Here, the status information DB 121 is searched based on the ID (tracking number) which is the main case of the article, and whether the corresponding case frame (the latest case frame and the case frame immediately before) is registered. Judge whether or not. If there is no case frame corresponding to the state information DB 121 (No in step S1701), this process ends.

  On the other hand, when the case frame of the article is registered in the state information DB 121 (Yes in step S1701), the process proceeds to step S1702, and in this step, it is determined whether or not a place movement has occurred. This determination is based on obtaining the previous case frame (state information) that is temporally close to the latest case frame (state information) at that time, and whether there is a change in the location case of the article between them. This is done by determining whether or not. If it is determined in this step that the location has not moved (No in step S1702), the process proceeds to step S1709, and in this step, the “last update time” of the article management DB 126 is set to the time status of the latest state information of the article. Update to information.

  If it is determined in step S1702 that the location has been moved (Yes in step S1702), the process proceeds to step S1703, where the “last updated position” in the article management DB 126 is updated to the location information of the latest state information. To do.

  Next, information on the state item is acquired from the article management DB 126 (step S1704), and it is determined whether this information is “storage” or “take-out” (step S1705).

  If it is determined in step S1705 that the status item information is “storage” (Yes in step S1705), the status item is updated to “take out” (step S1706).

  Next, the status information of the person who is closest to the target article is acquired from the status information DB 121 (step S1707). Here, based on the location case and the time case obtained from the case frame (state information) of the article registered in the state information DB 121, the state information for the person who is closest to the article in space and time (Especially the main character).

  Then, the final handler item in the article management DB 126 is updated with the status information (main character) for the person acquired in step S1707 (step S1708).

  Finally, the “last update time” of the article management DB 126 is updated to information on the time case of the latest status information of the article (step S1709).

  On the other hand, if it is determined in step S1705 that the status item information is not “storage”, that is, “take-out” (No in step S1705), the process proceeds to step S1710. In this step, the storage position is stored from the article registration DB 123. Get information about. Then, it is determined whether or not the location of the state information of the target article matches the storage position acquired in step S1710 (step S1711).

  If it is determined in step S1711 that the location of the state information of the target article matches the storage position acquired in step S1710 (Yes in step S1711), the state item in the article management DB 126 is updated to “storage” ( Step S1712) and the process proceeds to Step S1707. On the other hand, if it is determined that the location of the state information of the target article does not match the storage position acquired in step S1710 (No in step S1711), the process proceeds to step S1707 without doing anything. The processing after step S1707 is as described above.

  Information in the article management DB 126 updated as described above can be accessed and referenced from the management PC 200 or another PC 200 '. Therefore, it is possible to easily know the state of the article now.

  It should be noted that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 10 Monitoring system 100 Security apparatus 110 Control part 111 Image input part 112 Image processing part 113 State information construction part 114 Data input part 115 Abnormality determination part 116 Fixed asset management part 117 Access control part 120 Storage part 121 State information DB
122 Action derivation rule DB
123 article registration DB
124 Environmental Information DB
125 person DB
126 Goods management DB
200 Management PC
200 'PC
300 ₁ to 300 _n imaging apparatus 400 ₁ to 400 _n RFID reader

Claims (9)

Information indicating the motion state of the object generated based on the captured image information and the read information read from the wireless tag, and is a verb indicating the type of motion of the object and a noun related to the motion of the object State information constructing means for constructing state information represented by a case frame including a case element and indicating a semantic relationship between the verb and the case element;
Storage means for storing status information constructed by the status information construction means;
State information acquisition means for acquiring a plurality of state information representing the operation states of the plurality of objects from the storage means;
Detecting means for detecting an operation state of an object corresponding to the combination based on a rule that predetermines an operation state of the object according to a combination of the verbs included in each of the acquired plurality of state information;
An abnormality determining means for determining whether or not there is an abnormality based on a detection result by the detecting means;
An operation state detection apparatus comprising: The state information includes a time when the movement of the object occurs as the case element,
The status acquisition means acquires a plurality of the status information having substantially the same time from the status storage means;
The operation state detection device according to claim 1.   2. The status information construction unit constructs status information for a person based on captured image information, and constructs status information for an article based on at least read information read from a wireless tag. The operation state detection apparatus according to claim 2.   The abnormality determination means acquires state information for a person who is closest in time and space to the article from the storage means when state information for the target article is stored in the storage means, The operation state detection device according to claim 3, wherein when the state information for the article and the person deviates from the use conditions defined for the article and the person, it is determined as abnormal. Information indicating the motion state of the object generated based on the captured image information and the read information read from the wireless tag, and is a verb indicating the type of motion of the object and a noun related to the motion of the object State information constructing means for constructing state information represented by a case frame including a case element and indicating a semantic relationship between the verb and the case element;
First storage means for storing state information constructed by the state information construction means;
Second storage means for storing management information including at least a storage state of the object, a final handler, and a position of the object;
State information acquisition means for acquiring a plurality of pieces of state information representing respective operation states of the plurality of objects from the first storage means;
When the state information for the object that is the target article can be acquired from the first storage unit, the presence / absence of movement of the place is detected from the state information for the object, and when the movement of the place is further detected, Fixed asset management means for updating the management information stored in the second storage means;
An operation state detection apparatus comprising: The state information includes a time when the movement of the object occurs as the case element,
The status acquisition means acquires a plurality of the status information having substantially the same time from the status storage means;
The operation state detection device according to claim 5.   When the movement of the location from the storage location of the object that is the article is detected, the fixed asset management means obtains the state information for the object that is the person closest in time and space to the object. The management information acquired from the storage means and updated in the second storage means is updated based on the respective state information for the article and the person. Operating state detection device. The state information construction means is information representing the motion state of the object generated based on the captured image information and the read information read from the wireless tag, and a verb representing the motion type of the object and the motion of the object State information that includes a case element that is a noun related to, and that constructs state information represented by a case frame indicating a semantic relationship between the verb and the case element, and stores the constructed state information in a storage means Construction steps and
State information acquisition means for acquiring a plurality of the state information representing the operation state of each of the plurality of objects from the storage means,
A detecting step in which the detecting means detects the motion state of the object corresponding to the combination based on a rule that predetermines the motion state of the object according to the combination of the verbs included in each of the acquired plurality of state information. When,
An abnormality determination step for determining whether the abnormality determination means is abnormal based on the detection result of the detection step;
An operation state detection method comprising: The state information construction means is information representing the motion state of the object generated based on the captured image information and the read information read from the wireless tag, and a verb representing the motion type of the object and the motion of the object A state element represented by a case frame that indicates a semantic relationship between the verb and the case element, and stores the built state information in the first storage means State information construction step to be
A state information acquisition unit that acquires a plurality of the state information representing the operation states of the plurality of objects from the first storage unit;
When the state information for the object that is the target article can be acquired from the first storage means, the fixed asset management means detects the presence / absence of movement of the place from the state information for the object, and further detects the movement of the place. A fixed asset management processing step for updating management information stored in second storage means for storing management information including at least a storage state, a final handler, and a position of the object when the object is stored;
An operation state detection method comprising:

Images