JP2007219948A - User abnormality detection equipment and user abnormality detection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide user abnormality detection equipment which appropriately detects abnormalities about a user by using user's position information and a photographed image which has photographed locations where the user passes through. <P>SOLUTION: This equipment includes; a photographed image reception section 11 which receives a photographed image; a position information reception section 13 which receives position information which indicates user's position; a movement track extraction section 15 which extracts person's movement track in a photographed image; a comparison section 16 which compares an extracted movement track and a user's movement track indicated by the position information; a user identification section 17 which identifies a user whose position is indicated by the position information from the person's image in the photographed image based on the comparison results; an abnormality detection section 18 which detects abnormalities of a user identified by the user identification section 17; and an abnormality output section 19 which outputs abnormality detection information on detection of abnormalities when the abnormal detection section 18 detects abnormalities. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a user abnormality detection device that detects an abnormality relating to a user.

  In recent years, surveillance systems that take pictures of street corners and perform surveillance using the pictures for crime prevention have been used. By using such a monitoring system, it is possible to detect a person who is performing suspicious behavior or a person who has fallen due to illness or accident. As a related prior art, for example, an apparatus has been developed that detects an abnormal behavior of a user who behaves differently from a behavior history stored in advance on the premise of a physical structure of a building (see Patent Document 1). ).

On the other hand, a technique for specifying the position of a user by using GPS (Global Positioning System) or the like has been developed, and in recent years, a mobile phone or the like incorporating such a technique has also been developed. By using such a device, it is possible to specify a position over a wide range of the user.
Japanese Patent Laying-Open No. 2005-141319 (first page, FIG. 1 etc.)

  However, in the conventional monitoring system, in order to specify the person in the image, the user must be specified using information unique to the user such as facial features, Only a user who has registered information unique to the user, such as features, could identify a person in the image. In addition, when the shooting angle or size of a person's face or the like shown in the image is not appropriate, the person may not be properly specified. Further, when the number of users who register information unique to the user increases, there is a problem of search speed that the processing load for specifying the user increases.

  Further, in the conventional method of specifying the position of the user using GPS or the like, since the error is large, the occurrence of abnormality cannot be detected strictly. For example, even if it can be detected that the user stays in the same place for a long time depending on the position of the user specified using GPS or the like, it is hit by a car and falls in the middle of the road, or It was not possible to judge whether he / she stayed at the same place for a long time by waiting for traffic lights or talking. Furthermore, there is also a problem that only a user holding a device having a function such as GPS can track the behavior. In addition, for example, even when a certain user is being tracked by a stalker, this cannot be detected.

  The present invention has been made to solve the above-described problems, and appropriately detects an abnormality related to the user by using position information indicating the position of the user and a captured image obtained by photographing a place where the user passes by the camera. An object of the present invention is to provide a user abnormality detection device and the like that can be used.

  In order to achieve the above object, a user abnormality detection device according to the present invention receives a captured image reception unit that receives a captured image captured by one or more cameras at a location where a user passes, and position information that is information indicating the position of the user. Comparison comparing a position information receiving unit, an action locus extracting unit that extracts a person's action locus in the captured image, an action locus extracted by the action locus extracting unit, and a user action locus indicated by the position information A user specifying unit that specifies a user whose position is indicated by position information received by the position information receiving unit from a person image in the captured image based on a comparison result by the comparing unit, and the user specifying unit An abnormality detection unit that detects an abnormality related to the user identified by the user, and an abnormality when the abnormality detection unit detects an abnormality related to the user. An abnormality output unit for outputting an abnormality detection information which is information related to the detected, those having a.

  With such a configuration, an abnormality relating to the user can be detected using a captured image captured by the camera and position information indicating the position of the user even if the user's facial features or the like are not registered in advance. it can. Further, even if the user's face or the like is not properly captured in the photographed image, an abnormality relating to the user can be detected, and further, a problem relating to search speed does not occur. Further, when an abnormality relating to the user is detected, the abnormality detection information is output, so that the abnormality that has occurred can be appropriately dealt with.

Moreover, in the user abnormality detection apparatus by this invention, the said abnormality detection part may detect abnormality regarding the user's action which the said user specific part specified.
With such a configuration, an abnormality relating to the user's behavior can be detected.

  In the user abnormality detection device according to the present invention, the abnormality detection unit includes user past information storage means for storing user past information, which is information indicating a user's past behavior specified by the user specifying unit, and the user When the user's action specified by the specifying unit is different from the user's past action indicated by the user past information, a detecting unit that detects an abnormality related to the user's action may be provided.

  With such a configuration, when the user's behavior is different from the user's past behavior, the user's abnormality can be detected. Therefore, for example, when the user takes an action different from the past action due to an incident, accident, illness, or the like, this can be detected as an abnormality.

  In the user abnormality detection device according to the present invention, the abnormality detection unit includes normal behavior information storage means for storing normal behavior information, which is information related to a normal human behavior, and the user behavior specified by the user identification unit. However, in the case where the behavior of the user is different from the behavior of the normal person indicated by the normal behavior information, a detecting unit that detects an abnormality related to the behavior of the user may be provided.

  With such a configuration, when the user's behavior is different from normal human behavior, the user's abnormality can be detected. Therefore, for example, when a user takes an action different from that of a normal person due to an incident, accident, illness, or the like, this can be detected as an abnormality.

  Moreover, in the user abnormality detection device according to the present invention, the abnormality detection unit is configured such that the behavior of a person other than the user specified by the user identification unit included in the captured image is abnormal in relation to the user's behavior. In addition, an abnormality relating to the user may be detected.

  With such a configuration, when an action of a person other than the user is abnormal in relation to the action of the user, the abnormality can be detected. For example, an abnormality that the user is being tracked by a stalker can be detected.

Moreover, in the user abnormality detection device according to the present invention, the abnormality detection unit is an action that compares the action locus of one or more persons extracted by the action locus extraction unit with the action locus of the user specified by the user specifying unit. The trajectory comparing means and the behavior trajectory comparing means may include a detecting means for detecting an abnormality relating to the user when both have a certain similarity or more.
With such a configuration, it is possible to detect that the user is following.

Further, in the user abnormality detection device according to the present invention, the abnormality detection unit includes a predicted behavior trajectory calculation unit that calculates a predicted behavior trajectory that is a predicted behavior trajectory of the user identified by the user identification unit, and the predicted behavior trajectory. An abnormality related to the user is detected when a person other than the user included in the photographed image is stopped at a position within a certain range from the predicted action trajectory calculated by the calculation unit and where the normal person does not stop. And a detecting means for performing.
With such a configuration, it is possible to detect that the user is ambushed.

  In the user abnormality detection device according to the present invention, the captured image received by the captured image reception unit is a captured image of two or more regions that are captured by two or more cameras, and the abnormality detection unit includes: You may detect abnormality using the picked-up image of two or more area | regions.

  With such a configuration, even when the user does not exist in the imaging region of two or more cameras, an abnormality relating to the user can be detected. For example, when the user does not reach the other captured image area even after a certain period of time has elapsed after leaving the certain captured image area, an abnormality related to the user may be detected.

  According to the user abnormality detection device and the like according to the present invention, it is possible to specify a user who is shown in a captured image by using position information indicating the position of the user and a captured image obtained by capturing a location where the user passes by the camera. . As a result, an abnormality relating to the user can be detected appropriately.

  Hereinafter, a user abnormality detection apparatus according to the present invention will be described using embodiments. In the following embodiments, components and steps denoted by the same reference numerals are the same or equivalent, and repetitive description may be omitted.

(Embodiment 1)
A user abnormality detection device 1 according to Embodiment 1 of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a user abnormality detection device 1 according to the present embodiment. In FIG. 1, the user abnormality detection device 1 according to the present embodiment includes a captured image reception unit 11, a captured image storage unit 12, a position information reception unit 13, a position information storage unit 14, and an action locus extraction unit 15. The comparison unit 16, the user identification unit 17, the abnormality detection unit 18, and the abnormality output unit 19 are provided.

  The captured image reception unit 11 receives a captured image. This photographed image is an image obtained by photographing a place where the user passes with one or more cameras. Here, the user is a user who uses the user abnormality detection device 1, that is, a user whose position information is received by the position information receiving unit 13 described later. For convenience of explanation, FIG. 1 shows a case where the captured image reception unit 11 receives a captured image from one camera. However, the captured image reception unit 11 is a captured image captured by two or more cameras. There may be. When the captured image reception unit 11 receives captured images captured by two or more cameras, the two or more captured images are captured images of two or more regions that do not overlap as shown in FIG. May be. For example, one camera takes a picture of the vicinity of an intersection, and another camera takes a picture of the vicinity of the next intersection, so that the areas of the two photographed images taken by both cameras do not overlap. May be. This photographed image is a moving image. The frame rate of this moving image is arbitrary as long as the trajectory of the person's action shown in the captured image can be extracted. For example, it may be 30 fps (frame per second), or 1 fps or 2 fps. The data format of the captured image is not limited. For example, the data format of the photographed image may be uncompressed, may be MPEG, DV, etc., and the file format of the photographed image may be the RAW format or the ASF format. It may be in AVI format or the like. For example, the captured image reception unit 11 may receive a captured image transmitted via a wired or wireless communication line, may receive a captured image directly input from the camera, or may be a predetermined recording medium (for example, A photographed image read from an optical disk, a magnetic disk, a semiconductor memory, or the like may be received. Note that the captured image reception unit 11 may or may not include a device (for example, a modem or a network card) for reception. The captured image reception unit 11 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

  The captured image storage unit 12 stores the captured image received by the captured image reception unit 11 in a predetermined recording medium. Here, it is assumed that captured images are stored in the predetermined recording medium so that the shooting date and time of each frame of the captured image can be known. Therefore, information indicating the shooting date and time may be associated with each frame of the captured image, or information indicating the shooting date and time of the first frame is associated with the captured date and time of each frame using the information. May be calculated. Further, the information indicating the shooting date and time may be set by the camera or may be set by the shot image storage unit 12. In the former case, the camera may hold a clock or the like, and in the latter case, the captured image storage unit 12 may hold a clock or the like. Instead of the information indicating the shooting date and time, information indicating only the shooting time or information indicating a relative time of shooting (for example, a time code) may be used. The recording medium on which the captured image is stored is, for example, a semiconductor memory, an optical disk, a magnetic disk, or the like, and may be included in the captured image storage unit 12 or may exist outside the captured image storage unit 12. Also good. In the present embodiment, the former is assumed. The recording medium may or may not temporarily store the captured image.

  The position information receiving unit 13 receives position information that is information indicating the position of the user. In the present embodiment, it is assumed that the position information received by the position information receiving unit 13 is acquired using the GPS function of the information terminal held by the user and transmitted via the network. In the present embodiment, it is assumed that the position information is information in time series. Here, the position information being information in time series means that it is possible to know from the position information how the user's position has changed over time. Therefore, the position information may be information in which information indicating the date and time is associated with information indicating the position of the user, or information indicating the position of the user is arranged in time series. It may be information. In the latter case, for example, in the position information, information indicating the position of the user is arranged in time series at regular time intervals, and the date and time corresponding to the information indicating the position of the first user is included in the position information. By being included, the date and time corresponding to the information indicating the user's position may be calculated. Further, when the position information receiving unit 13 receives position information including only information indicating the position of the user, information indicating the date and time (for example, the date and time when the information is received) in the information indicating the position of the user. By associating, information indicating the date and time and information indicating the position may be associated with each other. In this case, the position information receiving unit 13 or the like may hold a clock or the like that can acquire information indicating the date and time. Instead of the information indicating the date and time, information indicating only the time or information indicating a relative time (for example, a time code) may be used. In the present embodiment, it is assumed that the position information receiving unit 13 receives position information in which information indicating the date and time is associated with information indicating the position of the user. Further, the information indicating the user's position may be any information as long as the information can identify the user's position. For example, the information indicating the position of the user may be information indicating longitude and latitude, or information indicating the position on a two-dimensional plane from a certain reference point (for example, the value of x in the xy two-dimensional orthogonal coordinate system) , Y value, etc.). Further, the information indicating the position of the user included in the position information may be converted during the process. For example, conversion may be performed between an orthogonal coordinate system and a spherical coordinate system. The position information receiving unit 13 may receive position information related to one user, or may receive two or more position information related to two or more users. In the latter case, position information associated with a user ID or the like for identifying the user may be received so that the position information can be distinguished for each user. The position information receiving unit 13 may receive information transmitted using a GPS function of an information terminal held by a user and transmitted via a wired or wireless communication line, for example. Information read from (for example, an optical disk, a magnetic disk, a semiconductor memory, etc.) may be received. In the present embodiment, the former is assumed. Note that the position information receiving unit 13 may or may not include a device (for example, a modem or a network card) for receiving. In addition, the position information receiving unit 13 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

  The position information storage unit 14 stores the position information received by the position information receiving unit 13 in a predetermined recording medium. The recording medium is, for example, a semiconductor memory, an optical disk, a magnetic disk, or the like, and may be included in the position information storage unit 14 or may exist outside the position information storage unit 14. In the present embodiment, the former is assumed. Further, this recording medium may or may not store the position information temporarily. The recording medium may or may not be the same as the recording medium in which the captured image storage unit 12 stores captured images.

  Note that it is preferable that the information indicating the shooting date and time of each frame of the captured image and the information indicating the date and time in the position information are synchronized for comparison processing described later. Both are synchronized, for example, information indicating the shooting date and time of a frame of a captured image shot when a certain user is at a certain position, information indicating the date and time corresponding to the position of the user in the position information, etc. Are equal. For example, information indicating the shooting date and time of each frame of the captured image may be acquired using a function similar to the GPS function of the information terminal held by the user so that the two are synchronized.

  The behavior trajectory extraction unit 15 extracts a human behavior trajectory in the captured image received by the captured image reception unit 11. In other words, the action locus extraction unit 15 extracts information indicating a change along the time series of the position of the person shown in the captured image. The action trajectory may be information indicating the action trajectory. The extraction of the action trajectory may be performed, for example, by detecting a person's area in the photographed image and then detecting a change in the position of the person's area, or extracting a speed field of the photographed image, You may carry out by detecting the change of the position of the moving object reflected in the picked-up image. As a method for detecting a human region in the former case, for example, a method using skin color detection, a method for extracting a difference from a background image, a method using pattern matching, and the like are known. Further, the change in the position of the person area can be detected, for example, by tracking a change in a specific position of the detected person area (for example, the position of the nose, the position of the top of the head, the position of the waist). it can. Further, in the latter case, for example, an optical flow technique can be used in the extraction of the action trajectory. When detecting a change in the position of a moving object using an optical flow technique or the like, an action locus of an animal body (for example, a car) other than the person shown in the photographed image is also extracted as a person's action locus. The problem will be described later. Usually, the camera does not shoot from directly above, but shoots from diagonally above. Therefore, even if an action trajectory in a captured image is extracted, the action trajectory is different from an action trajectory on a two-dimensional plane (for example, an actual map) viewed from directly above. In such a case, processing for converting an image viewed obliquely from above into an image viewed from directly above may be performed. Such processing includes, for example, planar projective transformation. The conversion process may be performed before extracting the action trajectory, and the action trajectory may be extracted from the photographed image after the conversion. Alternatively, the action trajectory may be extracted from the photographed image before conversion, and the extracted action trajectory may be extracted. Then, the conversion process may be performed. In general, the latter is lighter in processing load. Needless to say, the method of extracting a person's action trajectory from a captured image is not limited to the above description. Also, a method for extracting a person's action trajectory from a captured image is conventionally known, and detailed description thereof is omitted.

  The comparison unit 16 compares the behavior trajectory extracted by the behavior trajectory extraction unit 15 with the user's behavior trajectory indicated by the position information received by the position information reception unit 13. This comparison may be, for example, a comparison of information on positions that change depending on time indicated by both, a comparison of speed vectors that change depending on time indicated by both, or It may be a comparison between the two. It is assumed that the comparison unit 16 calculates the degree of similarity, the probability of matching between the two, by comparing the action locus extracted from the captured image with the action locus indicated by the position information. For example, the comparison unit 16 may perform the comparison using an approximation method such as a sequential Monte Carlo method, may use a model such as Bayesian estimation, or may use other methods. A method of comparing the action trajectory with the action trajectory indicated by the position information is conventionally known, and detailed description thereof is omitted.

  The user specifying unit 17 specifies the user whose position is indicated by the position information received by the position information receiving unit 13 from the image of the person in the captured image based on the comparison result by the comparing unit 16. That is, the user specifying unit 17 performs a process of associating the person shown in the captured image with the user whose position is indicated by the position information on a one-to-one basis. In addition, since a person other than the user is generally captured in the photographed image, the person depicted in the photographed image may not be associated with the user whose position is indicated by the position information on a one-to-one basis. The identification of the user is, for example, a process of associating an identifier for identifying a person area in a captured image with a user ID for identifying the user in association with a predetermined recording medium. For example, the user specifying unit 17 specifies position information indicating an action trajectory that matches the action trajectory of the person in the captured image, and specifies the person in the captured image as a user corresponding to the specified position information. Also good. Here, the action trajectory of the person in the captured image matches the action trajectory indicated by the position information, for example, not only when the two are exactly the same, but also when the degree of similarity between them and the probability that they match are predetermined. If it is above the threshold or if the similarity between them is the highest compared to the similarity with others, the similarity between them is above the threshold and the similarity with others A case where the degree of similarity between the two is the highest as compared with the degree or the like may be included.

  Note that the method for specifying the user in the captured image is not particularly limited as long as it is possible to know the correspondence between the image of the person included in the captured image and the user as a result. For example, the user specifying unit 17 may be performed for each still image in the captured image, or after the user is specified in one still image, the movement of the image of the person corresponding to the user is tracked. By doing so, the correspondence between the image of the person included in the captured image and the user may be known. In the latter case, the comparison unit 16 starts comparing the action trajectory of the person in the photographed image with the action trajectory indicated by the position information at a timing when a new person image is included in the photographed image. The user may be specified according to the result.

  Further, even if the behavior trajectory extraction unit 15 extracts the behavior trajectory of the moving object other than the person using the optical flow technique or the like, it is determined that the behavior trajectory does not match when the behavior trajectory is compared with the position information of the user. Therefore, it will not be a problem.

  In addition, since the position of the user is indicated by the position information, the comparison unit 16 considers an action trajectory extracted from a certain captured image and an error of the position information, and the user's position included in the captured area of the captured image You may make it compare with the action locus | trajectory which position information shows. By doing in this way, compared with the case where the action locus | trajectory extracted from the picked-up image and the action locus | trajectory which all the user's positional information shows is compared, the processing load of the comparison by the comparison part 16 can be reduced. .

  The abnormality detection unit 18 detects an abnormality related to the user specified by the user specifying unit 17. The abnormality detection unit 18 may detect (A) an abnormality related to the user's behavior specified by the user specifying unit 17, and (B) a behavior of a person other than the user specified by the user specifying unit 17 included in the captured image may be detected. When an abnormality is associated with the user's behavior, an abnormality relating to the user may be detected, or an abnormality may be detected in other cases. The anomaly detection unit 18 may detect an anomaly based on the user's current behavior trajectory and information on past behaviors of the user or other persons, or may be preset with the user's current behavior trajectory. An abnormality may be detected based on a predetermined rule. As the user's current action trajectory, the action trajectory extracted by the action trajectory extraction unit 15 may be used, which is calculated from the action trajectory extracted by the behavior trajectory extraction unit 15 and the position information received by the position information reception unit 13. The action trajectory may be used. In the latter case, the abnormality can be detected even when the user is outside the range of the captured image. Here, with reference to FIG. 3, the configuration of the abnormality detection unit 18 and the processing in the cases (A) and (B) for detecting an abnormality relating to the user will be described.

  FIG. 3 is a block diagram illustrating a configuration of the abnormality detection unit 18. In FIG. 3, the abnormality detection unit 18 includes user past information storage means 21, normal action information storage means 22, action locus comparison means 23, predicted action locus calculation means 24, and detection means 25.

(A) Detection of abnormality related to user's behavior (A-1) Case where user's behavior is different from normal Here, processing for detecting an abnormality when the current behavior of the user is different from the normal behavior of the user explain.

  The user past information storage unit 21 stores user past information, which is information indicating the past actions of the user specified by the user specifying unit 17. The user past information may be any information as long as it is information indicating the user's past behavior. The user past information may be, for example, the user's action trajectory extracted from the captured image, may be information created using the action trajectory, or indicates a rule regarding the user's past action. It can be information. The user past information created using the action trajectory may be, for example, an average of the user's action trajectory. One captured image and other information may be obtained using the user's action trajectory and position information. It may be an action trajectory predicting an action between the captured image and other information. The rule regarding the past behavior of the user is, for example, a rule of moving a route from a certain point (for example, the user's house) to another point (for example, a train station used by the user for commuting) in the morning. In the evening, the rule is to move the route in the reverse direction. This rule may be calculated from the user's past action trajectory or the like, or may be set manually by the user or the like. Moreover, the user past information may be set for each season, month, day of the week, and time zone, for example. The user past information storage means 21 is realized by a predetermined recording medium (for example, a semiconductor memory, a magnetic disk, an optical disk, etc.). The process in which the user past information is stored in the user past information storage unit 21 is not limited. For example, the user past information may be stored in the user past information storage means 21 via a recording medium, and the user past information transmitted via a communication line or the like is stored in the user past information storage means 21. The user past information input via the input device may be stored in the user past information storage unit 21 or may be extracted by the action trajectory extraction unit 15. The action trajectory may be stored in the user past information storage unit 21.

  When the user behavior specified by the user specifying unit 17 is different from the user's past behavior indicated by the user past information, the detection unit 25 detects an abnormality related to the user's behavior. Specifically, the detection unit 25 compares the user's past information with the current action trajectory, and when the similarity between the two and the probability that both match are lower than a predetermined threshold, the user's action You may detect the abnormality regarding. As this comparison method, for example, pattern matching or the like may be used. Further, when the user past information is set for each season or day of the week, the comparison may be performed using a time series analysis method. As described above, when the user takes an action different from usual, an abnormality is detected. In this way, for example, if the user usually falls without walking from the house to the station, but the user falls down due to illness or accident on the way from the house to the station, It will stop for a long time, and an abnormality will be detected. In addition, for example, if a child who normally goes to school passes away from school because he is invited by a suspicious person, it will be different from normal behavior and abnormalities will be detected. .

(A-2) When a user's action differs from a normal person Here, the process in which abnormality is detected when a user's present action differs from a normal person's action is demonstrated.

  The normal behavior information storage unit 22 stores normal behavior information that is information related to normal human behavior. The normal behavior information may be any information as long as it is information that can know the behavior of a normal person as a result. For example, the normal behavior information may indicate a behavior pattern that a normal person performs, or by indicating a behavior pattern that a normal person does not perform, It may be information that can be known to be normal human behavior. The normal behavior information may be, for example, behavior trajectories of a large number of unspecified people in the past, or may be information indicating rules regarding the behavior of normal people. For example, a rule regarding a normal person's behavior may be a rule that a normal person does not walk staggered like a drunk person, and a normal person stops for a long time (for example, 1 minute or more). It may be a rule indicating a possible place (for example, a place waiting for a signal or a bus stop), or a rule indicating a normal walking time from one captured image to another captured image. This rule may be calculated from the user's past action trajectory or the like, or may be set manually by the user or the like. Moreover, the user past information may be set for each season, month, day of the week, and time zone, for example. The normal behavior information storage unit 22 is realized by a predetermined recording medium (for example, a semiconductor memory, a magnetic disk, an optical disk, etc.). The process in which normal action information is memorize | stored in the normal action information storage means 22 is not ask | required. For example, the normal action information may be stored in the normal action information storage unit 22 via the recording medium, and the normal action information transmitted via the communication line or the like is stored in the normal action information storage unit 22. The normal action information input via the input device may be stored in the normal action information storage unit 22 or may be extracted by the action trajectory extraction unit 15. The normal action information indicating the predetermined rule extracted from the action locus may be stored in the normal action information storage unit 22.

  When the user behavior specified by the user specification unit 17 is different from the normal human behavior indicated by the normal behavior information, the detection unit 25 detects an abnormality related to the user behavior. Specifically, when the normal behavior information is information indicating the behavior of a normal person, the detection unit 25 compares the normal behavior information with the current behavior trajectory of the user, and the degree of similarity between the two. If the probability that the two match is lower than a predetermined threshold, an abnormality relating to the user's behavior is detected. On the other hand, when the normal action information is information indicating an action that a normal person does not perform, the detection unit 25 compares the normal action information with the current action trajectory of the user, When the probability that both coincide is a predetermined threshold abnormality, an abnormality relating to the user's behavior is detected. As this comparison method, for example, pattern matching or the like may be used. Further, when the normal action information is set for each season or day of the week, the comparison may be performed using a time series analysis method. In this way, when the user takes an action different from that of a normal person, an abnormality is detected. By doing so, for example, when a user falls down due to illness or accident in a place where a normal person does not stop, the user will behave differently from a normal person, and an abnormality will be detected become. In addition, for example, when the user is moving from one image area toward another image area, the next walking time after the user no longer appears in the first image area If it does not appear in the image area, an abnormality is detected because a disease or accident may have occurred in the middle of the image area. Further, for example, when the user is walking staggered with staggered feet, an abnormality may be detected.

  In the above (A-1) and (A-2), the detection means 25 weights, for example, the probability that the user's behavior is different from the normal and the probability that the user's behavior is different from the normal person. The result may be compared with a threshold value, and an abnormality may be detected when the weighted probability is equal to or greater than the threshold value.

(B) Detection of abnormality related to behavior of person other than user (B-1) When user is tailed Here, a process of detecting an abnormality when the user is tailed by another person will be described.

  The behavior trajectory comparison unit 23 compares the behavior trajectory of one or more persons extracted by the behavior trajectory extraction unit 15 with the user's behavior trajectory specified by the user specifying unit 17. This comparison is performed, for example, in the same manner as the comparison in the comparison unit 16 described above. However, the action trajectory comparing means 23 performs a comparison for detecting whether the user is being followed by another person, so as to compare whether or not they are exactly the same as in the comparison unit 16. It does not have to be. For example, the behavior trajectory comparison means 23 matches the high similarity between the two even when the behavior trajectory of another person follows the user's behavior trajectory within a certain time interval (for example, about 10 seconds). A high probability of performing may be calculated. In addition, when the action locus of a person other than the user precedes the action locus of the user, that is, when another person is walking in front of the user, it can be said that the user is followed. Therefore, even if the behavior trajectories of both are approximated, the behavior trajectory comparison means 23 may calculate the similarity between the two low.

  The detecting means 25 detects an abnormality related to the user when the action trajectory comparing means 23 has a certain similarity or more. More specifically, when the user's action trajectory and another person's action trajectory are compared by the action trajectory comparing means 23 and have a certain degree of similarity or a certain probability of matching, the detecting means 25. Determines that both have a certain degree of similarity and detects an abnormality related to the user. By doing in this way, for example, when a user is followed by another person, an abnormality relating to the user can be detected.

  Note that the behavior trajectory comparison unit 23 may compare the user's behavior trajectory with the behavior trajectory of another person using two or more captured images. In this case, the detecting unit 25 multiplies the similarity between the user's action trajectory and another person's action trajectory in two or more captured images, thereby obtaining the user's action trajectory and the other person's action trajectory. The final similarity may be calculated. Further, the action trajectory comparison means 23 may determine whether or not other persons appearing in two or more photographed images are the same using the feature points of the faces of the persons, and perform the determination. It does not have to be.

(B-2) When the User is Ambushed Here, a process for detecting an abnormality when the user is ambushed by another person will be described.

  The predicted behavior trajectory calculation unit 24 calculates a predicted behavior trajectory that is a predicted behavior trajectory of the user specified by the user specifying unit 17. The predicted behavior trajectory calculating means 24 may calculate a predicted behavior trajectory using, for example, past behavior trajectories of the user or others, map information indicating a route through which the user passes, speed and direction in which the user is moving The user's destination may be estimated from the user's past action trajectory, etc., and the estimated destination and map information indicating the route taken by the user may be used. The predicted behavior trajectory may be calculated, or the user's predicted behavior trajectory may be calculated by other methods. As described above, a method for calculating a user's predicted behavior trajectory is conventionally known, and a detailed description thereof will be omitted.

  The detecting unit 25 is a position within a certain range from the predicted behavior trajectory calculated by the predicted behavior trajectory calculating unit 24, and a person other than the user included in the captured image is stopped at a position where a normal person does not stop. In addition, an abnormality relating to the user is detected. More specifically, the detection unit 25 identifies a region within a certain range from the predicted behavior trajectory calculated by the predicted behavior trajectory calculation unit 24 and that does not stop a normal person. Then, the detection unit 25 determines whether or not a person other than the user is stopped in the specified area. And when it has stopped, the detection means 25 detects abnormality, and when it has not stopped, the detection means 25 does not detect abnormality. Note that the area may be specified by specifying pixels around the area, or by specifying pixels included in the area, for example. For example, as described above, whether or not a normal person does not stop may be determined from the behavior trajectories of a large number of unspecified people in the past, or a normal person stops or does not stop. You may judge using the rule which shows a position. In this way, for example, when there is a person who is expected to pass by and a person who stops at a position where a normal person does not stop, an abnormality related to the user, that is, an ambush is detected. be able to.

  In addition, although the case where the abnormality detection part 18 detects each abnormality regarding the above-mentioned (A-1), (A-2), (B-1), (B-2) was demonstrated, Any one or more abnormalities may be detected, or other abnormalities may be detected. If any abnormality of (A-1), (A-2), (B-1), and (B-2) is not detected, even if the abnormality detection unit 18 does not include a corresponding component. Good. In the above description, the case where one detection means 25 is provided has been described. However, for each detection of abnormality relating to (A-1), (A-2), (B-1), and (B-2). You may make it provide a separate detection means.

  The abnormality output unit 19 outputs abnormality detection information when the abnormality detection unit 18 detects an abnormality related to the user. Here, the abnormality detection information is information regarding that an abnormality has been detected. The content of the abnormality detection information may be any information as long as the abnormality detection information is output so that the user or other people are informed that an abnormality has occurred. The abnormality detection information may be, for example, information indicating the occurrence of an abnormality, or may be information indicating a position where an abnormality has occurred. Here, the output may be, for example, display on a display device (for example, a CRT or a liquid crystal display), transmission via a communication line to a predetermined device, printing by a printer, or audio output by a speaker. Alternatively, it may be stored in a recording medium, or may be an output that controls lighting, blinking, or the like of a lamp indicating the occurrence of an abnormality. In the present embodiment, the abnormality output unit 19 transmits abnormality detection information. The transmission destination of the abnormality detection information may be a server such as a police or a security company, may be a user information terminal (for example, a mobile phone or a PDA), an information terminal of a user's family or friends, etc. It may be. If the address indicating the output destination of the abnormality detection information is necessary, the abnormality output unit 19 holds an address indicating the output destination of the abnormality detection information in advance, or outputs the abnormality detection information. Sometimes, the address or the like may be obtained from a predetermined server or the like. The abnormal output unit 19 may or may not include a device (for example, a display device or a printer) that performs output. Further, the abnormality output unit 19 may be realized by hardware, or may be realized by software such as a driver for driving these devices.

  Next, the operation of the user abnormality detection device 1 according to the present embodiment will be described using the flowcharts of FIGS. The flowchart of FIG. 4 is a flowchart showing the overall operation of the user abnormality detection device 1.

(Step S101) The captured image reception unit 11 determines whether a captured image has been received. If a captured image is received, the process proceeds to step S102. If not, the process proceeds to step S103.
(Step S102) The captured image storage unit 12 stores the captured image received by the captured image reception unit 11 in a predetermined recording medium. Then, the process returns to step S101.

  (Step S103) The position information receiving unit 13 determines whether position information has been received. If position information has been received, the process proceeds to step S104. If not, the process proceeds to step S105.

  (Step S104) The position information accumulating unit 14 accumulates the position information received by the position information receiving unit 13 in a predetermined recording medium. Then, the process returns to step S101. In addition, when the position information received by the position information receiving unit 13 does not include date and time information, the position information storage unit 14 may store position information including the date and time when the position information is received. . When position information associated with a user ID is received, the position information may be accumulated in association with the user ID.

  (Step S105) The user specifying unit 17 determines whether it is time to specify a user from a person included in a captured image. If it is the timing, the process proceeds to step S106, and if not, the process proceeds to step S107. Here, this timing may be every predetermined period (for example, every 1 second or every 5 seconds), and is triggered by a predetermined event (for example, reception of a new captured image). It may be a thing and other timing may be sufficient.

  (Step S106) The action trajectory extraction unit 15, the comparison unit 16, and the user identification unit 17 perform processing for identifying a user. Then, the process returns to step S101. Details of the processing in step S106 will be described later with reference to the flowchart of FIG.

  (Step S107) The abnormality detection unit 18 determines whether it is time to detect an abnormality. If it is the timing, the process proceeds to step S108; otherwise, the process returns to step S101. Here, this timing may be every predetermined period (for example, every 5 seconds), and is triggered by a predetermined event (for example, reception of a new captured image). Alternatively, other timing may be used.

  (Step S108) The abnormality detection unit 18 detects an abnormality relating to the user. Details of the processing in step S108 will be described later with reference to the flowchart of FIG.

  (Step S109) The abnormality output unit 19 determines whether an abnormality is detected. And when abnormality is detected, it progresses to step S110, and when that is not right, it returns to step S101.

(Step S110) The abnormality output unit 19 outputs abnormality detection information. Then, the process returns to step S101. Details of the processing in step S110 will be described later with reference to the flowchart of FIG.
In the flowchart of FIG. 4, the process ends when the power is turned off or the process ends.

  FIG. 5 is a flowchart showing details of the user specifying process in step S106 of the flowchart of FIG. In the flowchart of FIG. 5, a case will be described in which it is determined in step S105 that it is time to specify a user every time a captured image is received. In step S103, position information associated with the user ID is accepted.

(Step S201) The action trajectory extraction unit 15 reads the photographed image received last by the photographed image accepting unit 11 from the photographed image storage unit 12, and specifies the image area of the person in the photographed image. As described above, for example, there are a method using skin color detection, a method for extracting a difference from a background image, and a method using pattern matching. Here, it is assumed that the action trajectory extraction unit 15 assigns a person ID to the identified person image area. When a person ID has been assigned before, the same person ID is assigned to the image area of the same person. Whether or not a person's image area in two or more photographed images is the same person's image area may be determined, for example, by using information about a feature point of the person or a difference between the photographed images. Assigning a person ID to a person image area means, for example, associating information indicating a person image area with a person ID and storing the information in a predetermined recording medium.
(Step S202) The action trajectory extraction unit 15 sets the counter I to “1”.

  (Step S203) The action trajectory extraction unit 15 extracts the action trajectory of the I-th person specified in step S201. For example, the action trajectory extraction unit 15 extracts the position of the person's image area, sets the position of the person's image area to the person ID given in step S201, and the date and time corresponding to the captured image processed in step S201. The information is stored in a predetermined recording medium in association with the indicated date / time information.

(Step S204) The action trajectory extraction unit 15 increments the counter I by 1.
(Step S205) The action trajectory extraction unit 15 determines whether or not the I-th person exists in the captured image. If it exists, the process returns to step S203. If it does not exist, action trajectories have already been extracted for all persons in the captured image, and the process proceeds to step S206.

(Step S206) The user identifying unit 17 sets the counter J to “1”.
(Step S207) The comparing unit 16 compares the action trajectory indicated by the position information of the J-th user among the position information received by the position information receiving unit 13 with the action trajectory extracted in Step S203.

  (Step S208) The user specifying unit 17 determines whether or not the action trajectory indicated by the position information of the J-th user matches any action trajectory extracted in Step S203. If they match, the process proceeds to step S209, and if they do not match, the process proceeds to step S210.

(Step S209) The user specifying unit 17 specifies that the person in the captured image having the action locus that matches the action locus indicated by the position information of the Jth user is the Jth user. For example, the user specifying unit 17 may store a person ID that identifies the person's image area and a user ID that identifies the J-th user in a predetermined recording medium in association with each other.
(Step S210) The user identifying unit 17 increments the counter J by one.

  (Step S211) The user specifying unit 17 determines whether or not the J-th user exists. If the J-th user exists, the process returns to step S207. If the J-th user does not exist, the process related to user identification has already been performed for all users. Become.

  In the flowchart of FIG. 5, the case where the person image area is specified in step S <b> 201 has been described. However, as described above, when the technique such as the optical flow is used, the person image area is specified. It does not have to be done. However, when the optical flow technique is used, it is effective to limit the area in which the optical flow processing is executed. This is to reduce the processing load. For example, the area where the optical flow processing is performed may be limited to an area where a new person appears (for example, an edge of a captured image, an area where a person enters or exits), and an area where motion is detected in a previous frame You may limit to. In other areas, the motion detected in the area where the optical flow processing is performed may be tracked. This tracking method is conventionally known, and detailed description thereof is omitted. Further, although the flowchart of FIG. 5 shows processing when there are a plurality of users, the counter J may not be used when there is one user.

FIG. 6 is a flowchart showing details of the abnormality detection process in step S108 of the flowchart of FIG.
(Step S301) The detecting means 25 sets the counter K to “1”.

  (Step S <b> 302) The detection unit 25 calculates the probability that the behavior of the Kth user is different from the past behavior, using the user past information stored in the user past information storage unit 21.

  (Step S303) Using the normal action information stored in the normal action information storage unit 22, the detecting unit 25 calculates a probability that the K-th user's action is different from the normal person's action.

  In the processes of steps S302 and S303, as described above, the detection unit 25 may use the Kth user's action trajectory extracted by the action trajectory extraction unit 15 as the information indicating the Kth user's action. Alternatively, the action trajectory of the Kth user extracted by the action trajectory extraction unit 15 and the action trajectory of the Kth user indicated by the position information may be used.

  (Step S304) The detecting means 25 calculates the probability of weighting each probability calculated in steps S302 and S303. That is, the detection means 25 calculates a probability by multiplying each probability by a predetermined coefficient and adding the result.

  (Step S305) The detection means 25 determines whether or not the probability is equal to or higher than a predetermined threshold value. And when it is more than a threshold value, it progresses to step S306, and when that is not right, it progresses to step S307.

  (Step S306) The detection means 25 detects an abnormality related to the behavior of the Kth user. Note that the detection means 25 may store information indicating that an abnormality relating to the behavior of the Kth user has been detected in a predetermined recording medium or the like.

  (Step S307) The behavior trajectory comparison means 23 compares the behavior trajectory of the Kth user with the behavior trajectories of other persons. The behavior trajectory comparison unit 23 may use the Kth user's behavior trajectory extracted by the behavior trajectory extraction unit 15 as information indicating the behavior of the Kth user, and the Kth user's behavior trajectory extraction unit 15 extracts the Kth. The user's action trajectory and the K-th user's action trajectory indicated by the position information may be used. Further, the behavior trajectory comparison unit 23 may use the behavior trajectory of another person extracted by the behavior trajectory extraction unit 15 as the behavior trajectory of another person.

  (Step S308) The detection unit 25 determines whether there is an action locus of another person having a certain similarity or more with the action locus of the Kth user. This determination is performed as described in (B-1) above. If there is an action locus of another person having a certain similarity or more with the action locus of the Kth user, the process proceeds to step S309, and if not, the process proceeds to step S310.

(Step S309) The detecting means 25 detects an abnormality that the Kth user is being followed. Note that the detection unit 25 may store information indicating that the abnormality that the Kth user is being followed is detected in a predetermined recording medium or the like.
(Step S310) The predicted action trajectory calculating means 24 calculates the predicted action trajectory of the Kth user.

  (Step S311) The detection unit 25 determines whether there is an ambush on the predicted behavior trajectory of the Kth user calculated by the predicted behavior trajectory calculation unit 24. This determination is performed as described in (B-2) above. If there is an ambush on the predicted behavior trajectory of the Kth user, the process proceeds to step S312; otherwise, the process proceeds to step S313.

(Step S <b> 312) The detecting means 25 detects an abnormality that the Kth user is ambushed. Note that the detection unit 25 may store information indicating that the abnormality that the K-th user is awaited is detected in a predetermined recording medium or the like.
(Step S313) The detection means 25 increments the counter K by 1.

  (Step S314) The detection means 25 determines whether or not the Kth user exists. If it exists, the process returns to step S302, and if not, the series of processing ends.

  In the flowchart of FIG. 6, the case where the abnormality detection unit 18 detects each of the abnormality related to the user's behavior, the abnormality that the user is being followed, and the abnormality that the user is ambushed is described. 18 may be any one that detects at least one of the abnormalities. Further, the abnormality detection unit 18 detects only an abnormality when the user's behavior is different from the user's normal behavior or an abnormality when the user's behavior is different from the normal person as an abnormality related to the user's behavior. May be.

  Further, the detection means 25 reads out information indicating what kind of abnormality is detected for which user after detecting all the abnormalities from, for example, a predetermined recording medium and passes it to the abnormality output unit 19. May be.

FIG. 7 is a flowchart showing details of the abnormality information output processing in step S110 of the flowchart of FIG. In addition, the abnormality output part 19 shall transmit abnormality detection information to the information terminal and server which a user has.
(Step S401) The abnormality output unit 19 sets the counter L to “1”.

(Step S402) The abnormality output unit 19 determines whether an abnormality relating to behavior has been detected for the L-th user from which an abnormality has been detected. For example, the abnormality output unit 19 may make this determination on the basis of information received from the detection unit 25 and indicating what abnormality is detected for which user. If detected, the process proceeds to step S403, and if not, the process proceeds to step S406.
(Step S403) The abnormality output unit 19 transmits information for confirming the safety of the user to the information terminal of the Lth user.

(Step S404) The abnormality output unit 19 determines whether or not a receiving unit (not shown) has received information indicating that there is no abnormality from the information terminal of the Lth user in response to the transmission of information in Step S403. . If it is received, the process proceeds to step S406, and if there is no information indicating that the user has no abnormality even after a lapse of a certain time from the transmission in step S403, or an abnormality has occurred from the user's information terminal or the like. If the receiving unit (not shown) has received the information indicating that it is, the process proceeds to step S405.
(Step S405) The abnormality output unit 19 transmits abnormality detection information indicating that an abnormality has occurred to the Lth user to the server.

  (Step S406) The abnormality output unit 19 determines whether an abnormality in which the Lth user is followed and / or an abnormality that has been ambushed is detected. If at least one of them is detected, the process proceeds to step S407, and if not, the process proceeds to step S408.

(Step S407) The abnormality output unit 19 transmits abnormality information indicating that the user is following and / or waiting to the information terminal of the Lth user.
(Step S408) The abnormality output unit 19 increments the counter L by 1.

  (Step S409) The abnormality output unit 19 determines whether there is an L-th user in which an abnormality is detected. If it exists, the process returns to step S402. If it does not exist, the series of processing ends.

In the flowchart of FIG. 7, the case where abnormality detection information is transmitted to the server after confirming safety to the user when an abnormality related to the user's behavior is detected has been described. You may transmit abnormality detection information to a server directly, without performing. Needless to say, the output destination of the abnormality detection information is not limited to the flowchart of FIG.
Next, the operation of the user abnormality detection device 1 according to the present embodiment will be described using a specific example.

  First, processing for receiving a captured image output from the camera will be described. In this specific example, it is assumed that captured images from a plurality of cameras are accepted. Assume that each camera sequentially outputs a camera ID for identifying the camera, date information indicating the date and time when each frame of the captured image was captured, and frames constituting the captured image in association with each other. In this specific example, for convenience of explanation, it is assumed that the frame rate of the captured image is 1 fps.

  When the captured image reception unit 11 receives the captured image transmitted from each camera, the captured image reception unit 11 passes the captured image to the captured image storage unit 12 (step S101). As shown in FIG. 8, the captured image storage unit 12 stores the camera ID, the date / time information, and the frames constituting the captured image in association with each other in a predetermined recording medium (step S102). In FIG. 8, P001 and the like are information for identifying the frames constituting the photographed image.

  Next, processing for receiving position information will be described. In this specific example, a mobile phone as an information terminal possessed by a user has a GPS function, and position information acquired using the GPS function is sequentially transmitted to the user abnormality detection device 1 together with a user ID for identifying the user. Suppose that it is transmitted via a network. This network is, for example, the Internet, a public telephone line, an intranet, or the like. FIG. 9 is a diagram illustrating an example of a position information packet transmitted from a user information terminal. The payload of the packet includes a user ID and position information. This payload may be encrypted or unencrypted. In the position information, it is assumed that information indicating the position of the user is associated with date information indicating the date and time when the position is acquired.

  When the position information receiving unit 13 receives the position information transmitted from the user information terminal, the position information receiving unit 13 passes the position information to the position information storage unit 14 (step S103). As shown in FIG. 10, the position information storage unit 14 stores the user ID and the position information in association with each other in a predetermined recording medium (step S104). In this position information, as shown in FIG. 10, date / time information is associated with longitude / latitude indicating the position of the user.

  Next, processing for identifying a user from a person in a captured image will be described. As described above, it is assumed that the process of specifying a user is executed at a timing when a new captured image is received. When a frame of a new captured image is received by the captured image reception unit 11 (step S105), the action trajectory extraction unit 15 detects this and specifies an image area of a person in the frame of the newly received captured image. Then, a person ID for specifying a person is assigned to the image area of each specified person (step S201). At the time of assigning the person ID, as described above, the same person ID as the person ID assigned in the frame of the past photographed image is assigned to the image area of the person included in the frame of the past photographed image. Thereafter, the behavior trajectory extraction unit 15 extracts a behavior trajectory for each person's image area in the newly accepted frame of the captured image (steps S202 to S205). That is, the action trajectory extraction unit 15 is information indicating the date and time information of the frame of the newly accepted captured image, the person ID assigned to the person image area, and the position of the person image area identified by the person ID. Are stored in a predetermined recording medium in association with each other. It is assumed that the information indicating the position of the person's image area at the time of accumulation is converted into a position seen from above by plane projective transformation or the like. FIG. 11 is a diagram showing the person IDs thus accumulated and information indicating the action trajectory. In the action trajectory, information indicating the position of the person's image region is shown as coordinates (x, y) from a predetermined reference point in the two-dimensional plane orthogonal coordinate system.

  Thereafter, the comparison unit 16 extracts the position information accumulated by the position information accumulation unit 14 from the action trajectory indicated by the position information and the captured image by performing, for example, Gaussian gel conversion using the GRS80 ellipsoid model. It is possible to compare with the action trajectory. By this Gaussian gel conversion, the WGS-84 coordinate system (GPS data coordinate system) is converted to a plane orthogonal coordinate system (public survey coordinate system). It is assumed that the plane orthogonal coordinate system after the Gaussian gel conversion and the plane orthogonal coordinate system of the action trajectory extracted from the captured image have the same reference point and the same coordinate axis. In addition, about the parameter of the Gaussian gel conversion, you may use the value shown by "Ministry of Land, Infrastructure, Transport and Tourism No. 9" published by the Ministry of Land, Infrastructure, Transport and Tourism of Japan, for example. FIG. 12 is a diagram illustrating position information obtained by converting longitude / latitude information indicated by the position information illustrated in FIG. 10 into coordinates (x, y) in a plane orthogonal coordinate system.

  The comparison unit 16 compares the action locus indicated by the converted position information with the action locus extracted by the action locus extraction unit 15 for each user (steps S206 and S207). Using the comparison result, the user identification unit 17 determines whether or not the action trajectory indicated by the user's position information matches any one of the extracted action trajectories (step S208). Here, the action trajectory of the person identified by the person ID “F002” (FIG. 13A) and the action trajectory indicated by the position information of the user identified by the user ID “U001” (FIG. 13B). Is similar to “0.95”. Further, it is assumed that the user trajectory of the user ID “U001” has the highest similarity with the behavior trajectory of the person ID “F002”. If the similarity of the action trajectory is equal to or greater than the threshold value “0.9”, and the user identification unit 17 is set to determine that both match, the user identification unit 17 identifies the person identified by the person ID “F002”. The user identified by the user ID “U001” is specified (step S209). Then, as illustrated in FIG. 14, the user specifying unit 17 stores information that associates the person ID “F002” with the user ID “U001” in a predetermined recording medium. Note that the comparison process by the comparison unit 16 and the user identification process by the user identification unit 17 are executed for all users for whom position information has been received (steps S207 to S211).

  Next, the process which detects the abnormality regarding a user's action among the processes which detect a user's abnormality is demonstrated. In this specific example, a user identified by a user ID “U001” (hereinafter, sometimes referred to as “user U001”) will be described. The process for detecting an abnormality can be executed in the same manner for other users.

  In this specific example, it is assumed that user past information indicating an action locus extracted from a captured image is stored for each user in the user past information storage unit 21. This user past information is the same as the information shown in FIG. 11 except that it has an action locus associated with the user ID. In addition, it is assumed that the normal behavior information storage unit 22 stores information indicating an area through which a normal person walks.

  First, the detection unit 25 calculates a probability that the current behavior of the user U001 is different from the past behavior trajectory of the user U001 indicated by the user past information (step S302). In this specific example, pattern matching is performed on the past behavior trajectory of the user U001 indicated by the user past information and the current behavior trajectory of the user U001 sequentially extracted by the behavior trajectory extraction unit 15, and the probability that the two do not match is calculated. By doing this, the probability that the current action of the user U001 is different from the past action locus is calculated. As shown in FIG. 15, when the person with the person ID “F002” corresponding to the user U001 performs an action different from the past action trajectory, the current action of the user U001 is indicated by the user past information. The probability that it is different from the past action trajectory of U001 is high, for example, “0.80” is calculated.

  The detecting unit 25 calculates a probability that the current action of the user U001 is different from the action of the normal person indicated by the normal action information (Step S303). In this specific example, as indicated by the hatched area in FIG. 16, a normal person's passing range is shown by the normal action information, and when the user passes through the range, the user's action is normal. A probability “0” that is different from the behavior of the person is calculated. If not, that is, if the user passes outside the range, a probability “1” that the user's behavior is different from that of the normal person is calculated. . Here, the information indicating the hatched area in FIG. 16 may be, for example, information specifying pixels around the hatched area, or information specifying pixels included in the hatched area. May be. As illustrated in FIG. 16, when the person with the person ID “F002” corresponding to the user U001 moves beyond the range through which the normal person passes, the probability “ 1 "is calculated.

  Here, in this specific example, weighting is performed by multiplying each of the probability different from the user's past behavior and the probability different from the normal human behavior by multiplying by “0.5”. That is, the average of both is calculated. Further, it is assumed that the threshold value to be compared with the result of the weighting of the probability is set to “0.8”.

  Then, in the above case, the result of the weighting of each probability is “0.9” (step S304), it is determined that the threshold is “0.8” or more (step S305), and the detection means 25 An abnormality relating to the action of the user U001 is detected (step S306). It is assumed that the detection unit 25 holds in a predetermined recording medium that an abnormality relating to the action of the user U001 has been detected.

  Next, of the processes for detecting a user abnormality, a process for detecting a tail abnormality for the user will be described. First, the behavior trajectory comparison unit 23 compares the current behavior trajectory of the user U001 extracted by the behavior trajectory extraction unit 15 with the current behavior trajectory of each person other than the user U001 extracted by the behavior trajectory extraction unit 15 ( Step S307). For example, as shown in FIG. 17, when a person with a person ID “F010” moves after a person with a person ID “F002” corresponding to the user U001 by a similar action locus, action locus comparison means Suppose that it is determined that the similarity between them is “0.90”.

  In this specific example, if the degree of similarity is “0.80” or more, it is determined that there is a certain degree of similarity. Then, the detection unit 25 determines that the action trajectory of the person with the person ID “F010” has a certain similarity or more with the action trajectory of the user U001 (step S308), and detects an abnormality that the user U001 is followed. It detects (step S309). It is assumed that the detection unit 25 holds in a predetermined recording medium that an abnormality regarding the tail of the user U001 has been detected.

Next, a process for detecting a user abnormality will be described, particularly a process for detecting an ambush abnormality for the user. First, the predicted behavior trajectory calculating unit 24 calculates the predicted behavior trajectory of the user U001 (step S310). In this specific example, it is assumed that the predicted behavior trajectory calculation unit 24 calculates the predicted behavior trajectory using the past behavior trajectory of the user U001. In other words, the predicted action trajectory calculating unit 24 obtains information indicating the past action trajectory of the user U001 (in this specific example, the one shown in FIG. 15) from the user past information storage unit 21 through a route (not shown). read out. Then, as indicated by the dashed arrow in FIG. 18, the predicted behavior trajectory of the person with the person ID “F002” corresponding to the user U001 is calculated. Thereafter, the detection means 25 determines whether or not there is a person who stops within a certain range from the predicted action trajectory, for example, within 2 meters from the predicted action trajectory, and the person stops at a place where a normal person does not stop. If there is, the ambush abnormality is detected (step S311). Here, as shown in FIG. 19, the detection means 25 determines whether or not a place where a normal person does not stop using information indicating a place where a normal person stops because of waiting for a signal or the like. . In this specific example, as shown in FIG. 18, the person with the person ID “F011” stops within 2 meters from the predicted behavior trajectory of the user U001, and the position is the place where the normal person shown in FIG. 19 stops If not, the detection means 25 detects an abnormality that the user U001 is awaited (step S312). It is assumed that the detection unit 25 holds in a predetermined recording medium that an abnormality regarding the ambush of the user U001 has been detected.
Next, processing in which the abnormality output unit 19 outputs abnormality detection information will be described.

  First, the case where abnormality regarding a user's action is detected about user U001 is demonstrated. When the detection means 25 of the abnormality detection unit 18 reads out information indicating that each of the abnormality relating to the behavior of the user U001, the tailing and the ambush from the predetermined recording medium and passes it to the abnormality output unit 19, The output unit 19 determines that an abnormality has been detected (step S109), and starts a process of outputting abnormality detection information (step S110).

  First, the abnormality output unit 19 determines that an abnormality relating to the action of the user U001 has been detected (step S402), and confirms the safety of the user U001 at the address of the mobile phone as the information terminal of the user U001 registered in advance. Is transmitted (step S403). When the information indicating that the user U001 has no problem is returned to the information, the abnormality detection unit 18 detects an abnormality related to the user's behavior, but there is actually no problem. On the other hand, when there is no response from the user U001 for a certain time or when the user abnormality detection device 1 receives information indicating that a problem has occurred from the user U001, the abnormality output unit 19 generates an abnormality in the user U001. The abnormality detection information including the effect is transmitted to a server such as a security company or the police (step S405). The abnormality detection information transmitted to the security company or the like may be transmitted including information indicating the current location of the user U001. As a result, a person for rescue of the user U001 can be dispatched from a security company or the like.

  Next, a case will be described in which a user's tail and ambush abnormality is detected for the user U001. The abnormality output unit 19 determines that the user U001's tailing and ambush abnormality has been detected (step S406), and indicates that the tailing and ambush abnormality has been detected in the address of the mobile phone as the information terminal of the user U001. Abnormality detection information is transmitted (step S407). This anomaly detection information is, for example, tailed or ambushed so that the person who is following or who is ambushing will not notice that the anomaly detection information has been received by the mobile phone of user U001. Information that controls the ringing melody that is set in advance to ring in the case may be played on the mobile phone of the user U001.

  In this specific example, the case where the processing such as comparison is performed after the position information is converted into the two-dimensional plane orthogonal coordinate system has been described. However, the coordinates used in the position information are action trajectories extracted from the captured image. Processing such as comparison may be performed after conversion to a system.

  As described above, in the user abnormality detection device 1 according to the present embodiment, it is possible to specify a user shown in a captured image by using the user's position information and the captured image captured by the camera. Therefore, even if the user's facial features or the like are not registered in advance, it is possible to specify the user who is shown in the captured image. In addition, since the user abnormality detection device 1 according to the present embodiment can know the user's position more precisely than the GPS or the like from the captured image, the abnormality relating to the user identified by the user identification unit 17 is detected more appropriately. Thus, the abnormality detection information can be output. As a result, it is possible to detect, for example, an abnormality such as the user being involved in an accident or an incident, or an abnormality such as a fall due to illness. In addition, in the user abnormality detection device 1 according to the present embodiment, by using a captured image, it is possible to track the behavior of a person other than that specified by the user specifying unit 17, and that the user is following or waiting. Abnormalities can be detected. Further, by using captured images of two or more regions that are not overlapped and captured by two or more cameras, even if the user does not exist within the region of the captured image, the user's abnormality is detected. . For example, from the area of one captured image to the area of another captured image, the user's past behavior pattern moves in 2 minutes, but from the area of one captured image to another captured image. If the user does not appear in another captured image area even after 5 minutes have passed from the user facing the area, an abnormality relating to the user can be detected. As described above, the user abnormality detection device 1 according to the present embodiment can also detect a wider range of abnormality. In addition, the user abnormality detection device 1 according to the present embodiment can notify the user himself / herself, the user's family, a security company, or the like that such an abnormality has occurred. Therefore, it is possible to appropriately cope with an abnormality that has occurred in the user.

  In the present embodiment, the case where the user's position information is acquired using the GPS function of the information terminal held by the user has been described. However, the user's position information may be acquired by other methods. There is no limitation on the method of acquiring information. For example, the user's position information may be acquired using an RF tag, an optical tag, a wireless LAN technology, or the like. When acquiring user position information using an RF tag, for example, the user has an RF tag, and a receiver installed on a road or indoor has received radio waves from the user's RF tag. In some cases, information indicating the position of the receiver may be acquired as the user's position information, or the user's position information may be acquired by performing a three-point survey using radio waves from the RF tag of the user. Good. Alternatively, an RF tag may be installed on a road or indoor, and the user's location information may be acquired by receiving radio waves from the RF tag by a receiver that the user has. The position information of the user may be acquired in the same manner using an optical tag, wireless LAN, Bluetooth (registered trademark), or the like. These methods are conventionally known and will not be described in detail.

  Moreover, although this Embodiment demonstrated the case where position information was the information along a time series, position information may not be the information along a time series. That is, the position information may be information indicating the position where the user has passed.

  In the above description, the case where the photographed image is a street corner photographed by the camera has been described. However, the photographed image may be a photograph taken indoors, and the photographed place where the user passes by the camera. If so, the shooting location by the camera does not matter.

  In the above embodiment, each process or each function may be realized by centralized processing by a single device or a single system, or may be distributedly processed by a plurality of devices or a plurality of systems. It may be realized by doing.

  In the above embodiment, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component is realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. In addition, the software which implement | achieves the user abnormality detection apparatus in the said embodiment is the following programs. That is, the program includes a captured image reception step that receives a captured image captured by one or more cameras at a location where the user passes, a position information reception step that receives position information that is information indicating the position of the user, An action trajectory extraction step for extracting a person's action trajectory in the captured image, a comparison step for comparing the action trajectory extracted in the action trajectory extraction step with a user's action trajectory indicated by the position information, and the comparison step Based on the comparison result, a user specifying step for specifying a user whose position is indicated by the position information received in the position information receiving step from an image of a person in the captured image, and an abnormality related to the user specified in the user specifying step Detecting an abnormality, and detecting the abnormality detection step. When detecting an abnormality relating user-up, but abnormal for executing the abnormal output step of outputting an abnormality detection information which is information related to that detected.

  In the above program, the output step for outputting information, the accepting step for receiving information, etc. do not include at least processing performed only by hardware, for example, processing performed by a modem or an interface card in the output step. .

  Further, this program may be executed by being downloaded from a server or the like, and a program recorded on a predetermined recording medium (for example, an optical disc such as a CD-ROM, a magnetic disc, a semiconductor memory, etc.) is read out. May be executed by

  Further, the computer that executes this program may be singular or plural. That is, centralized processing may be performed, or distributed processing may be performed.

  FIG. 20 is a schematic diagram illustrating an example of an external appearance of a computer that executes the program and realizes the user abnormality detection apparatus 1 according to the embodiment. The above-described embodiment is realized by computer hardware and a computer program executed on the computer hardware.

  20, a computer system 100 includes a computer 101 including a CD-ROM (Compact Disk Read Only Memory) drive 105 and an FD (Flexible Disk) drive 106, a keyboard 102, a mouse 103, and a monitor 104. Here, the abnormality output unit 19 may display the abnormality detection information on the monitor 104, may transmit the information via the network as described above, or may perform other output. When abnormality detection information is transmitted, the computer system 100 has a device corresponding to the abnormality detection information.

  FIG. 21 is a diagram illustrating a computer system. In FIG. 21, in addition to the CD-ROM drive 105 and the FD drive 106, a computer 101 includes a CPU (Central Processing Unit) 111, a ROM (Read Only Memory) 112 for storing a program such as a bootup program, A CPU (Random Access Memory) 113 that is connected to the CPU 111 and temporarily stores application program instructions and provides a temporary storage space, a hard disk 114 that stores application programs, system programs, and data, a CPU 111 and a ROM 112. Etc. to each other. The computer 101 may include a network card (not shown) that provides connection to the LAN.

  A program for causing the computer system 100 to execute the function of the user abnormality detection device 1 according to the above embodiment is stored in the CD-ROM 121 or the FD 122, inserted into the CD-ROM drive 105 or the FD drive 106, and the hard disk 114. May be forwarded to. Instead, the program may be transmitted to the computer 101 via a network (not shown) and stored in the hard disk 114. The program is loaded into the RAM 113 at the time of execution. The program may be loaded directly from the CD-ROM 121, the FD 122, or the network.

The program does not necessarily include an operating system (OS) or a third party program that causes the computer 101 to execute the function of the user abnormality detection device 1 according to the above-described embodiment. The program may include only a part of an instruction that calls an appropriate function (module) in a controlled manner and obtains a desired result. How the computer system 100 operates is well known and will not be described in detail.
Further, the present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these are also included in the scope of the present invention.

  As described above, according to the user abnormality detection device and the like according to the present invention, it is possible to detect an abnormality that has occurred in the user, which is useful as a system for detecting a user abnormality.

The block diagram which shows the whole structure of the user abnormality detection apparatus by Embodiment 1 of this invention. The figure for demonstrating imaging | photography of the picked-up image with two or more cameras in Embodiment 1 of this invention. The block diagram which shows the internal structure of the abnormality detection part by the embodiment The flowchart which shows the whole operation | movement of the user abnormality detection apparatus by the embodiment The flowchart which shows the user specific operation | movement of the user abnormality detection apparatus by the embodiment The flowchart which shows the operation | movement of abnormality detection of the user abnormality detection apparatus by the embodiment The flowchart which shows the operation | movement of the output of the abnormality information of the user abnormality detection apparatus by the embodiment The figure which shows an example of the picked-up image which the picked-up image storage part by the same embodiment accumulate | stored The figure which shows an example of the packet of the positional information in the embodiment The figure which shows an example of the positional information in the embodiment The figure which shows an example of the extracted action locus | trajectory in the embodiment The figure which shows an example of the converted positional information in the embodiment The figure for demonstrating the process which identifies the user in the embodiment The figure which shows matching with user ID and person ID in the embodiment The figure for demonstrating the detection of the abnormality in the embodiment The figure for demonstrating the detection of the abnormality in the embodiment The figure for demonstrating the detection of the abnormality in the embodiment The figure for demonstrating the detection of the abnormality in the embodiment The figure for demonstrating the detection of the abnormality in the embodiment Schematic diagram showing an example of the appearance of the computer system in the embodiment The figure which shows an example of a structure of the computer system in the embodiment

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 User abnormality detection apparatus 11 Captured image reception part 12 Captured image storage part 13 Position information reception part 14 Position information storage part 15 Action locus extraction part 16 Comparison part 17 User specification part 18 Abnormality detection part 19 Abnormal output part 21 User past information storage Means 22 Normal action information storage means 23 Action trajectory comparison means 24 Predictive action trajectory calculation means 25 Detection means

Claims (8)

A photographed image receiving unit that accepts a photographed image taken by one or more cameras in a place where the user passes;
A position information receiving unit that receives position information that is information indicating the position of the user;
An action trajectory extraction unit for extracting an action trajectory of a person in the captured image;
A comparison unit that compares the behavior trajectory extracted by the behavior trajectory extraction unit with the user's behavior trajectory indicated by the position information;
A user specifying unit that specifies a user whose position is indicated by the position information received by the position information receiving unit from the image of the person in the captured image based on the comparison result by the comparing unit;
An abnormality detection unit for detecting an abnormality related to the user specified by the user specifying unit;
A user abnormality detection device comprising: an abnormality output unit that outputs abnormality detection information that is information relating to an abnormality being detected when the abnormality detection unit detects an abnormality relating to a user. The user abnormality detection device according to claim 1, wherein the abnormality detection unit detects an abnormality related to a user behavior specified by the user specifying unit. The abnormality detection unit
User past information storage means for storing user past information that is information indicating the past actions of the user specified by the user specifying unit;
The detection means which detects the abnormality regarding the said user's action when the user's action which the said user specific part specified differs from the said user's past action which the said user past information shows is provided. User abnormality detection device. The abnormality detection unit
Normal behavior information storage means for storing normal behavior information, which is information related to normal human behavior,
The detection means which detects the abnormality regarding the said user's action when the user's action which the said user specific part specified differs from the action of the normal person which the said normal action information shows is provided. 3. The user abnormality detection device according to 3. The abnormality detection unit detects an abnormality related to the user when the behavior of a person other than the user identified by the user identification unit included in the captured image is abnormal in relation to the behavior of the user. Item 1. The user abnormality detection device according to Item 1. The abnormality detection unit
Action trajectory comparison means for comparing the action trajectory of one or more persons extracted by the action trajectory extraction unit with the user's action trajectory specified by the user specifying unit;
The user abnormality detection device according to claim 5, further comprising: a detection unit configured to detect an abnormality related to the user when the behavior locus comparison unit has a certain similarity or more. The abnormality detection unit
A predicted behavior trajectory calculating means for calculating a predicted behavior trajectory that is a predicted behavior trajectory of the user identified by the user identifying section;
When a person other than the user included in the photographed image stops at a position within a certain range from the predicted action trajectory calculated by the predicted action trajectory calculation means and does not stop a normal person The user abnormality detection device according to claim 5, further comprising: a detecting unit that detects an abnormality relating to the user. A captured image accepting step for accepting a photographed image taken by one or more cameras in a place where the user passes;
A position information receiving step for receiving position information which is information indicating the position of the user;
An action trajectory extraction step for extracting an action trajectory of a person in the captured image;
A comparison step for comparing the action locus extracted in the action locus extraction step with the action locus of the user indicated by the position information;
A user specifying step of specifying a user whose position is indicated by the position information received in the position information receiving step from the image of the person in the captured image based on the comparison result in the comparing step;
An abnormality detection step of detecting an abnormality relating to the user specified in the user specifying step;
A user abnormality detection method comprising: an abnormality output step of outputting abnormality detection information that is information relating to detection of an abnormality when an abnormality relating to a user is detected in the abnormality detection step.

Images