JP2006221355A - Monitoring device and monitoring system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To identify a person by means of even the image other than his or her front face highly precisely. <P>SOLUTION: In the monitoring device in which a photographed image is inputted from a camera and information is acquired from a storage part for storing various information, the storage part stores past information including information related with an image inputted in the past to the monitoring device and geographical information including the position information of the camera, and extracts the face image from the inputted image, and estimates a figure corresponding to the extracted face image on the basis of the past information, and decides the person corresponding to the extracted face image out of the estimated persons on the basis of the geographical information. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a monitoring system for monitoring an image taken by a camera, and more particularly to a technique for identifying a person appearing in an image using geographical information.

  In recent years, personal authentication based on facial image identification has become widespread. Face image identification is a technique for identifying a person from a face image. Specifically, a human face area is extracted from the image. Next, the extracted face area is collated with a pre-registered face image to determine similarity. Then, a person is specified from the determined similarity.

  Personal authentication using face images has little psychological resistance. This is because personal authentication using a face image is performed in a non-contact manner using an image photographed from a remote location, so that an input operation to a reading device is not required as in fingerprint authentication.

  However, personal authentication using a face image has a problem that it is vulnerable to variations in illumination, variations in the size of the face area in the image, variations in the orientation of the face, and the like.

  Therefore, a conventional personal authentication device using a face image performs authentication by acquiring an image suitable for authentication. For example, the personal authentication device acquires an image suitable for authentication by photographing the front face of the user placed at an appropriate position in front of the camera. However, this personal authentication device must take a picture of a user standing at an appropriate position for each authentication.

  Therefore, a monitoring device that solves this problem is known (see, for example, Patent Document 1). This monitoring apparatus can authenticate even if an image other than the front face is acquired by storing face images from a plurality of directions in advance. In other words, this monitoring apparatus can authenticate the user without stopping at a predetermined position.

  Also, a person tracking process is known as a technique using a person image (see, for example, Non-Patent Document 1). In the person tracking process, a face area is extracted from the surveillance camera image, and the face size and the face direction are obtained from the extracted face area. Next, the moving direction of the person, the moving speed of the person, and the like are obtained from the obtained face size and face orientation. Then, the person is traced based on the obtained person moving method and the moving speed of the person. This person tracking process is applied to monitoring devices and marketing.

  This person tracking process can be performed only by normal image processing using a difference or the like if various conditions are satisfied. The various conditions include that the person to be tracked is always within the screen and that the light intensity is abnormal.

  Therefore, in the conventional person tracking process, if a person leaves the screen and then re-enters, tracking cannot be continued. In other words, the person tracking process has a problem that a person who is out of the screen and a person who re-enters the person are treated as different persons.

In order to solve this problem, a technique of tracking with a plurality of cameras is known. However, this technique also has the same problem when there is a blind spot between the cameras.
JP 2002-92601 A Edited by Mikio Takagi, “New Image Analysis Handbook”, Full Revised Edition, The University of Tokyo Press, September 10, 2004

  The monitoring device described in Patent Document 1 described above can be authenticated from an image other than the front face of a person. However, since this monitoring device identifies a person only by the similarity of images, the accuracy of authentication becomes a problem.

  Accordingly, an object of the present invention is to provide a monitoring apparatus that can authenticate even an image other than the front face of a person and can authenticate with high accuracy.

  The present invention provides a monitoring device in which captured images are input from a camera and acquires information from a storage unit that stores various types of information. The storage unit includes past information including information related to images input to the monitoring device in the past. Storing information and geographic information including position information of the camera, extracting a face image from the input image, and estimating a person corresponding to the extracted face image based on the past information; Based on the geographic information, a person corresponding to the extracted face image is determined from the estimated persons.

  According to the present invention, even an image other than the front face of a person can be authenticated, and further, it can be authenticated with high accuracy.

Embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram of a monitoring system according to the first embodiment of this invention.

  The monitoring system monitors facilities such as buildings. The monitoring system includes a camera 101, a server 102, an entrance / exit control device 103, person appearance data 108, and geographical data 109.

  The server 102 is connected to the camera 101 and the entrance / exit control device 103 via the network 112.

  A plurality of cameras 101 are provided and are installed in different places. For example, the camera 101 is installed at a point where a branching occurs in the flow of people. In addition, the camera 101 captures an image of the place where it is installed, and transmits the captured image to the server 102. In addition, the area number which identifies each place uniquely is attached | subjected to the place where the camera 101 is installed.

  The server 102 is a computer that includes a CPU, a memory, and a storage unit. Various programs are stored in the storage unit, and the CPU executes these programs. As a result, the server 102 processes the image received from the camera 101. Then, the entrance / exit control device 103 is instructed to open and close the gate (not shown) according to the result of the processing. Note that the gate is installed in a facility monitored by the monitoring system of this embodiment. The memory temporarily stores various types of information including images received from the camera 101.

  Further, the server 102 includes a geography / time limit calculation unit 105 and a face identification unit 106. The face identification unit 106 extracts a face area from the image received from the camera 101, and estimates a person (corresponding person) appearing in the extracted face area. The geography / time limit calculation unit 105 identifies a corresponding person among the persons estimated by the face identification unit 106 based on the location and time when the image was taken.

  In addition, the server 102 acquires the person appearance data 108 and the geographical data 109. The person appearance data 108 and the geographical data 109 may be stored inside the server 102 or may be stored in a storage device outside the server 102.

  As will be described later with reference to FIG. 2, the person appearance data 108 includes information regarding an image captured by the camera 101. As will be described later with reference to FIG. 3, the geographic data 109 includes information regarding the location where the camera 101 is installed.

  The entrance / exit control device 103 controls the opening and closing of the gate of the facility monitored by the monitoring system.

  FIG. 2 is a configuration diagram of the person appearance data 108 according to the embodiment of this invention.

  The person appearance data 108 includes a face image data number 1081, an image feature amount 1082, an estimated person name 1083, a reliability 1084, an area number 1085, an appearance date 1086, and other attributes 1087.

  The face image data number 1081 is an identifier for uniquely identifying face image data. The face image data is an image obtained by extracting a face area from an image photographed by the camera 101.

  The image feature amount 1082 is a set of a predetermined number of numerical values, and is a value that characterizes the face image data. The feature amount 1082 is calculated by various known methods. The server 102 calculates the feature value 1082 from the face image by using a method suitable for this embodiment.

  Further, the server 102 calculates the similarity between the two face image data to be compared based on the feature amount 1082. The similarity is a degree of similarity between two face image data to be compared. That is, the larger the similarity, the more similar are the two images being compared.

  For example, the server 102 may obtain the similarity by regarding the image feature quantity 1082 as a vector. In this case, the server 102 calculates the distance between the image feature values 1082 of the two image data to be compared. Then, the server 102 obtains the similarity from the calculated distance. Specifically, the similarity becomes larger as the calculated distance is shorter.

  Further, the server 102 may calculate a correlation from the image feature values 1082 of two image data to be compared, and obtain the similarity from the calculated correlation.

  The estimated person name 1083 is the name of a person estimated to be a person (corresponding person) shown in the face image data. The estimated person name 1083 may have a plurality of records for one face image data number 1081. In this case, the corresponding person of the image is one of the estimated person names 1083.

  The reliability 1084 is a possibility that the estimated person becomes a corresponding person.

  The area number 1085 is an identifier that uniquely identifies the place where the face image data was taken. The appearance date 1086 stores the date and time when the face image data was taken.

  The other attribute 1087 stores information related to the face image data. In this explanatory diagram, the other attribute 1087 stores the orientation of the person, weather information, lighting information, and the like. The direction of the person is the direction in which the person shown in the face image data is facing, for example, an angle with the front direction of the camera 101 being 0 degrees. The weather information is the weather when the face image data is captured, and for example, information such as “sunny”, “cloudy”, and “rain” is stored. Illumination information is the state of illumination when the face image data is captured. For example, information such as “forward light”, “backlight”, “right oblique light”, “left oblique light”, and “indoor” is stored. The

  The person appearance data 108 may store personal information for each person in addition to these pieces of information. The personal information includes a face image data number of the person, an average value of all the image feature amounts related to the person, authentication information of the person, and the like.

  FIG. 3 is a configuration diagram of the geographic data 109 according to the embodiment of this invention.

  The geographic data 109 is used by the geography / time limit calculation unit 105 in the process of identifying the corresponding person. The geographic data 109 stores an average time required for moving from the location of the starting area number 1091 to the location of the destination area number 1092.

  The starting area number 1091 is an identifier that uniquely identifies a location of the person before the movement. The destination area number 1092 is an identifier that uniquely identifies the location of the person after movement.

  For example, the average travel time from the place where the starting area number 1091 is “0001” to the place where the destination area number 1092 is “0002” is “2 seconds”.

  Each box may store a plurality of average movement times. In this case, it indicates that there are a plurality of movement methods. The moving method is, for example, an elevator, an escalator, a staircase, or the like.

  For example, the average travel time from the location where the origin area number 1091 is “0002” to the location where the destination area number 1092 is “0034” varies depending on the travel method. This average travel time is “10 seconds” when the elevator is used, “125 seconds” when the escalator is used, and “250 seconds” when the stairs are used.

  If the movement from the location of the starting area number 1091 to the location of the destination area number 1092 can only be made via another location where the camera 101 is installed, “non-adjacent” is stored in the corresponding box. . In this case, the server 102 searches for a movement route from the location of the origin area number 1091 to the location of the destination area number 1092. Then, the server 102 calculates the average travel time from the location of the origin area number 1091 to the location of the destination area number 1092 by adding up all the average travel times on the travel route.

  FIG. 4 is a flowchart of image processing of the server 102 according to the first embodiment of this invention.

  When an image captured by the camera 101 is input from the camera 101, the server 102 starts processing (201). At this time, the server 102 may input the area number of the place where the image was taken, the time when the image was taken, and the like.

  First, an image of a face area (face image) is extracted from the input image (202). If there are a plurality of human face areas in the input image, each face image is extracted. Then, the following processing is performed on each extracted face image.

  Next, an image feature amount is calculated from the extracted face image (203). Next, the image feature amount 1082 of the person appearance data 108 is extracted in order from the top record. Next, the similarity is calculated from the image feature amount calculated in step 203 and the extracted image feature amount 1082 (204). That is, the similarity is calculated for all records of the person appearance data 108.

  Next, one of the calculated similarities is extracted in descending order of value. Next, a record corresponding to the extracted similarity is selected from the person appearance data 108 (205).

  Next, it is determined whether or not the extracted similarity is a threshold value or more (206).

  If the similarity is smaller than the threshold, it is determined that the two face images to be compared are not similar. Therefore, the person shown in the face image extracted in step 202 is determined as a new person (210), and the process ends. The new person is a person whose face image data is not stored in the person appearance data 108.

  On the other hand, if the similarity is equal to or greater than the threshold, it is determined that the two face images to be compared are similar, and the area number 1085 of the record selected in step 205 is extracted. Next, the extracted area number 1085 and the origin area number 1091 of the geographic data 109 match, and the area number input in step 201 and the destination area number 1092 of the geographic data 109 match are stored in the box. The obtained average travel time is extracted from the geographical data 109 (207).

  Next, the appearance date 1086 of the record selected in step 205 is subtracted from the time input in step 201.

  Next, it is determined whether or not the subtracted value is included in a range (for example, within ± 20%) centered on the extracted average travel time. Accordingly, it is determined whether the person can naturally move from the place where the image corresponding to the record selected in step 205 is taken to the place where the image inputted in step 201 is taken (208).

  If it is determined that it cannot move naturally, the person shown in the two images to be compared is determined as a different person. Then, the process returns to step 205 and the same processing is repeated for the image having the next highest similarity.

  On the other hand, if it is determined that it can move naturally, the estimated person name 1083 of the record selected in step 205 is determined as a corresponding person (209), and the process ends. When a plurality of person names are stored in the estimated person name 1083, the highest one is extracted from the reliability 1084 of the record selected in step 205. Then, the estimated person name 1083 corresponding to the extracted reliability 1084 is determined as the corresponding person.

  Note that information regarding the face image extracted in step 203 may be registered in the person appearance data 109 before the processing ends.

  Specifically, first, a new record is created in the person appearance data 108. Next, a non-overlapping face image data number is assigned to the face image extracted in step 202. The assigned face image data number is stored in the face image data number 1081 of the new record. Next, the image feature amount calculated in step 203 is stored in the image feature amount 1082 of the new record.

  Next, the determined name of the corresponding person is stored as the estimated person name 1083 of the new record. Next, the reliability is calculated based on the similarity calculated in step 204, the area number input in step 201, the time of shooting, and the like. Then, the calculated reliability is stored in the reliability 1084 of the new record.

  Next, the area number input in step 201 is stored in the area number 1085 of the new record. Next, the time input in step 201 is stored in the appearance time 1086 of a new record. Further, when other information (for example, lighting information including the weather at the time of shooting) is input from the camera 101, the input other information is stored in the other attribute 1087 of the new record.

  As described above, according to the present embodiment, it is possible to determine persons having similar images by referring to the person appearance data 108. Furthermore, by referring to the geographic data 109, it is possible to specify a corresponding person among persons having similar images.

(Second Embodiment)
In the second embodiment, the server 102 tracks a person to reduce the image processing (FIG. 4) of the first embodiment.

  FIG. 5 is a block diagram of the configuration of the monitoring system according to the second embodiment of this invention.

  The monitoring system according to the second embodiment of the present invention is the same as the monitoring system according to the first embodiment (FIG. 1) except for the configuration included in the server 102. The same number is attached | subjected to the same structure and description is abbreviate | omitted.

  The server 102 includes a person detection / tracking unit 104 and an entrance / exit management unit 107 in addition to the configuration included in the server 102 (FIG. 1) of the first embodiment.

  The person detection / tracking unit 104 performs a human flow tracking process (FIG. 6) on the face image extracted by the face identification unit 106. The entrance / exit management unit 107 instructs the entrance / exit control device 103 to open and close the gate.

  FIG. 6 is an explanatory diagram of the human flow tracking process of the server 102 according to the second embodiment of this invention.

  In this explanatory diagram, a process of tracking a person 601 who is walking will be described.

  The camera 101A continuously captures the person 601 and inputs the captured images to the server. The server 102 extracts a face image from the image input from the camera 101A. The server 102 calculates the moving direction of the person 601 and the moving speed of the person 601 based on the extracted face image.

  The server 102 determines whether the position of the face image in the image captured by the camera 101A is consistent with the calculated movement direction and movement speed. For example, if the moving direction of the position of the face image is opposite to the calculated moving direction, it is determined that there is a contradiction. If the moving speed of the face image position is too fast compared to the calculated moving speed, it is determined that there is a contradiction.

  If the positions of the face images are not inconsistent, the server 102 determines that the face images extracted from the two images photographed in succession are the same person 601. Then, the server 102 registers the face image in the person appearance data 108 as information on the person 601.

  Note that if the extracted face image is unclear, the server 102 does not register the face image in the person appearance data 108. Thus, since the server 102 registers only a clear face image in the person appearance data 108, the person appearance data 108 clearly stores the characteristics of the person. As described above, the server 102 tracks the person 601.

  On the other hand, if the position of the face image is inconsistent, the server 102 determines that the face image extracted from the two images photographed consecutively belongs to another person 602.

  Further, the server 102 determines that the face image extracted from the image photographed by another camera 101 </ b> B belongs to another person 603.

  If the server 102 determines that the face image is that of another person 602, 603, the server 102 determines the corresponding person of the face image using the image processing (FIG. 4) of the first embodiment.

  FIG. 7 is a flowchart of image processing of the server 102 according to the second embodiment of this invention.

  The server 102 starts processing when an image photographed by the camera 101 is input from the camera 101 (301). At this time, the server 102 may input the area number of the place where the image was taken, the time when the image was taken, and the like.

  First, an image of a face area (face image) is extracted from the input image (302). Next, the size of the extracted face image is obtained.

  Further, the direction of the face shown in the extracted face image is estimated. For example, the orientation of the face is estimated based on the person appearance data 108 that stores the image feature quantities of the face image taken from a plurality of angles. In this case, the similarity is calculated from the image feature amount stored in the person appearance data 108 and the image feature amount of the face image extracted in step 302. Next, the image feature quantity (stored in the person appearance data) having the largest calculated similarity is specified. Then, the angle at which the image of the specified image feature amount is captured is estimated as the orientation of the face shown in the face image.

  Next, based on the obtained size of the face image and the estimated face orientation, a human flow tracking state is calculated. The human flow tracking situation includes the moving direction of the person reflected in the face image and the moving speed of the person. The human flow tracking situation can be calculated using, for example, existing image processing (see, for example, Non-Patent Document 1).

  Next, the direction and amount of change from the position of the face area in the image input from the camera 101 to the position of the face area in the image input immediately before from the camera 101 is obtained. Then, referring to the obtained change direction, change amount, and human flow tracking status (303), it is determined whether or not the face image is a tracking image (304). Specifically, when the obtained change direction matches the moving direction included in the human flow tracking situation and the obtained change amount is not inconsistent with the moving speed included in the human flow tracking situation, the image is determined to be a tracking image.

  If the face image is not the image being tracked, person determination processing is performed (307), and the process proceeds to step 308. The person determination process 307 is the same as steps 203 to 210 in the image processing (FIG. 4) of the first embodiment, and a description thereof will be omitted.

  On the other hand, if the face image is an image being tracked, the corresponding person of the image input immediately before from the camera 101 that captured the face image is determined as the corresponding person of the face image (305).

  Specifically, the face image data number of the image input immediately before from the camera 101 where the image input in step 201 was taken is searched. Next, a record in which the searched face image data number matches the face image data number 1081 of the person appearance data 108 is specified. Next, the highest reliability 1084 is selected from the identified record. Next, the estimated person name 1083 corresponding to the selected reliability 1084 is determined as the person corresponding to the face image.

  Next, the reliability of the determined person is calculated based on the change direction of the person, the amount of change of the person, the flow tracking situation, and the like obtained in step 303.

  Next, an image feature amount is calculated from the face image extracted in step 302 (306). Next, when personal information of a person is included in the person appearance data 108, the average value of the image feature amount of the personal information is corrected based on the calculated image feature amount.

  Next, information about the face image is stored in the person appearance data 108 (308).

  Specifically, first, a new record is created in the person appearance data 108. Next, a non-overlapping face image data number is assigned to the extracted face image. The assigned face image data number is stored in the face image data number 1081 of the new record.

  Next, the image feature amount calculated in step 306 is stored in the image feature amount 1082 of the new record. Next, the determined person name is stored in the estimated person name 1083 of the new record.

  Next, the reliability calculated in step 305 is stored in the reliability 1084 of the new record. Next, the area number input in step 301 is stored in the area number 1085 of the new record.

  Next, the time input in step 201 is stored in the appearance date 1086 of the new record. When other information (for example, lighting information including the weather when the image was taken) is input from the camera 101, the input other information is stored in the other attribute 1087 of the new record.

  Next, it is determined whether the camera 101 that has input the image is the camera 101 installed in front of the gate (309). Specifically, it is determined whether the area number input in step 301 is the area number before the gate.

  If the camera 101 is not installed in front of the gate, it is not necessary to authenticate, and the process ends.

  On the other hand, if the camera 101 is installed in front of the gate, it is determined whether the corresponding person determined in step 305 or step 307 is permitted to enter (310). Specifically, it is determined whether the corresponding person is permitted to enter by referring to the authentication information stored in the person appearance data 108.

  If entry is permitted, it is determined whether or not the reliability calculated in step 305 is greater than or equal to a threshold (311).

  If the reliability is equal to or greater than the threshold value, it is determined that the person shown in the image is permitted to enter, and the entrance / exit control device 103 is instructed to unlock the gate (312). Then, the process ends. The entrance / exit control device 103 unlocks the gate.

On the other hand, if it is determined in step 310 that the person is not permitted to enter, or if it is determined in step 311 that the reliability is smaller than the threshold, the image is transmitted to the administrator (313). Then, the process ends. The administrator can maintain the security of the facility by referring to the received image.

  Next, a case where the monitoring system of the present embodiment is used for ensuring security in a facility such as a building will be described.

  This facility is kept secure by authenticating at the facility gate and the gate of each office in the facility.

  As the camera 101, a normal surveillance camera is used. The camera 101 is mainly installed at a place where the flow of a person branches. The place where the flow of the person branches is, for example, an elevator doorway or a staircase. In addition, the camera 101 is installed for each range covered by the field of view.

  Next, processing of the monitoring system for a person (registrant) who is permitted to enter the facility will be described. The registrant is a person who regularly and frequently enters and exits the facility, such as a person who works at the facility. This will be explained when the registrant goes from the gate of the facility to the target office.

  The monitoring system registers the registrant's face image and facility entry permission information in person appearance data 108 in advance. Not only the face image of the registrant's front face but also face images taken from a plurality of directions are registered. For example, 15 faces comprising a combination of five horizontal directions of front, right 45 degrees, right 90 degrees, left 45 degrees and left 90 degrees and three vertical directions of front, depression angle 10 degrees and depression angle 45 degrees. Register the image.

  First, the registrant approaches the facility gate, and the camera 101 installed at the facility gate photographs the registrant. Then, the server 102 specifies a person shown in the image input from the camera 101 installed at the gate of the facility. In this case, it is specified that the person shown in the image is a registrant. Then, the server 102 instructs unlocking of the facility gate.

  In addition, the said monitoring system may use other authentication means together, when high safety | security is required. Other authentication means are, for example, input of a personal identification number, an ID card, biometrics, or the like.

  At this time, the server 102 extracts a face image from the image and registers the extracted face image in the person appearance data 108 as registrant information.

  When the facility gate opens, the registrant moves toward the destination office. The camera 101 on the registrant's movement path captures the registrant and inputs the captured image to the server 102. The server 102 extracts a face image from the input image. Next, the server 102 tracks the extracted face image. As a result, the server 102 can prevent a person different from the registrant from being mixed, and further reduce the processing.

  If the server 102 cannot track, the server 102 determines a corresponding person of the extracted face image. For example, the registrant is out of the field of view of the camera 101 installed at the gate and enters the field of view of the camera 101 installed in the elevator hall on the first floor. In this case, the server 102 identifies persons with similar images based on the image feature value 1082 of the person appearance data 108. Next, the server 102 calculates the moving direction and moving speed of the person shown in the face image from the extracted face image. Next, the server 102 refers to the calculated moving direction and moving speed, and determines the corresponding person shown in the extracted face image from the persons with similar images. Specifically, the server 102 determines whether the distance between the camera 101 installed at the gate and the camera 101 installed in the elevator hall and the moving time of the registrant are consistent with the calculated moving direction and moving speed. Thus, the corresponding person is determined. The server 102 may determine the corresponding person by referring to the appearance date 1086 of the person appearance data 108 and other attributes 1087 together. Thereby, the reliability of the determined corresponding person can be improved.

  If the extracted face image is clear, the server 102 registers the face image in the person appearance data 108 as registrant information.

  Next, the registrant moves from the elevator hall on the first floor to the elevator hall on the sixth floor, and then moves to the destination office on the sixth floor.

  The server 102 determines the name of the person shown in the image input from the camera 101 installed in front of the target office. Here, the server 102 determines that the person shown in the image is a registrant. Then, the server 102 unlocks the gate of the target office when it is determined that the registrant has arrived before the gate of the target office.

  As described above, the monitoring system of the present embodiment can unlock the gate without causing the registrant to perform a conscious authentication operation. Also, in the monitoring system of the present embodiment, the more face images are accumulated, the more registrants appear. Then, since this monitoring system can improve the accuracy of authentication, safety and convenience are increased.

  Next, processing of the monitoring system for a person who visits for the first time (visitor) will be described. The explanation is given when the visitor goes from the gate of the facility to the target office.

  First, a visitor contacts an administrator at the facility gate. When the administrator allows the visitor to enter, the administrator unlocks the facility gate. In addition, the administrator inputs the area number of the target office to the server 102. At this time, the camera 101 installed at the facility gate captures a visitor and inputs the image to the server 102. The server 102 extracts a face image from the input image. Then, the extracted face image is stored in the person appearance data 108 together with the area number input by the administrator.

  Next, the visitor moves toward the destination office. At that time, the server 102 performs the same process as the process for the registrant described above. Accordingly, when the visitor arrives at the destination office without mistake in the route, the server 102 unlocks the gate of the destination office. In this case, the visitor can be unlocked without performing a conscious authentication operation.

  If the visitor makes a mistake in the route, the server 102 detects the departure of the route and notifies the administrator. Further, the server 102 may notify a visitor of a route error using a communication means. As a result, the server 102 can prevent the visitor from leaving the route.

  Next, processing of the monitoring system for a registrant who moves from his / her office to another place in the facility will be described. For example, a registrant moves from his office to a shared corner for refresh. In the conventional monitoring system, it is necessary to perform an authentication operation every time the user enters or leaves the office. Therefore, the conventional monitoring system has lost convenience.

  When the registrant leaves the office, the camera 101 installed at the office gate captures the registrant and inputs the captured image to the server 102. The server 102 extracts a face image from the input image and stores the extracted face image in the person appearance data 108. When a face image similar to the face image stored in the person appearance data 108 is input from the camera 101 installed in front of the office gate within a predetermined time, the server 102 instructs the unlocking of the gate. This eliminates the need for the registrant to perform an authentication operation every time he enters or leaves the office.

  Next, a process when a registrant set with an entry prohibited area enters the entry prohibited area will be described.

  When the server 102 determines that the corresponding person of the image input from the camera 101 installed in the entry prohibited area is the registrant, the server 102 determines that the person has entered the entry prohibited area. When the server 102 determines the entry into the entry prohibited area, the server 102 notifies the administrator. Further, the server 102 may notify the registrant of entry prohibition using communication means. Further, the server 102 may determine the frequency of entry of the registrant into the entry prohibited area by referring to the person appearance data 108. When the frequency of entry is high, the server 102 notifies the administrator of the registrant as a person requiring attention. Furthermore, since the number of face images stored in the person appearance data 108 increases for the registrant, the server 102 improves the accuracy of determination for the registrant.

  Through all the above processes, the monitoring system of the present embodiment can monitor the facility without making the authentication operation conscious of all persons.

  Next, a case where the monitoring system of the present embodiment is applied to a visitor notification service in a residential area will be described.

  In recent years, from the viewpoint of safety, there is an increasing number of blocks with a structure that restricts visitors by providing a gate at the entrance of the block. Such a block has a gate at the entrance of the city, and each house also has a gate. Therefore, except for the difference between indoor and outdoor, the monitoring system performs the same processing as that applied to ensuring the security in the facility described above.

  Note that the monitoring system may notify the visited person of the name of the person corresponding to the image taken near the entrance of the block. In addition, this monitoring system may transmit an image taken near the entrance of a block to a visited place. In addition, when a visitor makes a mistake in the route, the monitoring system may notify the visitor that the route is off.

  As described above, the monitoring system applied to the visitor notification service can secure the security of the block.

(Third embodiment)
In the third embodiment, the image is corrected in accordance with the lighting conditions.

  FIG. 8 is a block diagram of a configuration of a monitoring system according to the third embodiment of this invention.

  The monitoring system according to the third embodiment is the same as the monitoring system according to the first embodiment (FIG. 1) except that the lighting condition data 110 is provided and the configuration included in the server 102. The same number is attached | subjected to the same structure and description is abbreviate | omitted.

  The illumination condition data 110, which will be described later with reference to FIG. Illumination includes external light from the sun and internal light from indoor lighting.

  The server 102 includes a lighting / weather condition estimating unit 111 in addition to the configuration included in the server 102 (FIG. 1) of the first embodiment.

  The illumination / weather condition estimation unit 111 corrects the face image extracted by the face identification unit 106 based on the illumination condition data 110.

  FIG. 9 is a configuration diagram of the illumination condition data 110 according to the third embodiment of this invention.

  The illumination condition data 110 includes an area number 1101, a shooting situation 1102, a corresponding face image data number 1103, an illumination mode 1104, a feature amount correction coefficient 1105, and preprocessing content 1106.

  The area number 1101 is an identifier that uniquely identifies the place where the camera 101 is installed. The shooting situation 1102 is a situation when an image is taken by the camera 101 installed in the area. In this explanatory diagram, the shooting situation 1102 stores the orientation of the person shown in the image, the time zone when the image was shot, the weather when the image was shot, and the like.

  The corresponding face image data number 1103 is an identifier for uniquely identifying an image shot in the shooting situation. If there are a plurality of corresponding images, the corresponding face image data number 1103 stores a plurality of identifiers.

  The illumination mode 1104 is an external light situation in which an image taken under the shooting situation is affected. In this explanatory diagram, the illumination mode 1104 stores the type of external light and the intensity of external light. The type of outside light is, for example, “forward light”, “backlight”, “right oblique light”, “left oblique light”, “indoor”, or the like. When the type of outside light is “indoor”, this indicates that the image is not affected by outside light. In addition, the intensity of external light is, for example, “strong”, “medium”, “weak”, or the like.

  The feature amount correction coefficient 1105 is a coefficient for correcting the image feature amount of an image shot in the shooting situation. The server 102 corrects the image feature amount using the feature amount correction coefficient 1105, thereby removing the influence of external light. The feature amount correction coefficient 1105 is determined in advance so that the luminance distribution and color distribution of the face images taken by the plurality of cameras 101 are the same.

  For example, when the weather is fine and the time zone is daytime, the feature amount correction coefficient 1105 increases. Conversely, when the weather is cloudy or the time zone is nighttime, the feature amount correction coefficient 1105 is small. When the window is facing west and the time zone is evening, the feature amount correction coefficient 1105 increases the component for correcting backlight.

  The pre-processing content 1106 is the content of pre-processing for an image shot in the shooting situation. For example, the preprocessing content 1106 stores “light unevenness”, “luminance”, “backlight”, and the like. When the server 102 extracts the face image, the server 102 performs processing corresponding to the preprocessing content 1106. If the preprocessing content 1106 is “light unevenness”, the server 102 makes the brightness of the image uniform. If the preprocessing content 1106 is “luminance”, the server 102 darkens the entire image. If the preprocessing content 1106 is “backlight”, the server 102 brightens the entire image.

  Note that the illumination condition data 110 may store information related to indoor lighting. For example, the presence / absence of an indoor lamp is stored in the shooting situation 1102, and the feature amount correction coefficient 1105 is set to a value considering the indoor lamp. As a result, the server 102 can perform correction in consideration of indoor lighting.

  FIG. 10A is an explanatory diagram of the influence of illumination according to the third embodiment of this invention.

  Here, a case where three cameras 101C, 101D, and 101E are provided will be described.

  First, the illumination / weather condition estimation unit 111 acquires the weather and time when the image is taken. The illumination / weather condition estimation unit 111 records a record in which the area number of the camera 101 that captured the image matches the area number 1101 of the illumination condition data 110, and the acquired weather, time, and the like match the shooting condition 1102. select. Next, the illumination mode 1104 is extracted from the selected record. The extracted illumination mode 1104 is a lighting situation in which the image is affected.

  For example, in this explanatory diagram, the weather is fine and the time is 9 o'clock.

  In this case, the camera 101C is affected by strong right oblique light. The camera 101D is also affected by strong backlight. The camera 101E is installed in a room that is not affected by outside light.

  FIG. 10B is an explanatory diagram of the influence of external light according to the third embodiment of this invention.

  In this illustration, the weather is cloudy and the time is 17:00.

  In this case, the camera 101C is affected by weak follow light. The camera 101D is also affected by weak left oblique light. The camera 101E is installed in a room that is not affected by outside light.

  FIG. 11 is a flowchart of image processing of the server 102 according to the third embodiment of this invention.

  When an image captured by the camera 101 is input from the camera 101, the server 102 starts processing (401). At this time, the server 102 may be input with the area number of the place where the image was taken, the time when the image was taken, and the like.

  First, an image of a face area (face image) is extracted from the input image (402).

  Next, the shooting situation of the input image is specified (403). The shooting situation includes the time zone when the image was shot, the weather when the image was shot, and the like. Specifically, the weather when the image is taken is specified based on weather information from an external network, luminance information from an external light sensor, or input information from an administrator. Also, based on the time input from the camera 101, the time zone when the image was taken is specified.

  Next, a record in which the area number input from the camera 101 matches the area number of the lighting condition data 110 and the specified shooting situation matches the shooting situation 1102 of the lighting condition data 110 is read from the lighting condition data 110. select. Next, the feature amount correction coefficient 1105 and the preprocessing content 1106 are extracted from the selected record.

  Next, preprocessing corresponding to the extracted preprocessing content 1106 is performed on the face image. Next, an image feature amount is calculated from the preprocessed face image (404).

  Next, the calculated image feature quantity is corrected using the extracted feature quantity correction coefficient 1105 (405). Then, person determination processing is performed (406). The person determination process 406 is the same as steps 204 to 210 in the image processing (FIG. 4) of the first embodiment, and a description thereof will be omitted.

  As described above, it is possible to improve the accuracy of the corresponding person by performing correction on the face image according to the lighting condition in which the image is captured.

  Next, the case where the monitoring system of this Embodiment is used for security ensuring in a city area etc. is demonstrated.

  In a monitoring system in an urban area, a plurality of cameras 101 are installed as in the monitoring system in a facility. The monitoring system stores in advance the relative position of the installed camera 101, the orientation of the camera 101, and the like. For example, the cameras 101 are installed on crime prevention pillars that exist at regular intervals along a street. The camera 101 may use a surveillance camera already installed in the downtown area.

  First, a process for searching for a specific person will be described. The specific person is, for example, a wanted crime or a lost child.

  First, the monitoring system extracts a face image from an image photographed by the camera 101. Next, the extracted face image is corrected based on the illumination condition data 110. Next, the corrected face image is tracked. Then, all face images that can be tracked are determined as face images of the same person.

  If the monitoring system cannot track the person, the person corresponding to the captured face image is determined. Then, the monitoring system accumulates the extracted face image and information related to the face image for each person.

  Next, the monitoring system calculates the matching degree between the accumulated face image and the face image of the specific person for all the accumulated face images. The monitoring system notifies the administrator when there are a large number of calculated collation degrees equal to or greater than the threshold.

  Since the monitoring system of the present embodiment accumulates a large number of face images in various environments, it is possible to improve the accuracy of identifying a person. Further, even if the degree of matching with a part of the stored face images is high, no notification is made if the degree of matching with other stored face images is low, so that false alarms can be reduced.

  The monitoring system may detect the abnormal behavior of the person together with the search process for the specific person. For example, the monitoring system detects an abnormal behavior of a person using an existing image processing technique. Abnormal behavior is when a person stays at the same position for a long time. When the monitoring system detects abnormal behavior, it notifies the administrator. In addition, the monitoring system may accumulate face images of persons who have performed abnormal behavior and specify the person of the accumulated face images.

  Next, the monitoring system of the present embodiment can also be used for congestion detection.

  By adopting face identification by the monitoring system of this embodiment, it is possible to know the relationship of the congestion situation at a plurality of points. For example, the monitoring system determines that 80% of people appearing at point A appear at point B. In this case, the administrator can understand that there is a high correlation between the person visiting the point A and the person visiting the point B.

  In the monitoring system according to the present embodiment, it becomes difficult to identify a person as the number of persons increases. However, since this monitoring system only needs to know the relevance of persons at a plurality of points, the monitoring system can be operated even if the accuracy of identification of persons is low.

  The following are typical examples of aspects of the present invention other than those described in the claims.

In a monitoring device in which a captured image is input from a camera and acquires information from a storage unit that stores various types of information,
The storage unit stores past information including information related to images previously input to the monitoring device, and geographic information including position information of the camera,
A face image is extracted from the input image, and a moving direction and a moving speed of a person corresponding to the face image are obtained from the extracted face image;
Based on the obtained moving direction and moving speed, it is determined whether or not the extracted face image can be tracked,
If tracking is possible, the person of the tracked face image is determined as the person corresponding to the extracted face image;
If tracking is impossible, based on the past information, the person corresponding to the extracted face image is estimated,
A monitoring device that determines a person corresponding to the extracted face image from the estimated persons based on the geographic information.

  According to this, the monitoring apparatus can determine the corresponding person of the face image by the tracking process of the face image. Furthermore, since the monitoring apparatus performs image processing for determining the person corresponding to the face image only when the face image cannot be tracked, the processing can be reduced.

  Since the present invention tracks a person in an image taken by a camera, it can be widely applied to indoor and outdoor monitoring devices. In addition, since the present invention also has a function of an authentication device, it can be applied as an alternative to a conventional authentication device.

It is a block diagram of the monitoring system of a 1st embodiment of the present invention. It is a block diagram of the person appearance data of embodiment of this invention. It is a block diagram of the geographical data of embodiment of this invention. It is a flowchart of the image processing of the server of the 1st Embodiment of this invention. It is a block diagram of a structure of the monitoring system of the 2nd Embodiment of this invention. It is explanatory drawing of the human flow tracking process of the server of the 2nd Embodiment of this invention. It is a flowchart of the image processing of the server of the 2nd Embodiment of this invention. It is a block diagram of a structure of the monitoring system of the 3rd Embodiment of this invention. It is a block diagram of the illumination condition data of the 3rd Embodiment of this invention. It is explanatory drawing of the influence of the illumination of the 3rd Embodiment of this invention. It is explanatory drawing of the influence of the illumination of the 3rd Embodiment of this invention. It is a flowchart of the image processing of the server of the 3rd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Camera 102 Server 103 Entrance / exit control device 104 Person detection / tracking unit 105 Geography / time limit calculation unit 106 Face identification unit 107 Entrance / exit management unit 108 Person appearance data 109 Geographic data 110 Illumination condition data 111 Illumination / weather information estimation unit 112 network

Claims (9)

In a monitoring device in which a captured image is input from a camera and acquires information from a storage unit that stores various types of information,
The storage unit stores past information including information related to images captured by the camera in the past, and geographic information including area information captured by the camera,
A face identifying unit that extracts a face image from the input image and estimates a person corresponding to the extracted face image based on the past information;
A monitoring apparatus comprising: a determination unit that determines a person corresponding to the extracted face image from the persons estimated by the face identification unit based on the geographic information. The past information includes a feature amount indicating a feature of a face image extracted in the past by the monitoring device,
The face identification unit
From the extracted face image, a feature amount indicating the feature of the face image is obtained,
The monitoring apparatus according to claim 1, wherein a person corresponding to the extracted face image is estimated by comparing a feature amount included in the past information with the obtained feature amount. The geographic information includes a time for a person to move in an area captured by the camera,
The past information includes a time when the face image extracted in the past by the monitoring device is captured,
The determination unit
Search the past information for a face image in which the estimated person was photographed immediately before,
The time when the searched face image was taken is acquired from the past information,
Subtract the acquired time from the time when the extracted face image was taken,
The person corresponding to the extracted face image is determined from the persons estimated by the face identification unit by comparing the subtracted time difference and the travel time included in the geographic information. The monitoring device according to claim 1.   A tracking unit that obtains a movement direction and a movement speed of a person corresponding to the face image from the extracted face image and tracks the extracted face image based on the obtained movement direction and movement speed is provided. The monitoring apparatus according to claim 1. The storage unit stores illumination information including a lighting situation when the face image is captured,
A correction unit that corrects the extracted face image based on the illumination information;
The monitoring apparatus according to claim 1, wherein the face identification unit estimates a person corresponding to the extracted face image based on the face image corrected by the correction unit. The determination unit
When determining a person corresponding to the extracted face image, a reliability indicating the possibility that the person to be determined and the person corresponding to the face image are the same is obtained;
The monitoring device according to claim 1, wherein the obtained reliability is stored in the past information. In a monitoring system for monitoring facilities, comprising one or more cameras, a storage device for storing various types of information, and a monitoring server in which captured images are input from the camera and obtain information from the storage device,
The storage device stores past information including information on images captured by the camera in the past, and geographic information including region information captured by the camera,
The monitoring server is
A face identifying unit that extracts a face image from the input image and estimates a person corresponding to the extracted face image based on the past information;
A monitoring system comprising: a determination unit that determines a person corresponding to the extracted face image from the estimated person based on the geographic information. The monitoring system includes an entrance / exit control device that controls opening / closing of the entrance / exit of the facility,
The storage device stores authentication information including information on whether or not a person is permitted to enter the facility,
The determination unit determines whether the determined person is allowed to enter based on the authentication information,
The monitoring system according to claim 7, wherein the entrance / exit control device controls to open an entrance / exit of a facility when the determination unit determines that the person is permitted to enter. The monitoring server is
From the extracted face image, a movement unit and a movement speed of a person corresponding to the image are obtained, and a tracking unit that tracks the extracted face image based on the obtained movement direction and movement speed is provided.
The monitoring system according to claim 7, wherein the determination unit determines a person of the tracked face image as a person corresponding to the extracted face image.

Images