JP2009194711A - Region user management system and management method of the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a region user management system for managing the entrance and exit to/from a region by using images which can prevent somebody else from taking away children, and to provide a management method of the same. <P>SOLUTION: The region user management system determines, from faces included in a shot image photographed by an image input section 12, whether the faces are the face of a person of note. When the faces included in the shot image are determined not to be the face of the person of note, the region user management system estimates an accompanying person of the person of note from the faces determined not to be the face of the person of note. The accompanying person of the person of note is estimated by an accompanying person estimating means from a plurality of images photographed at different points of time, and it is determined whether there is discrepancy between the accompanying persons in a plurality of images photographed at different points of time, and the result is outputted to a result output section 26. When it is determined that there is discrepancy between candidates of the accompanying person, notice is provided by means of an output means such as an alarm and a warning light. Thus, without troubling users, criminal events such as the taking-away of children by somebody else can be prevented. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an area user management system, and more particularly to an area user management system that monitors entry / exit with image data and prevents children from being taken away in real time, and a management method thereof.

In recent years, crimes such as child abduction and removal have increased. Particularly in large shopping centers, children are often lost and become alone, and there are increasing cases of such children being targeted. In order to prevent such an incident, for example, Japanese Patent Laid-Open No. 2005-234765 (Patent Document 1) describes an image of a companion and a child taken together by a surveillance camera when the child is lost. And a system for searching the current location of a child with a surveillance camera based on the image is disclosed. JP-A-2007-60528 (Patent Document 2) describes that, when entering a region, each face image of a child and a companion is associated with each other and registered in a database, and from surveillance cameras installed at various locations in the region. A system is disclosed that identifies a face image from an obtained image and notifies the accompanying person that the child is lost when the child and the accompanying person are separated for a certain period of time.
JP 2005-234765 A Japanese Unexamined Patent Publication No. 2007-60528

  However, in such a conventional system, it is necessary to arrange security guards, which is expensive, and even if the security guard finds a child, it is difficult to distinguish between a companion and a kidnapper, or the child is lost. Until the companion noticed, the child had already been taken out of the area and could not be found. In addition, there is a method for registering the face images of children and accompanying persons in the database when entering the area, but registration of the relationship between the child and accompanying persons, shooting of the face images and registration work are required. There is a problem that it is necessary, and particularly in an area where there are a large number of users such as a large shopping center on holidays, the users are made to wait for registration work.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an area user management system and a management method therefor that eliminate such drawbacks of the prior art and prevent a child from being taken away by another person without bothering the user.

  According to the present invention, in order to solve the above-described problem, an area user management system that manages users of an area with image data determines whether or not the person is an attention person from the face included in the captured image. Attention person determination means, companion estimation means for estimating the accompanying person of the attention person from the face determined not to be the attention person, and the accompanying person estimation means from a plurality of images taken at different times in this area Including a companion person determination unit that estimates a person's companion and determines whether or not a disagreement has occurred in a plurality of images at different time points, and a result output unit that outputs a result of the companion person determination unit Features.

  In addition, the management method of the area user management system that manages the users of the area with the image data includes a person-of-interest determination process for determining whether or not the person included in the captured image is a person of interest, and is not a person of interest The companion estimation process for estimating the companion of the attention person from the face determined to be different from the companion estimation process in the companion estimation process from a plurality of images taken at different times in this area, and different. The method includes a companion person determination step for determining whether or not there is a discrepancy between companions in a plurality of images at the time, and a result output step for outputting a result of the companion person determination unit.

  Next, with reference to the accompanying drawings, an embodiment of the area user management system according to the present invention will be described in detail. First, the configuration of the embodiment will be described with reference to FIG.

  The area use management system 100 according to the first embodiment is disposed in a specific area including a facility such as a large shopping center, for example, and includes an image input unit 12, a face detection unit 14, an attention person determination unit 16, and a companion estimation. The unit 18, the face information registration unit 20, the attendee database 22, the accompanying person determination unit 24, and the result output unit 26 are connected and configured as illustrated.

  The image input unit 12 has a device that can acquire continuous images, such as a camera device, and is adjusted and set so that, for example, the face of the entrance / exit is completely photographed at the entrance / exit of an area such as a shopping center. The image signals 512 of all the visitors who have taken the images are sent to the face detection unit 14. Of course, when there are a plurality of entrances / exits, a plurality of entrances / exits are installed. However, since the functions are the same, the case where one unit is installed will be described here.

  The face detection unit 14 extracts a face region by a conventional technique from the image signal 512 of the entrance / exit sent from the image input unit 12, for example, the position of the face, the size of the face, the approximate distance from the camera 12, etc. The face area information 514 is sent to the attention person determination unit 16.

  The person-of-interest determination unit 16 performs further detailed analysis from the face area information 514 sent from the face detection unit 14 in comparison with a predetermined feature, determines whether or not the person should be noticed, and from the face detection unit 14 The noticed person information 516 including the sent face area information 514 is sent to the companion estimation unit 18.

  The companion estimation unit 18 calculates the distance between the face of the person photographed together with the person of interest from the person-of-interest information 516 sent from the person-of-interest determination unit 16, and the companion candidate based on the proximity of the face Is transmitted to the face information registration unit 20 and the accompanying person determination unit 24 as accompanying person candidate information 518 including the noticed person information 516.

  The face information registration unit 20 extracts facial features that can be distinguished from each other of the determined companion candidate candidates from the companion candidate information 518 sent from the companion estimation unit 18, and the identification number of the companion candidate candidates An identification number that uniquely identifies the person of interest is associated and sent to the visitor database 22 as visitor data 520.

  The visitor database 22 stores the visitor data 520 sent from the face information registration unit 20, and provides the identifiable information 522 to the accompanying person determination unit 24 when it is necessary to determine the accompanying person.

  A companion person determination unit 24 is a companion candidate information 518 of an exit person photographed by the image input unit 12 and sent via the face detection unit 14, the attention person determination unit 16, and the companion estimation unit 18, and a visitor It is determined from the identifiable information 522 stored in the database 22 whether or not the companion candidate at the time of leaving is the same as the admission companion candidate, and the determination result information 524 is used as the result output unit 26. Send to.

  The result output unit 26 outputs the determination result information 524 sent from the accompanying person determination unit 24 using, for example, output means such as voice or image. In particular, when it is determined that a companion candidate has been replaced at the time of admission, notification is given by an output means such as an alarm or a warning light.

  In addition to using a portable terminal that can display images and can be carried by security guards as an output means, it is not only a notification of alarms, but at the same time, images that are the basis of judgment such as images at the time of entry, and similar relatives It is also possible to send similarity information as to whether it is a face to the security guard so that it can be confirmed.

  Next, in the area user management system of the present embodiment, the operation for managing the entrance / exit will be described in detail with reference to the operation flowcharts of FIGS. 2 and 7 and the image diagrams of FIGS. Here, for easy understanding, the management operation at the time of entrance and the management operation at the time of exit will be described separately.

  First, an operation for managing visitors to the area will be described. The image signal 512 of the visitor continuously captured by the image input unit 12 set at the entrance of the area, for example, a camera device (step S102) is transmitted to the face detection unit 14.

  In the face detection unit 14, the image signal 512 sent from the image input unit 12 detects the face area using a conventional technique for detecting the face area (S104). The face area includes face area information 514 such as the position of the face in the image, the face size, and the approximate distance from the camera.

  For example, as shown in FIG. 3, the upper left corner of the image is the origin and the upper left corner and the lower right corner of the face area are indicated by coordinates (X1, Y1) and (X2, Y2), respectively. The position is ((X1 + X2) / 2, (Y1 + Y2) / 2), and the face size can be found from the difference between the two points. The distance is also estimated.

  The face area information 514 sent from the face detection unit 14 is further analyzed in detail by the person-of-interest determination unit 16, and it is determined whether or not it is a person to be noted (S106). The person to be noted here is a person having a predetermined characteristic.

  For example, in the case of a child that may be taken away, the predetermined characteristics are as shown in FIG. 5, in which the child has a shorter distance between eyes and mouth in the face than an adult and is shorter in height. Therefore, the distance from the ground to the face is short, and the feature such as a combination thereof is indicated.

  If it is determined that the person is a person of interest, the person-of-interest information 516 is sent to the accompanying person estimation unit 18. If it is determined that the person is not the person of interest, the process returns to the face detection unit 14, the person of interest is determined for the next person, and the process is repeated until all the faces in the image are determined (S116).

  The companion estimation unit 18 calculates the distance between the target person determined from the target person information 516 sent from the target person determination unit 16 and one person other than the target person shown in the same video (S108). .

  As shown in FIG. 4, as a characteristic of the image, a face close to the camera becomes larger on the image, and becomes smaller as the distance increases. If there are multiple people with different distances from the camera, the distance between the face areas in the image may be very different from the actual distance. It can happen.

  In such a case, the actual distance is estimated by adding the size of the face area as an estimation condition as follows.

Assuming that the two face areas for which the distance is obtained are A and B, as indicated by the face detection unit 14, the position of each face area is
(Ax, Ay) = ((Xa1 + Xa2) / 2, (Ya1 + Ya2) / 2), (Bx, By) = ((Xb1 + Xb2) / 2, (Yb1 + Yb2) / 2).

Also, the size of each face area on the image
(Rxa, Rya) = ((Xa2-Xa1) / 2, (Ya2-Ya1) / 2), (Rxb, Ryb) = ((Xb2-Xb1) / 2, (Yb2-Yb1) / 2) The actual distance AB between the face areas A and B is
AB = ((Ax−Bx) ² + (Ay &#8211; By) ² + α (Rxa−Rxb) ² ) ^1/2 Here, (Xa1, Ya1) and (Xa2, Ya2) are the coordinates of the upper left and lower right corners of the face area A, and (Xb1, Yb1) and (Xb2, Yb2) are the upper left and lower right edges of the face area B. Indicates the coordinates. Α is a constant indicating a change in the image size due to the camera characteristics depending on the distance, and is determined experimentally.

  The calculated distance from the person of interest is compared with the set reference distance (S110), and if it is within the reference distance, the person is determined as a companion candidate, and the face information registration unit as companion candidate information 518 Send to 20. If the distance is equal to or greater than the reference distance, another person other than the person of interest is selected (S114), and the distance from the person of interest is similarly calculated and determined.

  The companion candidate information 518 sent from the companion determination unit 18 to the face information registration unit 20 includes, for example, as shown in FIG. 6, an identification number that makes the person of interest unique, and facial feature information that can be distinguished from other people of the person of interest. It is stored in the visitor database 22 as visitor data 520 in which the number of companion candidates, the companion identification number, facial feature information that can be distinguished from other companions is associated (S112). Further, in order to increase the accuracy of the determination, information such as the pattern and color of clothes and the figure of a person may be stored together as the visitor data 520 in association with each other.

  At this time, the visitor data 520 is registered after collation with the registered visitor database 22 in order to avoid duplicate registration.

  Here, as a method for extracting facial features that can be distinguished from others, a conventional technique such as Japanese Patent Application Laid-Open No. 2007-156768 may be used.

  Next, an operation for managing a person leaving the area will be described. The operations up to the image input unit 12, the face detection unit 14, the attention person determination unit 16, and the accompanying person determination unit 18 are the same as the management of the visitors, and will be described briefly.

  When the image signal 512 of the exit person continuously photographed by the image input unit 12 (S122) is transmitted to the face detection unit 14, a face area is detected (S124), and face area information 514 is extracted.

  Subsequently, the face area information 514 is sent to the attention person determination unit 16 to determine whether or not the person is the attention person (S126). If it is determined that the person is a person of interest, the person-of-interest information 516 is sent to the accompanying person estimation unit 18. If it is determined that the person is not a person of interest, the process returns to the face detection unit 14 and is repeated until all the faces in the image are determined (S142).

  The companion estimation unit 18 calculates the distance between the target person determined from the target person information 516 sent from the target person determination unit 16 and one person other than the target person appearing in the same video (S128). Then, it is compared with the reference distance (S130), all persons other than the person of interest are confirmed (S140), and a companion candidate is determined.

  The result determined by the companion estimation unit 18 is sent to the companion person determination unit 24 as companion candidate information 518.

  In the companion person determination unit 24, the companion candidate information 518 sent from the companion estimation unit 18 is compared with the identifiable information 522 from the visitor database 22, for example, a face having a certain degree of matching or higher. By obtaining the feature identification number, the person of interest is identified (S132).

  When the person of interest is identified, the characteristics of the companion candidate in the companion candidate information 518 are compared with the characteristics of the companion candidate in the identifiable information 522 from the visitor database 22 (S134), and all the candidates are compared. Even so (S136), if there is no person who matches the companion candidate at the time of admission, it is determined that the companion person at the time of admission may have been switched.

  When the accompanying person determination unit 24 determines that there is a possibility that the accompanying person has been replaced, the determination result information 524 is sent to the result output unit 26. Directives such as protection are notified.

  Next, another embodiment of the area user management system according to the present invention, that is, the configuration of the second embodiment will be described with reference to FIG. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and repeated description is avoided.

  The area use management system 200 according to the second embodiment expands the range for estimating the companion candidate before and after the passing time of the person of interest, and as in the first embodiment, the image input unit 12, the face detection unit, 14, a person-of-interest determination unit 16, a companion estimation unit 18, a face information registration unit 20, an attendee database 22, a companion determination unit 24, a result output unit 26, and a new passer-by database 28.

  The passer-by database 28 receives the face area information 514 from the face detection unit 14, temporarily stores it, and detects the target person determined by the target person determination unit 16 at a certain time before and after the target person. The passed face area information of the passer is sent to the companion estimation unit 18 as front and rear passer face area information 515.

  The companion estimation unit 18 determines a companion candidate for the face area included in the front and rear passer face area information 515 sent from the passer database 28.

  Next, the operation of the second embodiment will be described with reference to the operation flow diagram of FIG. 9 and the image diagram of FIG. The same operation as that of the first embodiment will be described in a simplified manner, and detailed repetitive description will be avoided.

  First, an operation for managing visitors to the area will be described.

  As in the first embodiment, first, an image signal 512 photographed by the image input unit 12 (S202) is sent to the face detection unit 14, and a face area is detected by the same method as in the first embodiment (S204). .

  For example, as shown in FIG. 10, the face area information 514 detected by the face detection unit 14 is temporarily stored in the passer-by database 28 together with the image acquisition time information (S206).

  Further, the face area information 514 is determined by the attention person determination unit 16 by the same method as in the first embodiment, for example, whether or not the person is a person of interest such as a child (S208).

  If it is determined that the person is a person of interest, front / rear passer face area information 515 is sent from the passer-by database 28 to the companion estimation unit 18, and all persons who pass the person of interest within a certain period of time before and after the person of interest (S210, S212, S218). About fixed time, it can adjust by the situation at that time, such as the number of passing persons.

  Furthermore, it is possible to narrow down the companion candidates based on the condition of the companion like that the relatives have similar facial features (S214).

  Similarly, the process is repeated until determination is made for all the faces in the front and rear passer face area information 515 (S220).

  The operations to manage the exit from the area are the same operations as the management of the visitors, the operations up to the image input unit 12, the face detection unit 14, the attention person determination unit 16, and the accompanying person determination unit 18. The operations of the determination unit 24 and the result output unit 26 are the same as those in the first embodiment, and are omitted.

  Next, still another embodiment of the area user management system according to the present invention, that is, the configuration of the third embodiment will be described with reference to FIG. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and repeated description is avoided.

  The area use management system 300 according to the third embodiment captures moving images within the area and further narrows down the candidates for accompanying persons obtained from the image at the time of entrance. As in the first embodiment, the image input unit 12, the face detection unit 14, the attention person determination unit 16, the accompanying person estimation unit 18, the face information registration unit 20, the attendee database 22, the accompanying person determination unit 24, and the result output unit 26 and a companion candidate screening unit (screening unit) 30 are newly included.

  Similar to the first embodiment, the image input unit 12 includes a device such as a camera device that can acquire continuous images. For example, an entrance / exit of an area such as a shopping center, and an entrance / exit at any place in the area, and The image signal 512 of the entered / exited person and the user in the area is sent to the face detection unit 14.

  The companion candidate selection unit 30 displays the target person information 516 when the target person determination unit 16 detects the target person from the face area information 514 detected by the image signal 512 sent from the image input unit 12 in the area. A comparison is made with the visitor data 520 obtained from the attention person determination unit 16 and stored in the visitor database 22, and the matched result is sent to the visitor database 22 as companion candidate confirmation information 521.

  The attendee database 22 receives the companion candidate confirmation information 521 from the companion candidate selection unit 30, narrows down the companion candidates in the attendee data 520, and updates the attendee data 520.

  Next, the operation of the third embodiment will be described with reference to the operation flow diagram of FIG. 12 and the image diagram of FIG. The same operation as that in the first embodiment will be described in a simplified manner, and detailed repetitive description will be avoided.

  The operation for managing the visitors to the area is the same as that in the first embodiment, and the visitor data 520 is stored in the visitor database 22.

  Here, a user in the area is photographed by an image input unit 12 installed at various points in the area, for example, a camera (S322), and an image signal 512 is sent to the face detection unit 14, which is the same as in the first embodiment. A face region is detected by the method (S324).

  The face area information 514 detected by the face detection unit 14 is sent to the attention person determination unit 16, and it is determined by the same method as in the first embodiment whether or not the person is an attention person such as a child (S326). .

  If it is determined that the person is a person of interest, the person is compared with the visitor data 520 stored in the visitor database 22 to identify the person of interest (S328).

  When the person of interest is specified, the facial features of the surrounding people are compared with the facial features of the companion candidates stored in the visitor database 22, and it is determined whether they match (S330).

  For the matched person, the number of matches is recorded in the visitor data 520 (S332), and is updated in the visitor database 22 as shown in FIG. 13, for example.

  These operations (S326 to S332) are performed every time a person to be noticed is found for the image signal 512 from the cameras installed at various locations in the region (S334), and the number of matching candidates for the companion exceeds the threshold. A companion candidate is estimated from the characteristics of the person using the method of calculating the similarity as in the second embodiment, and is left as a companion candidate (S336).

  The person information below the threshold is deleted from the visitor database 22, and the entrance data 520 is sequentially updated.

  The operation for managing the exit from the area is the same as that in the first embodiment, and the description thereof will be omitted.

  Although various embodiments of the present invention have been described above, these have their respective effects. According to the first embodiment, it is automatically determined whether the companion at the time of leaving is the same as the companion at the time of entering using the facial features of the companion registered at the time of entry, and the result is By promptly notifying a guard or the like, it is possible to prevent the child from being taken out of the area.

  In addition, registration at the time of admission can be done automatically using an existing surveillance camera, which is not only convenient, but also requires customers to queue up and stop at crowded holiday shopping centers. In addition, safety can be provided without loss of time.

  According to the second embodiment, in addition to the effects of the first embodiment, a companion candidate can be determined even when a child leaves the parent's hand and enters, and eyes, mouth, etc. By adding the determination based on the similarity, it is possible to narrow down the companion candidates, and it is possible to suppress an increase in the storage capacity of the system.

  Further, by notifying the guards of the image of the companion candidate and the degree of similarity, it is possible to reliably prevent the child from being taken out of the area.

  According to the third embodiment, in addition to the effects of the first and second embodiments, the reliability is improved from a large number of companion candidates registered at the time of entrance, and the number of companion candidates is narrowed down. This makes it possible to specify a companion candidate with high accuracy.

  As described above, in the present embodiment, the prevention of the removal of the child has been described, but it is needless to say that it can be used not only for the child but also for an elderly person, a waiting person, and other specific persons.

It is a block diagram showing the configuration of an embodiment of an area user management system according to the present invention. It is an operation | movement flowchart which shows a part of operation | movement flow of the Example of the area | region user management system by this invention. It is an image figure which shows operation | movement description of the Example of the area | region user management system by this invention. It is an image figure which shows operation | movement description of the Example of the area | region user management system by this invention. It is an image figure which shows operation | movement description of the Example of the area | region user management system by this invention. It is an image figure which shows a part of Example of the area | region user management system by this invention. It is an operation | movement flowchart which shows a part of operation | movement flow of the Example of the area | region user management system by this invention. It is a block diagram which shows the structure of the 2nd Example of the area | region user management system by this invention. It is an operation | movement flowchart which shows a part of operation | movement flow of the 2nd Example of the area | region user management system by this invention. It is an image figure which shows a part of 2nd Example of the area | region user management system by this invention. It is a block diagram which shows the structure of the 3rd Example of the area | region user management system by this invention. It is an operation | movement flowchart which shows a part of operation | movement flow of the 3rd Example of the area | region user management system by this invention. It is an image figure which shows a part of 3rd Example of the area | region user management system by this invention.

Explanation of symbols

100,200,300 area user management system
12 Image input section
14 Face detector
16 Attention person judgment part
18 Companion Estimator
20 Face information registration section
22 Visitor database
24 Accompanying Person Judgment Department
26 Result output section
28 Transit database
30 Companion candidate screening department

Claims (16)

In an area user management system for managing area users with image data, the system includes:
Attention person determination means for determining whether or not the person is an attention person from the face included in the captured image,
Companion estimation means for estimating a companion of the attention person from a face determined not to be the attention person;
The accompanying person estimation means estimates the accompanying person of the person of interest from a plurality of images taken at different times in the region, and determines whether the accompanying person has a mismatch in the plurality of images at the different times. Companion person determination means,
A region user management system comprising: a result output means for outputting the result of the accompanying person determination means.   The area user management system according to claim 1, wherein the accompanying person estimation unit estimates a companion candidate using the distance of the face area to the person of interest in the captured image. .   2. The system according to claim 1, wherein the companion person estimation means estimates a companion candidate using a difference in size of a face area with the person of interest in the captured image. Management system.   2. The system according to claim 1, wherein the companion person estimation unit estimates a companion candidate using a combination of a distance and a size difference of a face area with the attention person in the captured image. A feature area user management system.   2. The system according to claim 1, wherein the accompanying person estimation unit estimates the companion candidate from a person existing in the photographed image of the person of interest and photographed images before and after the image. A feature area user management system.   6. The system according to claim 5, wherein the companion person estimation means includes the companion from a person similar to the person of interest existing in the photographed image of the person of interest and the photographed images of the images before and after the image. An area user management system characterized by estimating candidates.   2. The system according to claim 1, wherein the companion person estimation unit uses the images taken at a time other than the different time points in the region to determine whether the companion is the companion of the person of interest by the companion estimation unit. An area user management system characterized in that the candidate is estimated and the companion candidate is updated.   2. The area user management system according to claim 1, wherein the different time points are when entering the area and when leaving the area. In the management method of the area user management system that manages the users of the area with image data, the method includes:
A person of interest determination step for determining whether or not the person is a person of interest from the face included in the captured image;
A companion estimation step for estimating a companion of the attention person from a face determined not to be the attention person;
The accompanying person's accompanying person is estimated in the accompanying person estimation step from a plurality of images taken at different times in the region, and it is determined whether the accompanying person has a mismatch in the plurality of images at the different times. A companion person determination process,
A region user management system management method comprising: a result output step of outputting a result of the accompanying person determination means.   10. The area user management system according to claim 9, wherein the accompanying person estimation step estimates a companion candidate using a distance of a face area with the person of interest in the captured image. Management method.   10. The method according to claim 9, wherein the companion person estimation step estimates a companion candidate using a difference in size of a face area with the person of interest in the captured image. Management method of the user management system.   The method according to claim 9, wherein the companion person estimation step estimates a companion candidate using a combination of a distance and a size difference between the face area of the person of interest in the captured image. A management method for a feature area user management system.   The method according to claim 9, wherein the companion person estimation step estimates the companion candidate from a person existing in the photographed image of the person of interest and photographed images before and after the image. A management method for a feature area user management system.   13. The method according to claim 12, wherein the accompanying person estimation step includes the companion from a person similar to the person of interest existing in the photographed image of the person of interest and the photographed images of the images before and after the image. A method for managing an area user management system, characterized in that candidates are estimated.   The method according to claim 9, wherein the companion person estimation step uses the images taken at a time other than the different time points in the region, and the companion estimation step includes the companion person being an accompanying person of the attention person. A region user management system management method, wherein the candidate is updated and the companion candidate is updated.   10. The management method of the area user management system according to claim 9, wherein the different time points are when entering the area and when leaving the area.

Images