JP6854881B2 - Face image matching system and face image search system - Google Patents

Description

The present invention relates to a face image matching system that collates face images taken at a facility having an entrance and an entrance, and a face image search system that searches for a face image based on a designated search condition.

Traditionally, video surveillance systems have been installed in facilities such as hotels, buildings, convenience stores, financial institutions, dams and roads that are visited by an unspecified number of people for the purpose of crime prevention and accident prevention. This involves taking a picture of a person to be monitored with an imaging device such as a camera, transmitting the image to a monitoring center such as a management office or security room, and a resident observer monitors it for the purpose and necessity. Be careful or record the video accordingly.

In video surveillance systems, there are an increasing number of cases in which a random access medium typified by a hard disk drive (HDD) is used as a recording medium for recording video, instead of a conventional video tape medium. Furthermore, in recent years, the capacity of such recording media has been increasing.
Increasing the capacity of recording media has dramatically increased the amount of video that can be recorded, making it possible to record at multiple points and for a long time, but on the other hand, the burden of visually checking recorded images has become a problem. It's getting better.

Against this background, video surveillance systems equipped with a search function for more easily finding a desired video are becoming widespread. In particular, in recent years, a more advanced search function that automatically detects the occurrence of a specific event in a video in real time using image recognition technology, records it together with the video, and makes it possible to search for those events after the fact. A system equipped with is appearing. One of the typical ones is the person search function.
The person search function is a function that automatically detects the appearance of a person in a video, records it in real time, and can search for a person appearance image from the recorded images after the fact. From the functional aspect, the person search function is roughly classified into the following two types.

The first is the appearance event search function. The appearance event search function is a function that simply searches for the presence or absence of a person's appearance (event) in the video. As a search result, in addition to the presence or absence of an event, if it is determined that there is an event, the number of events, the time when each event occurs, the image pickup device number that captured the event, the captured image (person appearance image), etc. are presented. Will be done. In many cases, this search condition (search query) is given as information for narrowing down the search target range such as the event occurrence time and the imaging device number. The information for narrowing down the search target range is called a narrowing down parameter.

The second is a similar person search function. Whereas the above-mentioned appearance event search function is a search that does not specify a character, this is whether or not a specific person specified by the user is photographed by an image pickup device at another time or at another point. It is a function to search from recorded images. As the search result, in addition to the presence or absence of other images showing a specific person, if there is, the number and shooting time, the imaging device number, the shot image (person appearance image), the degree of similarity, etc. are presented. ..

The designation of a specific person is performed by the user designating one image (hereinafter, search key image) showing the person to be searched for. The search key image is specified from a recorded image or an arbitrary image in an external device. In the search, the image feature amount of the person in the search key image is extracted by the image recognition technology, collated with the image feature amount of the person in the recorded image, the similarity (similarity) is obtained, and the same person is determined. It is realized by doing. Extraction and recording of the person feature amount in the recorded image is performed in advance at another timing such as at the time of video recording. Even in this search condition, it is often possible to give a narrowing parameter.

Various inventions have been conventionally proposed for such a video surveillance system.
Patent Document 1 discloses an invention that enables the two-dimensional information such as time and similarity to be visually grasped at the same time in terms of magnitude and context.
By registering a plurality of images as search conditions in Patent Document 2, it is possible to reduce missed searches due to differences in the orientation and shooting angle of the target, and to obtain more images showing the target search person. The invention is disclosed.
Patent Document 3 discloses an invention in which a large number of images can be selected from the results of a similar image search and keywords can be assigned at once.

JP-A-2009-301501 Japanese Unexamined Patent Publication No. 2011-048668 Japanese Unexamined Patent Publication No. 2012-242878

Conventionally, attempts have been made to analyze how visitors who visit various facilities such as shopping centers and amusement parks behave in the facilities and utilize them in marketing. Until now, methods that rely on human power, such as the number of visitors assigned to various parts of the facility, have been mainly used, and improvements have been sought.
The present invention has been made in view of the above-mentioned conventional circumstances, and a first object of the present invention is to provide a face image matching system suitable for analyzing the behavior of visitors in a facility.

In addition, depending on the narrowing parameters, it may be desired to specify whether the image is displayed by one person or by multiple people. For example, in facilities such as amusement parks, which are used by most people, we are trying to prevent crimes in the park by carefully inspecting the baggage of those who intend to use the facility alone. This is because crimes such as deductions that occur in the park are mainly carried out alone. On the other hand, organized terrorism caused at airports and important facilities is often carried out by multiple people, so it is necessary to pay attention to those who act with multiple people. In the conventional face image search system, the photographed image can only be visually judged, which is a burden on the observer.
The present invention has been made in view of the above-mentioned conventional circumstances, and provides a face image search system capable of searching by designating whether the image is displayed by one person or by a plurality of people. The second purpose is to provide.

In the present invention, in order to achieve the first object, the face image matching system is configured as follows.
That is, in a face image matching system that collates face images taken at an entrance and a facility having an entrance, a first imaging device installed at the entrance and a first imaging device installed at the entrance and at the entrance are installed. A representative face image representing the person from among a plurality of face images determined to be the same person by the same person determination process based on the image captured by the second image pickup device and the first image pickup device. A selection means for selecting a face image, a registration means for registering a representative face image selected by the selection means in a database, and a representative for registering a face image included in an image captured by the second imaging device in the database. It is provided with a collation means for collating with a face image.

According to such a configuration, the representative face image obtained by the image pickup device installed at the entrance of the facility can be used to collate the face image obtained by the image pickup device installed in the facility and at the entrance. , It is effective for analyzing the behavior of visitors in the facility.

Here, as a configuration example, the selection means compares the plurality of face images with a pre-registered reference face, selects a representative face image of the person based on the degree of similarity with the reference face, and selects the representative face image of the person. As a reference face, a face image in which the eyes are opened facing the front may be used.

Further, as a configuration example, when the captured image used for the same person determination process includes a face image of a person different from the person, the registration means includes a group including the face image of the other person. When the attribute estimation data is associated with the representative face image of the person and registered in the database, and the face image matching system includes the face images of a plurality of people in the image captured by the second imaging device, the face image matching system is used. Based on the group attribute estimation data registered in the database, the configuration may further include a group attribute estimation means for estimating the group attribute indicating whether or not the plurality of persons are acting in a group.

Further, the group attribute estimation means determines the group attributes of the plurality of persons, the frequency with which the face images of each person are detected from the same captured image, the proximity indicating the distance between the face images of each person in the captured images, and the like. Estimated based on at least one of the intimacy indicating the relationship of the line of sight of each person in the captured image, and further based on at least one of the age or gender of each person estimated from the facial images of the plurality of persons. The configuration may be such that the relationship between the plurality of persons is estimated.

Further, the data obtained by the group attribute estimation means may be further provided with a totaling means for totaling based on at least one of the age or gender of each person, the date and time of entry or exit to the facility, and at least one of the group attributes. ..

In the present invention, in order to achieve the second object, the face image search system is configured as follows.
That is, in a face image search system that searches for a face image based on a specified search condition from a database that stores a face image included in the captured image by the imaging device, one person in the face image included in the captured image is alone. A determination means for determining whether or not an person is acting and storing the information of the determination result in the database in association with the face image of the person, and as the search condition, a search for a person who is acting alone or a plurality of people. When a search for a person who is acting in the above is specified, a search means for searching a face image associated with information of a determination result matching the designation from the database is provided.

Here, as a configuration example, when the same person is included in a plurality of images captured by different imaging devices, the determination means is based on the commonality of other persons included in the plurality of captured images. It may be configured to determine whether or not the same person is acting alone.

According to the present invention, it is possible to provide a face image matching system suitable for analyzing the behavior of a visitor in a facility. In addition, it is possible to provide a face image search system capable of specifying whether the image is displayed by one person or by a plurality of people.

It is a figure which shows the example of the schematic structure of the face image matching system which concerns on one Embodiment of this invention. It is a figure which shows the outline of the facility to which the face image matching system of FIG. 1 is applied. It is a figure which shows the example of the functional block of the collation server in the face image collation system of FIG. It is a figure which shows the operation example of the face image matching system of FIG. It is a figure which shows the data example of the processing result by the face image matching system of FIG. It is a figure which shows the processing flow of the number of close-up persons determination in the face image search system which concerns on another embodiment of this invention. It is a figure which shows the processing flow of the constant number of close persons determination in the face image search system which concerns on another embodiment of this invention.

The face image matching system according to the embodiment of the present invention will be described with reference to the drawings.
FIG. 1 shows an example of a schematic configuration of a face image matching system according to an embodiment of the present invention. Further, FIG. 2 shows an outline of the facility to which the face image matching system of FIG. 1 is applied. In the example of FIG. 2, there are stores A to G in the facility, and a person who enters the facility participates from the facility via store A, store F, store C, store G, and store E. Is shown.

The face image matching system of this example includes a plurality of N imaging devices 100 (1) to 100 (N), a matching server 200, a maintenance PC 300, a network switch 400, and a totaling server 500. ..
The image pickup devices 100 (1) to 100 (N) are installed at the entrance, each store in the facility, and the entrance. The image pickup device 100 (1) installed at the entrance is an image pickup device for face detection. Further, the image pickup devices 100 (2) to 100 (N-1) installed in each store in the facility and the image pickup device 100 (N) installed at the entrance are image pickup devices for face matching. Each imaging device 100 is composed of an IP camera (network camera) or the like, digitally converts a captured image obtained by imaging, and transmits the captured image to the collation server 200 via the network switch 400.

The collation server 200 accumulates captured images received from the image pickup apparatus 100 via the network switch 400, and performs various processes such as face detection, representative face selection, person attribute determination, face collation, and group attribute collation.
The maintenance PC 300 is connected to the collation server 200 via the network switch 400 and is used for maintenance of the collation server 200.
The aggregation server 500 is connected to the verification server 200 via the network switch 400 and the WAN line, receives output data transmitted from the verification server 200, and performs aggregation processing.

FIG. 3 shows an example of a functional block of the collation server 200.
The collation server 200 includes a face detection unit 201, the same person determination unit 202, a representative face selection unit 203, a format conversion unit 204, an attribute determination unit 205, an ID-assigned person information generation unit 206, a registration processing unit 207, a face database 208, and a face. Each functional unit of the collation unit 210, the group attribute collation unit 211, the collation history database 212, and the transfer processing unit 214 is provided.

The face detection unit 201 detects a face image from a captured image (hereinafter, referred to as “detection image”) obtained by the image pickup device 100 (1) at the entrance. Specifically, for example, a detection image is analyzed to detect a region in which a face is reflected, and image data in the region is extracted as a face image. The face image is detected for each consecutive image frame (or for each predetermined number of image frames) in chronological order. Therefore, a plurality of facial images can be obtained for each person passing through the entrance.

The same person determination unit 202 performs the same person determination process on the face image detected by the face detection unit 201. Specifically, for example, the movement of a person in the imaging area is tracked while comparing the position of the face image in the image frame between the image frames, and the face image is grouped for each person determined to be the same.

The representative face selection unit 203 selects a representative face image, which is a face image representing the person, from a plurality of face images determined by the same person determination unit 202 to be the same person. In this example, the representative face image is selected based on the similarity with the reference face registered in advance, instead of selecting the one having a large face detection size as the representative face image as in the conventional case. Specifically, a plurality of face images determined to be the same person are compared with the reference face, the degree of similarity with the reference face is calculated for each face image, and the one with the highest degree of similarity is the representative face image. Select as. A predetermined number may be selected as the representative face image in descending order of similarity.

As the reference face, a face image in which the eyes are opened facing the front is used. As a result, a face image that tends to have a high degree of matching similarity in subsequent face matching can be selected as the representative face image. In other words, it is possible to prevent a face image that can be expected to have a low matching similarity, such as a face image with closed eyes or a face image facing down or sideways, being selected as a representative face image. it can. In this example, four patterns of standard faces are used: male (wearing glasses), male (without glasses), female (wearing glasses), and female (without glasses), but considering the age group and hairstyle, etc. The number of reference face patterns may be increased.

The format conversion unit 204 converts the face image detected by the face detection unit 201 into a format suitable for the person attribute determination process in the attribute determination unit 205.

The attribute determination unit 205 performs a person attribute determination process for determining a person attribute of a person represented by the representative face image selected by the representative face selection unit 203 based on the face image converted by the format conversion unit 204. Examples of the person attribute include the age and gender of the person.

The ID-giving person information generation unit 206 issues a person ID associated with the representative face image selected by the representative face selection unit 203 and the person attribute determined by the attribute determination unit 205. The person ID is information that uniquely identifies each person who has visited the facility.

The registration processing unit 207 associates the person ID issued by the ID-giving person information generation unit 206, and registers the representative face image and the person attribute obtained for the person in the face database 208. The registration processing unit 207 further generates data for group attribute estimation used for group attribute estimation, associates it with a group ID, and registers it in the face database 208. The group ID is information that uniquely identifies each group of a plurality of people who act together, and is associated with the person ID.

The group attribute estimation data includes the face image of another person included in the detection image used for the same person determination process for the person to be registered (the person who registers the representative face image etc. in the face database 208). Is done. In addition, information for determining the detection frequency, proximity, and intimacy of the person to be registered and the other person is also included in the group attribute estimation data.

The face database 208 holds a representative face image, a person attribute (age and gender), and a group ID for each person identified by the person ID. In addition, the face database 208 holds group attribute estimation data for each group identified by the group ID.

The face matching unit 210 registers in the face database 208 the captured images (hereinafter referred to as “matching images”) obtained by the imaging devices 100 (2) to 100 (N) at each store and the entrance in the facility. It is collated with the representative face image. Face matching can be performed using, for example, a technique such as LFM (Live Face Matching). When the face matching unit 210 determines that the same person as the representative face image appears in the matching image, the face matching unit 210 registers the result in the matching history database 212. This makes it possible to identify how the person in the representative face image behaved in the facility (which store he / she visited). In addition, by registering the time information at which the collation image was taken in the collation history database 212, the time when the person with the representative face image entered each store in the facility can be specified. If imaging devices are provided at the entrance and exit of the store, the time when the store opens and the time spent in the store can be specified.

The group attribute collation unit 211 is the group attribute estimation data registered in the face database 208 for the collation images obtained by the imaging devices 100 (2) to 100 (N) at each store and the entrance in the facility. Group attribute estimation is performed using. That is, when a plurality of people are shown in the collation image (when a plurality of face images are included), a group attribute indicating whether or not these persons are acting in a group is estimated, and the results are collated. Register in the history database 212. Similarity determination techniques such as Enra-Enra are used for group attribute estimation.

As an example of the method for estimating the group attribute, the following method can be mentioned. (1) If the frequency at which each person's face image is detected from the same original image is higher than the reference value, it is judged that the group attribute = "Yes" (acting in a group), and if not, the group attribute. = Judge as "nothing" (acting alone). (2) If the proximity indicating the distance between the face images of each person in the original image is higher than the reference value, it is determined that the group attribute = "Yes", and if not, it is determined that the group attribute = "None". .. (3) If the intimacy indicating the relationship between the line of sight of each person (which person is looking) in the original image is higher than the reference value, it is judged that the group attribute = "yes", and if not, the group attribute = Judge as "none".
The "original image" may be a collation image, a detection image, or both. Further, the group attribute can be estimated by combining the methods (1) to (3).

The group attribute collation unit 211 further estimates the relationship between a plurality of persons determined to be acting in a group based on the analysis result of the facial image of each person. That is, based on the personal attributes (age and gender) of each person, for example, if the age difference is 20 years or older, the relationship between parents and children, if the age difference is 50 years or older, the relationship between grandparents and grandchildren, and the age difference is 10. If it is a man and a woman under the age, it is estimated that the relationship is a lover or a couple, and the result is registered in the collation history database 212.

The transfer processing unit 214 aggregates the data (process results by the face collation unit 210 and the group attribute collation unit 211) registered in the collation history database 212, and transfers the aggregated result data to the aggregation server 500 or the maintenance PC 300. Aggregation can be performed based on various conditions such as, for example, by age or gender of each person, by date and time of entry or exit to the facility, by group attribute, and by some combination thereof.

FIG. 4 shows an operation example of the face image matching system of this example.
The processing in the collation server 200 is a real-time (online) processing performed during the store opening period (9:00 to 21:00 in this example) and a store closing period (21:00 to 9:00 in this example). It is roughly divided into offline processing that is performed.

In the real-time processing, the following data collection processing is mainly performed.
When a person enters the facility, it detects that the person has entered the facility based on the detection image taken by the camera at the entrance. That is, a representative face image is selected from a plurality of face images obtained from the detection image and registered in the face database 208. At this time, information such as the detection date and time (the shooting date and time of the detection image), the person ID, the entrance place (information on the camera installation location), the person attribute 1 (age), and the person attribute 2 (gender) are also registered. In addition, preprocessing for group attribute estimation is performed based on the detection image. By this preprocessing, the detection date and time (the shooting date and time of the detection image), the group ID (common to multiple people included in the detection image), the entrance location (information on the camera installation location), the group attribute estimation data, etc. The information is registered in the face database 208.

After that, every time a person visits a store, it detects that the person has visited the store based on the collation image taken by the camera in the store. In addition, when a person participates from the facility, it is detected that the person has participated from the facility based on the collation image taken by the camera at the entrance. At these timings, based on the face image included in the collation image, collation with the representative face image registered in the face database 208, group attribute estimation based on the detection frequency, proximity, intimacy, etc. are performed. The result is recorded in the collation history database 212.

In the offline processing, the following aggregation / transfer processing and database update processing are mainly performed. These processes are automatically started when a pre-scheduled time arrives.
The aggregation process aggregates the data registered in the collation history database 212 to generate an output file (for example, CSV format data).
In the transfer process, the output file created in the aggregation process is transferred to the aggregation server 500.
The database update process updates the contents of each database (face database 208 and collation history database 212).

FIG. 5 shows a data example of the processing result by the face image matching system of this example.
In this data example, person 1 (female) and person 2 (male) are determined to be a group (group attribute = "yes"), and are determined to have a parent-child relationship based on their age differences. In addition, person 3 (female) and person 4 (male) are judged to be a group (group attribute = "yes"), and are judged to be friends or lovers based on their age differences. On the other hand, person 5 (female) is determined to be acting alone (group attribute = "none"). Further, in this data example, the time when each person (or group) entered the facility, the store visited and the time thereof, and the time when the person (or the group) entered the facility are also recorded.
In this way, the face image matching system of this example can not only analyze how each person behaves in the facility, but also determine whether or not they are acting in a group, and act in a group. If so, it can be seen that other viewpoints such as the behavior as a group can be analyzed.

As described above, the face image matching system of this example includes the face detection imaging device 100 (1) installed at the entrance and the face matching imaging device installed at each store and the entrance in the facility. A face representing the person from 100 (2) to 100 (N) and a plurality of face images determined to be the same person by the same person determination process based on the image taken by the image pickup device for face detection. A representative face selection unit 203 that selects a representative face image that is an image, a registration processing unit 207 that registers a representative face image selected by the representative face selection unit 203 in the face database 208, and an image taken by an imaging device for face matching. The face matching unit 210 for collating the face image included in the face image with the representative face image registered in the face database 208 is provided.
This makes it possible to analyze how visitors are behaving in the facility without the need for personnel assigned to various parts of the facility to count visitors as in the past. You will be able to carry out marketing that makes the most of. Moreover, since the person attributes (age and gender) of the visitors can be automatically identified from the face image and registered in association with the representative face image, it is necessary for the personnel at each facility to count while distinguishing the person attributes of the visitors. There is no.

Further, in the face image matching system of this example, the representative face selection unit 203 compares a plurality of face images with a pre-registered reference face and selects a representative face image of a person based on the degree of similarity with the reference face. At this time, as a reference face, a face image with the eyes facing forward is used.
As a result, the face image that tends to have a high degree of matching similarity by the face matching unit 210 can be selected as the representative face image, so that face matching can be performed with high accuracy.

Further, in the face image matching system of this example, when the registration processing unit 207 includes a face image of a person different from the person to be registered in the captured image used for the same person determination processing, the face image matching system of this example is used for another person. The group attribute estimation data including the face image is associated with the representative face image of the person to be registered and registered in the face database 208. Further, the images taken by the image pickup device for face matching include a plurality of people. When a face image is included, a group attribute matching unit 211 that estimates a group attribute indicating whether or not these a plurality of persons are acting in a group based on the group attribute estimation data registered in the face database 208 is provided. Further prepared.
This allows visitors who enter the facility together to be registered in advance as possible groups, and how they actually act as a group based on the commonality of subsequent actions. You will be able to know if you are doing it. As a result, it is possible to analyze how a group of visitors behaves in the facility, and it becomes possible to carry out marketing that makes use of the results.

Further, in the face image matching system of this example, the group attribute matching unit 211 detects group attributes for a plurality of people from the same shot image, the frequency at which each person's face image is detected from the same shot image, and the face image of each person in the shot image. Estimated based on at least one of the proximity indicating the distance between the images and the intimacy indicating the relationship of the line of sight of each person in the captured image, and further, the age or gender of each person estimated from the facial images of these multiple persons. The relationship between these multiple persons is estimated based on at least one of the above.
This makes it possible to accurately identify a group of visitors who act in the facility with multiple people. In addition, since the relationship between each visitor is estimated from the personal attributes of each visitor, it becomes possible to analyze the difference in behavior according to the type of group (for example, parent and child, grandparents and grandchildren, friends, lovers, couples, etc.).

In addition, the face image matching system of this example aggregates the data obtained by the group attribute matching unit 211 based on at least one of the age or gender of each person, the date and time of entry or exit to the facility, and the group attribute. Further, a transfer processing unit 214 for transferring the resulting data is provided.
This makes it possible to analyze not only each person of the visitors but also various information obtained about each group from various viewpoints, and when and when the group composed of what kind of person By analyzing where and what you did, you can use it for marketing not only to each person but also to a specific attribute group.

Here, for example, the staying time of the facility may be calculated from the entry date and time and the exit date and time of the visitor, and the behavior analysis considering the staying time of the visitor (or group) may be performed and utilized for marketing.
In addition, by installing imaging devices not only in the stores in the facility but also in the aisles and plazas to verify the faces of visitors, we can analyze the behavior of visitors (or groups) in consideration of their flow lines for marketing. You may try to utilize it.

In the explanation so far, the case where the facility has one entrance and one entrance is taken as an example, but it goes without saying that the number of entrances and exits may be multiple. Further, the entrance and exit do not have to be separate, and there may be an entrance / exit that also serves as these. In this case, an image pickup device facing the outside of the facility and an image pickup device facing the inside of the facility may be installed at the entrance / exit.

Next, a face image search system according to another embodiment of the present invention will be described. The face image search system according to the present embodiment is an extension or modification of the above-mentioned face image matching system, and further includes a search server 600 and an operation terminal 700 as shown by a broken line in FIG. The search server 600 searches for a face image from the face database 208 (or collation history database 212) in the collation server 200 based on the search conditions specified by the operator of the operation terminal 700, and transmits the search result to the operation terminal 700. And perform the process of displaying and outputting.

In the face image search system according to the present embodiment, the group attribute collation unit 211 is provided with a group action determination function. The group behavior determination function determines whether the person in the face image included in the image captured by the imaging device 100 is acting in a group or alone, and the information of the determination result is used as the face of the person. It is stored in the face database 208 (or the collation history database 212) in association with the image. In this example, the group action determination function is realized by performing the proximity person number determination process and the constant proximity person number determination process.

In the proximity person number determination process, when a face image of a walking person is detected over a plurality of image frames (captured images) in the shooting range of one imaging device, each image frame is in the same image frame. The process of calculating the number of face images of another person (number of nearby people) detected in the area close to the detection position of the face image of the person in question and saving the maximum value of the calculated number of nearby people for each image frame. Is. The reason for finding the maximum value of the number of nearby people is that when a face image is detected in image processing, the face image may not be detected temporarily due to changes in shooting conditions such as the influence of obstacles and the amount of light. Is.

The process of determining the number of nearby persons will be described with reference to the process flow of FIG.
First, the memory is allocated and the maximum number of nearby people is initialized (step S101). After that, the following processing is performed in order from the image frame in which the determination target person is first detected.
First, the number of detected face images included in the image frame, the detection position, and the like are acquired (step S102).
Next, the number of face images of other persons existing in the vicinity of the face image of the person to be determined in the image frame is counted (step S103). In this example, another face image in the image frame whose distance from the face image of the judgment target person is equal to or less than a predetermined value is specified, and the number is counted as the number of nearby people. You may count the number of people in close proximity.

Next, the number of close-up people in the image frame counted in step S103 is compared with the maximum number of close-up people (step S104). As a result, if it is determined that the number of nearby people in the image frame is larger, the process proceeds to the maximum number of nearby people update process (step S105), and if not, the person area tracking process (step S106) is performed. Transition.
In the maximum number of nearby persons update process (step S105), the value of the maximum number of nearby persons is replaced with the value of the number of nearby persons in the image frame and saved, and then the process proceeds to the person area tracking process (step S106).

In the person area tracking process (step S106), the movement of a person is tracked based on the detection result of the immediately preceding or latest number of frames and the detection result of the current image frame. For example, the movement of a person is tracked by comparing the position and feature amount of a face image between image frames.
After that, it is determined whether or not the tracking is completed (step S107). Here, when there is a person whose tracking has been completed (for example, a person who is out of the imaging area), it is determined that the tracking has been completed, and when there is no person whose tracking has been completed, the tracking has been completed. It is determined that there is no such thing.

If it is determined in the person tracking determination in step S107 that the tracking has not been completed, the process returns to step S102 and the next image frame is processed. On the other hand, when it is determined that the tracking is completed, the maximum number of nearby persons with respect to the determination target person is determined, and the value is recorded (step S108). After that, processing such as release of the secured memory is performed.

The constant number of nearby persons determination process is a process that is performed in addition to the number of nearby persons determination process (FIG. 6) when a certain person is photographed by a plurality of imaging devices. In the constant proximity number determination process, it is determined whether the person photographed together with the determination target person happens to be near the determination target person or is always acting with the determination target person, and the action is always shared. Always save the number of people who are in the vicinity as the number of nearby people.
Hereinafter, a person who happens to be near the judgment target person is simply described as a "proximity person", and a person who always acts with the judgment target person is described as a "always close person".

The constant number of nearby persons determination process will be described with reference to the process flow of FIG. 7.
First, the memory is allocated (step S201).
Next, the number of nearby persons (that is, the maximum number of nearby persons) recorded in the proximity person number determination process is acquired (step S202).
Next, it is determined whether the number of nearby persons is 0 or 1 or more (step S203), and when the number of nearby persons is 0, 0 is always recorded in the number of nearby persons (step S209).

On the other hand, when the number of nearby persons is 1 or more, the shooting status is searched by another imaging device (step S204). That is, the feature amount of the face image of the judgment target person is used to search for an image captured by another imaging device in which the judgment target person is shown.
After that, it is determined whether or not there is a person to be determined and the number of nearby persons is 1 or more in the image captured by another imaging device (step S205). That is, it is determined whether or not the image captured by the other image pickup device includes the image captured by the person to be determined together with the other person.

As a result, when the captured image in which the determination target person is reflected together with the other person is not found, 0 is always recorded in the number of nearby persons (step S209).
On the other hand, when a captured image in which the determination target person appears together with another person is found, the feature amount of the face image of the other person included in the captured image is acquired (step S206).
After that, the feature amount of the acquired face image of another person is compared with the feature amount of the face image of the close person detected by the proximity person number determination process, and there is a close person whose similarity of the feature amount is equal to or more than a predetermined value. It is determined whether or not to do so (step S207).

When it is determined in step S207 that there are close-up persons with high similarity, the number of combinations of close-up persons with high similarity, that is, the number of close-up persons commonly appearing in a plurality of images captured by different imaging devices. Is calculated and recorded as the number of nearby persons at all times (step S208). Here, when there are a plurality of other imaging devices that have captured a photographed image in which a person to be determined exists and the number of nearby persons is 1 or more, the total number of constant proximity persons calculated for each of the other imaging devices. Should be recorded. However, it goes without saying that it is necessary to eliminate duplication so that the same nearby person is not counted twice.
On the other hand, when it is determined in step S207 that there is no close person having a high degree of similarity, 0 is always recorded in the number of close persons (step S209).
After recording the number of nearby persons at all times in steps S208 and S209, processing such as release of the secured memory is performed.

The number of constant proximity persons calculated by the above-mentioned group behavior determination function is stored in the face database 208 (or collation history database 212) in association with the face image of the determination target person. Then, when a search condition in which a single action or a plurality of person actions is specified is set by the operator of the operation terminal 700, the search server 600 searches for a face image associated with the number of always-closed persons matching the designation. The search result is displayed and output to the operation terminal 700. That is, when the single action is specified, the face image of the person whose number of close persons = 0 is always searched, and when the action of multiple people is specified, the face image of the person whose number of close persons ≥ 1 is always searched. Will be done.

As described above, the face image search system of this example searches the face image from the face database 208 (or the collation history database 212) that stores the face image included in the image captured by the imaging device based on the specified search conditions. It is a face image search system to search, determines whether or not a person in the face image included in the captured image is acting alone, and stores the information of the determination result in the above database in association with the face image of the person. When a search for a person acting alone or a search for a person acting with multiple people is specified as a search condition, the information of the judgment result matching the specification corresponds to the group attribute collation unit 211 to be performed. It is configured to include a search server 600 that searches the attached face image from the database.

This makes it possible to search by specifying whether the image is displayed by one person or by a plurality of people, which is highly convenient for the user. To give a specific example, it is effective when conducting an investigation focusing on crimes committed by one person or crimes committed by multiple people. That is, in the conventional face image search system, the captured image can only be visually judged, but in the face image search system of this example, the number of active persons can be narrowed down and the search can be performed, so that the efficiency of the survey can be greatly improved.

In addition, the face image search system of this example can also be used for the purpose of switching the content to be displayed on the digital signage for marketing purposes. For example, if a person who acts alone comes near the digital signage, the banquet course of the tavern is displayed on the digital signage, and when a person who acts alone comes near the digital signage, a bar with only a counter Can also be displayed on digital signage.

Here, when the same person is included in a plurality of images captured by different imaging devices, the face image search system of this example is the same based on the commonality of other persons included in the plurality of captured images. Since it is configured to determine whether or not a person is acting alone, it is possible to determine the number of people who are acting more accurately than when the number of people who are acting is determined only by the images captured by each imaging device.

In the face image search system of this example, the number of close persons is always used as the information of the judgment result of whether or not the person in the face image included in the photographed image is acting alone, so the number of actions is specified. It is also possible to correspond to the searched. That is, in the case of a search in which the number of action persons is specified, the face image of a person whose number of close persons = (number of action persons-1) may be always searched. Further, the search in which the number of actions is specified may be further expanded so that the range of the number of actions can be specified.

Further, in the face image search system of this example, considering that people who act by a plurality of people are more likely to enter together, the number of nearby persons is determined using the image taken by the image pickup device 100 (1) at the entrance. The processing is performed, and the number of nearby persons is constantly determined using other imaging devices 100 (2) to 100 (N) in the facility and at the entrance, but the configuration is not limited to this. That is, the number of nearby persons determination process and the constant number of nearby persons determination process can be performed by using an arbitrary imaging device.

Needless to say, the face image search system of this example can be applied to facilities other than those having entrances and exits. In addition, when tracking a person between image frames or determining the commonality of a person between different imaging devices, not only the feature amount of the face image but also other feature amounts such as clothes, belongings, and hairstyle are used. You may add it. Further, instead of (or together with) the face database 208 and the collation history database 212, an image database for recording images captured by the imaging device is provided, and group behavior determination is performed using the captured images stored in the image database. May be good.

The configuration of the system or device according to the present invention is not necessarily limited to the one shown above, and various configurations may be used. For example, the configurations described in each of the above embodiments may be used in combination.
Further, the present invention can be provided, for example, as a method or method for executing the process according to the present invention, a program for realizing such a method or method, a storage medium for storing the program, or the like.

The present invention can be used in a face image matching system for collating face images taken at a facility having an entrance and an entrance.

100 (1) to 100 (N): Imaging device, 200: Verification server, 300: Maintenance PC, 400: Network switch, 500: Aggregation server, 600: Search server, 700: Operation terminal,
201: Face detection unit, 202: Same person determination unit, 203: Representative face selection unit, 204: Format conversion unit, 205: Attribute determination unit, 206: ID-assigned person information generation unit, 207: Registration processing unit, 208: Face Database, 210: Face matching unit, 211: Group attribute matching section, 212: Matching history database, 214: Transfer processing section

Claims (7)

In a face image search system that searches for a face image based on specified search conditions from a database that stores the face image included in the image taken by the image pickup device.
A determination means for determining whether or not a person with a face image included in the captured image is acting alone, and storing the information of the determination result in association with the face image of the person in the database.
When a search for a person acting alone or a search for a person acting with a plurality of people is specified as the search condition, the face image associated with the information of the determination result matching the designation is described above. Search means to search from the database and
When a face image of the same person is detected over a plurality of captured images in the imaging range of any of the imaging devices, the person is used as a determination target person, and each of the plurality of captured images is included in the same captured image. The number of face images of other people detected in the area close to the detection position of the face image of the judgment target person is calculated as the number of close people, and the maximum value of the number of close people calculated for each captured image is the maximum number of close people. Equipped with a means for determining the number of nearby people to be saved as
The proximity person number determination means targets each of the plurality of captured images in order for counting processing, and another facial image existing at a position where the distance from the face image of the determination target person is equal to or less than a predetermined value in the target captured image. Is specified, the number of close-up people is counted, the number of close-up people in the target captured image is compared with the maximum number of close-up people so far, and if it is determined that the number of close-up people of the target is larger, the maximum proximity is reached. Transition to the number of people update process, otherwise transition to the count process for the next captured image, and in the maximum number of nearby people update process, the value of the maximum number of nearby people is used as the value of the maximum number of nearby people in the target captured image. A face image search system characterized by replacing it with a number value, saving it, and transitioning to a counting process for the next captured image. In the face image search system according to claim 1,
In the determination means, when the same person is included in a plurality of images captured by different imaging devices, the same person acts alone based on the commonality of other persons included in the plurality of captured images. A face image search system characterized by determining whether or not a person is present. In the face image search system according to claim 1 or 2.
The imaging device includes a first imaging device installed at the entrance of the facility and a second imaging device installed in the facility and at the entrance of the facility.
A selection means for selecting a representative face image, which is a face image representing the person, from a plurality of face images determined to be the same person by the same person determination process based on the image captured by the first imaging device. When,
A registration means for registering a representative face image selected by the selection means in the database, and a registration means.
A face image search system further provided with a collation means for collating a face image included in an image captured by the second image pickup device with a representative face image registered in the database. In the face image search system according to claim 3,
The selection means compares the plurality of face images with a pre-registered reference face, and selects a representative face image of the person based on the degree of similarity with the reference face.
A face image search system characterized in that a face image in which the eyes are opened facing the front is used as the reference face. In the face image search system according to claim 3 or 4.
When the captured image used for the same person determination process includes a face image of a person other than the person, the registration means obtains group attribute estimation data including the face image of the other person. Registered in the above database in association with the representative face image of
When the image captured by the second imaging device includes facial images of a plurality of persons, the determination means includes the facial image included in the captured image based on the group attribute estimation data registered in the database. Estimate a group attribute indicating whether the person is acting alone or in a group, include the group attribute in the information of the determination result, associate it with the face image of the person, and store it in the database. A face image search system featuring. In the face image search system according to claim 5,
The determination means determines the group attributes of the plurality of persons, the frequency with which the face image of each person is detected from the same captured image, the proximity indicating the distance between the face images of each person in the captured image, and each of the captured images. The plurality of persons are estimated based on at least one of the intimacy indicating the relationship between the lines of sight of the persons, and further based on at least one of the ages or genders of each person estimated from the facial images of the plurality of persons. A face image search system characterized by estimating the relationship between people. In the face image search system according to claim 5 or 6.
Further provided with an aggregation means for aggregating the data obtained for each group composed of a plurality of persons based on at least one of the age or gender of each person, the date and time of entry or exit to the facility, and the group attribute. A face image search system featuring.

Images