JP2021149854A - Information distribution system and information distribution method - Google Patents

Abstract

To provide desired information in a timely manner to users of a complex facility acting as a group.SOLUTION: An information distribution system comprises: a registration management device that acquires face images for registration of and registration information on users; photographing devices 7, 11 that creates group images; a face authentication device that executes face authentication processing based on face images for authentication of the users acquired from the group images and the face images for registration; a group determination device that determines a group distinction based on the group images; and an information distribution device that, when registrants of the complex facility are included in the group images, distributes information for a group generated or selected in accordance with a result of determination of the group distinction to portable terminals 10 of the registrants based on the registration information. The registration information includes information on whether the registrants desire distribution of information, and the information distribution device distributes the information for a group to the portable terminals of the registrants only when the registrants desire the distribution of information.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an information distribution system and an information distribution method for distributing information to users who act in a group in a complex facility including a plurality of facilities.

Conventionally, techniques for providing various information to a group and individuals belonging to the group via a display, an electric bulletin board, or the like have been developed.

For example, an information providing device that provides information based on a predetermined group or an individual belonging to the group based on the group classification (or the attribute of the group) is known (see Patent Document 1). This conventional information providing device acquires a face image of a person via an imaging means, acquires the number of people when there are a plurality of acquired face images, and acquires a feature amount from the acquired face image. The attributes of the person are acquired based on the feature amount, the group attribute to which the person belongs is determined from the attribute of the person and the number of the person, the provided information to be provided is determined according to the group attribute of the person, and the information to be provided is determined. The determined provision information is output to the person.

Japanese Unexamined Patent Publication No. 2004-227158

When the information providing device as described above distributes information to members (at least a part of) of a plurality of groups using a complex facility such as an integrated resort, the destination of the group (that is, the group uses it). It is desirable to carry out this depending on whether or not each facility) and each group member wishes to distribute information.

However, in the prior art described in Patent Document 1, the destination of the group and whether or not each group member desires information distribution are not considered at all.

Therefore, the main purpose of this disclosure is to provide an information distribution system and an information distribution method capable of providing desired information in a timely manner to users of a complex facility acting as a group.

The information distribution system of the present disclosure is an information distribution system that provides information to users of a complex facility that acts as a group, and is a face image for registration of the user registered as a registrant of the complex facility and a face image thereof. A registration management device that acquires registration information of the user other than the face image, a photographing device that generates a group image by photographing the group, and an authentication device for the user acquired from the group image. A face authentication device that executes face authentication processing based on the face image and the face image for registration, and a group determination device that determines the group classification for distinguishing the group according to the difference in its properties based on the group image. And, when the group image includes the registrant of the complex facility, the information for the group generated or selected according to the determination result of the group classification is carried by the registrant based on the registration information. The registration information includes an information distribution device for distributing to a terminal, and the registration information includes information on whether or not the registrant desires information distribution. In the information distribution device, the registrant distributes the information. The configuration is such that the information for the group is distributed to the mobile terminal of the registrant only when desired.

In addition, the information distribution method of the present disclosure is an information distribution method that provides information to users of a complex facility that acts as a group, and is a face image for registration of the user registered as a registrant of the complex facility. By acquiring the user's registration information other than the face image and the user's registration information and photographing the group, a group image is generated, and the user's face image for authentication and the registration obtained from the group image are obtained. Based on the face image for Is included, the information for the group generated or selected according to the determination result of the group classification is distributed to the mobile terminal of the registrant based on the registration information, and the registration information includes the registration. Information on whether or not a person wishes to distribute information is included, and the information for the group is transmitted to the mobile terminal of the registrant only when the registrant desires the information distribution. It is configured to be delivered to.

According to the present disclosure, it is possible to provide desired information in a timely manner to users of a complex facility that acts as a group.

Overall configuration diagram of the facility management system according to this embodiment Explanatory diagram showing an example of an event that is the target of user face recognition processing Block diagram showing the schematic configuration of the face recognition machine 1 Block diagram showing the schematic configuration of the management terminal 2 Block diagram showing a schematic configuration of the registration device 5 A block diagram showing a schematic configuration of the face management server 8. A block diagram showing a schematic configuration of the face matching server 9. Block diagram showing a schematic configuration of an external device A block diagram showing a schematic configuration of the group management server 4. Explanatory diagram showing an example of group division Explanatory diagram showing a larger group consisting of multiple groups Explanatory diagram showing an example of a group list Explanatory diagram showing an example of the history of facilities visited by the group Explanatory diagram showing an example of the history of facilities visited by the group Explanatory diagram showing the outline of the face image extraction process Explanatory drawing showing an example of the registration screen displayed on the registration device 5. Sequence diagram showing the procedure of registration process for users Explanatory drawing which shows the first example of the process of face recognition about a user Explanatory diagram showing a second example of face recognition processing regarding a user Explanatory drawing which shows an example of the authentication screen displayed on face recognition machine 1. Explanatory diagram showing a third example of face recognition processing regarding a user Explanatory diagram showing an example of information distribution processing for registrants Explanatory diagram showing a comparison between information distribution to individual users (comparative example) and information distribution by the group management server 4. Explanatory diagram showing an example of group behavior information generation processing Explanatory diagram showing an example of screen display processing for observing group behavior Explanatory diagram showing an example of a group action display screen Explanatory diagram showing an example of a group action display screen Explanatory diagram showing an example of a group action display screen Explanatory diagram showing an example of a group action display screen Explanatory diagram showing an example of a group action display screen Explanatory diagram showing an example of group integration processing to set multiple groups as a large group Explanatory diagram showing an example of personal connection chart creation processing Explanatory diagram showing an example of a personal connection chart Explanatory drawing showing an example of the registration information management screen

The first invention made to solve the above-mentioned problems is an information distribution system that provides information to users of a complex facility that acts as a group, and the user who is registered as a registrant of the complex facility. A registration management device that acquires a face image for registration and registration information of the user other than the face image, a photographing device that generates a group image by photographing the group, and acquisition from the group image. A face authentication device that executes face authentication processing based on the user's face image for authentication and the registration face image, and a group for distinguishing the group according to the difference in its properties based on the group image. The registration information includes a group determination device for determining the classification and information for the group generated or selected according to the determination result of the group classification when the group image includes the registrant of the complex facility. The information distribution device includes an information distribution device that distributes information to the registrant's mobile terminal based on the above, and the registration information includes information on whether or not the registrant desires information distribution. Only when the registrant wishes to distribute the information, the information for the group is distributed to the registrant's mobile terminal.

According to this, the group classification is determined based on the group image used for face recognition of the user of the complex facility, and the information for the group is distributed only to the mobile terminals of the registrants who wish to distribute the information. It is possible to provide desired information in a timely manner to the users of the complex facility that acts in.

The second invention is configured such that the registration management device acquires the face image for registration and the registration information from the mobile terminal.

According to this, it is possible to smoothly acquire the face image and registration information for registration of the user and distribute the information to the registrant (user after registration).

Further, the third invention has a configuration in which the information for the group is generated or selected according to the shooting position of the group image.

According to this, it becomes possible to provide more appropriate information to the users of the complex facility acting in the group according to the current position of the group (that is, the shooting position of the group image).

Further, the fourth invention is configured such that the information for the group is information about one or more facilities included in the complex facility determined according to the group classification.

According to this, it becomes possible to provide information on more appropriate facilities to users of complex facilities acting in a group according to the group classification.

Further, the fifth invention is configured such that the one or more facilities are determined according to the usage status of each facility included in the complex facility by a plurality of other groups.

According to this, it is possible to provide information on more appropriate facilities to users of complex facilities acting in groups according to the usage status of each facility by other groups.

Further, the sixth invention is configured such that the usage status of each facility includes the number of users of each facility.

According to this, it is possible to provide information on more appropriate facilities to users of complex facilities acting in groups according to the number of users of each facility by other groups.

Further, the seventh invention is configured such that the usage status of each of the facilities relates to a predetermined time zone.

According to this, it becomes possible to provide the users of the complex facilities who act in a group with more appropriate information on the facilities according to the usage status of each facility that may change in each time zone.

Further, the eighth invention is an information distribution method for providing information to users of a complex facility that acts as a group, and is a facial image for registration of the user registered as a registrant of the complex facility and a face image thereof. By acquiring the registration information of the user other than the face image and photographing the group, a group image is generated, and the face image for authentication of the user and the registration information acquired from the group image are used. Based on the face image, the face recognition process is executed, and based on the group image, the group division for distinguishing the group according to the difference in its properties is determined, and the group image includes the registrant of the complex facility. In that case, the information for the group generated or selected according to the determination result of the group classification is distributed to the mobile terminal of the registrant based on the registration information, and the registrant is included in the registration information. Information on whether or not information distribution is desired is included, and the information for the group is provided only when the registrant desires the information distribution to the mobile terminal of the registrant. It will be delivered and configured.

According to this, the group classification is determined based on the group image used for face recognition of the user of the complex facility, and the information for the group is distributed only to the mobile terminals of the registrants who wish to distribute the information. It is possible to provide desired information in a timely manner to the users of the complex facility that acts in.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

FIG. 1 is an overall configuration diagram of a facility management system according to the present embodiment.

This facility management system includes a face authentication machine 1, a management terminal 2, a face authentication server 3 (face authentication device, registration management device), and a group management server 4 (group judgment device, information distribution device, mobile information generation device). A registration device 5, a facility display 6 (display device), and a facility camera 7 (photographing device) are provided. The face recognition server 3 includes a face management server 8 and a face matching server 9. A plurality of each of these components may be provided in the facility management system.

The facility management system can function as an information distribution system that provides useful information to users of a complex facility (particularly, a group consisting of a plurality of users). The facility management system can also function as a behavior observation system for observing the behavior of users (particularly, the behavior of a group of users) in a complex facility.

The complex is, for example, an integrated resort. Complex facilities include conference facilities, exhibition facilities, attractiveness promotion facilities (theaters, theaters, music halls, stadiums, movie theaters, museums, art galleries, restaurants, etc.), customer transfer facilities (stations, parking lots, etc.), accommodation facilities. And amusement facilities etc. may be included. Further, the complex facility may be a set of a plurality of public facilities and commercial facilities provided in an arbitrary area (for example, an area of a local government).

The face recognition machine 1, the management terminal 2, the group management server 4, the registration device 5, the facility display 6, the facility camera 7, the face management server 8 and the face matching server 9 are connected to each other so as to be able to communicate with each other via a network such as the Internet. Has been done. Further, the group management server 4, the registration device 5, the face management server 8, and the like can communicate with the mobile terminal 10 of the user of the complex facility via the network. The mobile terminal 10 includes a smartphone, a tablet terminal, and the like.

The face recognition machine 1 is installed at a plurality of places where face recognition is required, such as the entrance / exit of the area of the complex facility, the entrance / exit of the building of each facility constituting the complex facility, the entrance / exit of the room, and the like. The registration device 5 is arranged at a reception counter or the like that accepts visitors to the complex. The required number of face matching servers 9 is provided according to the number of face recognition machines 1 and the like.

The face recognition machine 1 is provided with a camera 11 (photographing device) for face recognition, and the camera 11 acquires a photographed image of a user of the complex facility (visitor to the complex facility). This captured image includes a group image showing a plurality of users acting in a group. A plurality of facility cameras 7 are installed at appropriate locations in the complex facility. Each facility camera 7 can acquire a photographed image of the user in the same manner as the camera 11 for face recognition. For example, a unidirectional camera is used for the face recognition camera 11, and an omnidirectional camera is used for each facility camera 7.

Further, the face recognition machine 1 has a display 12 for face recognition. The display 12 displays the face authentication result acquired from the face matching server 9. From this display, the user can grasp the face recognition result.

The management terminal 2 is composed of a PC and is operated by the administrator of the facility management system. On the display 32 (display device) of the management terminal 2, various information for the manager to manage the facility management system is displayed. In addition, a plurality of facility displays 6 are installed at appropriate locations in the complex facility. Each facility display 6 can display information for users or information for staff of a complex facility in response to a command from the management terminal 2.

A management application is installed on the management terminal 2. This management application is used to manage the operations of the face recognition machine 1, the group management server 4, the registration device 5, the facility display 6, the facility camera 7, the face management server 8, and the face matching server 9. The administrator can perform various management tasks by the management application. The management application is implemented as a Web application.

The group management server 4 manages users who act as a group in a complex facility. Specifically, the group management server 4 cooperates with the face recognition server 3 and the like to observe the behavior of the group in the complex facility and to the members of the group (one or more users belonging to the group). Distribute information.

The registration device 5 acquires information (hereinafter referred to as registrant information) used for processing the user's face authentication. The registrant information may include a face image for registration of each user and user registration information other than the face image. The user of the complex facility is treated as a registrant (registered user) of the complex facility by registering the registrant information in the facility management system. The registrant is a legitimate user of the complex, and can pass through a security gate based on face recognition and perform payment processing based on face recognition.

The registration information may include information that can permanently identify the user, such as the user's name. In addition, the registration information may include temporary information that can identify the user. Such temporary information includes the registrant number temporarily given to the user of the complex and the meeting room (including any space where interviews are possible) used by the registrant in the complex. The number and the number of the room where the registrant stays at the accommodation facility etc. are included.

In addition, the registration information may include information for confirming the intention of the user regarding the usage method of the complex facility. Such information includes information on whether or not the user desires information distribution to the mobile terminal 10 or the like.

The user can install the application (application program) 18 for using the complex facility on his / her mobile terminal 10. The application 18 has a function of acquiring registrant information from the user instead of the registration device 5 described above. That is, the mobile terminal 10 can function as the registration device 5. Hereinafter, the registration device 5 will be mainly described, but the same applies to the case where the mobile terminal 10 functions as the registration device 5.

The face management server 8 acquires information (including personal information) of each registrant and centrally manages them. Specifically, the face management server 8 stores and stores face image data for registration and registration information for each registrant in association with each other, and manages them. The face management server 8 acquires a face image for registration and registration information from the registration device 5 (or mobile terminal 10) at the time of user registration. Further, the face management server 8 may acquire at least a part of the information included in the registration face image and the registration information from the management terminal 2 (that is, the administrator of the facility management system).

At the time of face recognition, the face matching server 9 acquires face image data for authentication of one or more face recognition target persons from the face recognition machine 1, and obtains the face feature amount data of the target person from the face image data. It is determined whether or not the target person is a registrant by performing face matching by comparing the face feature amount data of the target person with the face feature amount data of the registrant stored in the own device. Perform face recognition.

Further, the face matching server 9 acquires face image data for user registration from the face management server 8 (or registration device 5) at the time of user registration prior to face recognition, and the face image data. The user's facial feature amount data is generated from the data and stored in the own device. In some cases, the face matching server 9 may acquire the photographed image of the user from the management terminal 2 (that is, the administrator) and acquire the face image data from the photographed image.

In the present embodiment, the face management server 8 and the face matching server 9 are assumed to exist in physically different information processing devices, but even if they exist in a single information processing device. good. Further, at least one of the face management server 8 and the face matching server 9 and the group management server 4 described in detail later may exist in a single information processing device.

Further, in the present embodiment, the management terminal 2 and the face management server 8 are provided, but the management terminal 2 and the face management server 8 can be configured by a single information processing device. For example, by installing the management application on the face management server 8, the face management server 8 can also serve as the management terminal 2.

Further, in the present embodiment, the face feature amount matching is performed in the face recognition process of the user of the complex facility, but the face matching is not limited to the face feature amount matching, and the matching method applying machine learning or the like is applied. May be adopted. Furthermore, this embodiment can be applied to biometric authentication other than face authentication.

Next, an example of an event that is the target of the user's face authentication processing in the complex facility to which the facility management system is applied will be described. FIG. 2 is an explanatory diagram showing an example of an event occurring in a complex facility.

As shown in FIG. 2, the group 100 composed of the users of the complex facility operates the registration in the facility management system (that is, the registration of the information used for the face recognition processing) by the registration device 5 arranged at the reception. It can be performed. A user who has completed the registration operation is treated as a registrant 100A in the complex. In addition, registered users who revisit the complex within a predetermined period are also treated as registrant 100A. In a complex, it is not mandatory for all users to register (ie, be registrants). However, in the complex facility, it is advisable to encourage the users to register so that at least one registrant 100A is included in the group 100 composed of a plurality of users. The group 100 is usually composed of a plurality of group members, but may include a single user.

In the complex facility, face recognition is executed for each of various events (see the two-dot chain line in FIG. 2) generated by the actions of the group 100 (including the registrant 100A) in the complex facility.

For example, when the registrant 100A receives face recognition in front of a security gate that manages the passage of users in the complex facility, the group 100 can pass through the security gate. A face recognition machine 1 that cooperates with the gate device (external device 16) (that is, controls the open / closed state of the door portion) can be installed in the security gate.

Further, for example, when the registrant 100A receives face recognition in front of the guest room in the accommodation facility, the group 100 can enter the guest room. In front of the guest room, a face recognition machine 1 that cooperates with a lock device that locks the door of the guest room (that is, controls the locking or unlocking of the door) can be arranged.

Further, for example, when the registrant 100A receives face recognition in the restaurant in the complex facility, the group 100 can perform the payment processing (automatic payment by face recognition) of the usage fee of the restaurant. A camera 11 capable of photographing the faces of each user belonging to the group 100 is installed in the restaurant, and the face recognition machine 1 cooperates with a payment device (not shown) used for payment of a usage fee at the restaurant.

Further, for example, the registrant 100A can perform the settlement processing of the usage fee of the gym by receiving face recognition at the gym in the complex facility. A camera 11 capable of photographing the face of the registrant 100A is installed in the gym, and the face recognition machine 1 cooperates with a payment device (not shown) used for payment of a usage fee at the gym. In addition, other users (unregistered persons) belonging to the group 100 can collectively perform the payment processing by using the gym together with the registrant 100A.

The events that occur in the complex are not limited to the above examples, for example, the settlement processing of the usage fee in the parking lot of the complex facility, the settlement processing of the usage fee of the laundry service in the accommodation facility in the complex facility, and the like. , Various events such as payment processing of the usage fee of the bathhouse in the complex may be included.

Next, the schematic configuration of the face recognition machine 1 will be described. FIG. 3 is a block diagram showing a schematic configuration of the face recognition machine 1.

The face recognition machine 1 includes a camera 11, a display 12, a communication unit 13, a storage unit 14, a control unit 15, and a control signal transmission unit 17.

The camera 11 constantly shoots a predetermined shooting area, and when a group consisting of users enters the shooting area, the group is shot and a shot image (group image) including a person subject to face recognition can be acquired. In order to save power, a motion sensor (not shown) may be provided to detect the arrival of a person and activate the camera 11.

The display 12 displays the progress of face recognition and the face recognition result, and the group receiving the face recognition can confirm them. The display 12 may be provided with a speaker as an output unit for outputting the authentication result, and the face authentication result may be notified by voice. Further, the display 12 can be omitted depending on the installation location of the face recognition machine 1 (for example, when the face recognition machine 1 is installed at the security gate).

The communication unit 13 communicates with the face matching server 9 via the network. In the present embodiment, the captured image (face image) data is transmitted to the face matching server 9. In addition, the user's authentication result is received from the face matching server 9. Further, the communication unit 13 communicates with the group management server 4 and the face management server 8 via the network.

The storage unit 14 stores captured image (face image) data, a control program executed by a processor constituting the control unit 15, and the like.

The control unit 15 includes a face image extraction unit 22, a face matching request unit 23, an authentication result notification unit 24, and an authentication result adjustment unit 25. The control unit 15 is composed of a processor, and each unit of the control unit 15 is realized by executing a program stored in the storage unit 14 by the processor.

The face image extraction unit 22 acquires a group shot image from the camera 11, detects the faces of one or more people from the shot images, determines whether the detected face sizes are appropriate, and shoots. The face area can be cut out from the image, and the data of the face image of one or more users can be acquired. The face image data may be only the image of the face area, but the data of the photographed image (the image of the predetermined image) and the position information (face frame information) of the face area on the photographed image data. It may be a combination of.

The face matching request unit 23 transmits a face matching request to the face matching server 9 by the communication unit 13 for each user included in the captured image.

The authentication result notification unit 24 notifies the group of the face authentication result by displaying the face authentication result acquired from the face verification server 9 by the communication unit 13 on the display 12.

The authentication result adjustment unit 25 controls to cooperate with the external device 16 based on the user's face matching result acquired from the face matching server 9. Examples of the external device 16 include a gate device that manages the passage of registrants in a complex facility, a door lock device that manages entry and exit of a hotel guest room or a predetermined area in a complex facility, and a registrant in the complex facility. A payment device for making payments for received services and purchased items can be used (see FIG. 2). The payment device can perform postpaid payment processing based on information such as the registrant's credit card. The authentication result adjustment unit 25 can acquire the payment processing result of each registrant from the payment device. The payment processing result includes information on the payment amount (consumption amount) for the purchase of goods and the use of services at each facility.

The face recognition device 1 and the external device 16 are directly connected by a communication cable or the like, or are communicably connected via a known network. The authentication result adjustment unit 25 can notify the user's face matching result to the server or the like of the external system including the external device 16 by being configured by the connection application.

The control signal transmission unit 17 transmits a control signal for controlling the operation to the external device 16.

Next, the management terminal 2 will be described. FIG. 4 is a block diagram showing a schematic configuration of the management terminal 2.

The management terminal 2 includes a communication unit 31, a display 32, an input device 33, a storage unit 34, and a control unit 35.

The communication unit 31 communicates with the face management server 8 via the network. In the present embodiment, screen information and the like are received from the face management server 8, and the administrator's operation information and the like corresponding to the screen information and the like are transmitted to the face management server 8. Further, the communication unit 31 communicates with the group management server 4 via the network.

The display 32 displays various screens. The input device 33 is a mouse, a keyboard, or the like, and operates the screen displayed on the display 32.

The storage unit 34 stores a program (management application) or the like executed by the processor constituting the control unit 35.

The control unit 35 includes a GUI control unit 38. The control unit 35 is composed of a processor, and each unit of the control unit 35 is realized by executing a program stored in the storage unit 34 by the processor.

The GUI control unit 38 displays on the display 32 screens related to various information distributed from the face management server 8 and the group management server 4. In addition, the input information is acquired and the screen is controlled according to the input operation of the administrator using the input device 33. In the present embodiment, the GUI control unit 38 controls the display and input regarding the screen for observing the behavior of the user (particularly, the behavior of the group consisting of the users) in the complex facility. In addition, the GUI control unit 38 controls the display and input of screens related to registrant management, specifically, screens related to user information registration (individual registration, batch registration), reference, update, and deletion. conduct.

Next, the schematic configuration of the registration device 5 will be described. FIG. 5 is a block diagram showing a schematic configuration of the registration device 5.

The registration device 5 includes an input device 40, a camera 41 for face photographing, a display 42, a communication unit 43, a storage unit 44, and a control unit 45.

The input device 40 includes a keyboard and the like. The input device 40 may be provided integrally with the display 42 as a touch panel. The user (or receptionist) can input the user's registration information by using the input device 40.

The face photographing camera 41 can acquire a photographed image of the user by, for example, photographing the face of the user who visited the reception desk of the complex facility.

Further, the registration device 5 can be provided with an information reading device (for example, an RFID reader) capable of reading the user's registration information by wireless communication from an information recording medium (for example, RFID tag) possessed by the user. be.

The display 42 can display a registration screen (see FIGS. 16 (A)-(F)) showing the procedure for registering the registrant information to the user. Further, by attaching a speaker to the display 42, the registration procedure and the like may be guided to the user by voice. Further, by configuring the display 42 with a touch panel, it is possible to acquire the information input by the user by touch operation.

The communication unit 43 communicates with the face management server 8 via the network. In the present embodiment, the face image data and the registration information are transmitted to the face management server 8.

The storage unit 44 stores data of a photographed image (face image) of the registrant, specific information of the registrant, a control program executed by a processor constituting the control unit 45, and the like.

The control unit 45 includes a face image extraction unit 46, a registration information acquisition unit 47, a registration request unit 48, and a GUI control unit 49. The control unit 45 is composed of a processor, and each unit of the control unit 45 is realized by executing a program stored in the storage unit 44 by the processor.

The face image extraction unit 46 acquires a photographed image of the user (registration target person) from the camera 41 for face photographing, detects a person's face from the photographed image, and has the user's face image properly acquired. It can be determined whether or not. If the acquired face image is not appropriate, the face image extraction unit 46 can display a message prompting the user to take a picture again on the display 42. The face image extraction unit 46 may have the same function as the face image extraction unit 22 of the face recognition machine 1 described above.

The registration information acquisition unit 47 acquires the registration information input from the input device 40.

The registration request unit 48 transmits a user registration request to the face management server 8 by the communication unit 43.

The GUI control unit 49 displays various guidance screens for the user distributed from the face management server 8 on the display 42. In addition, the input information is acquired and the screen is controlled according to the input operation of the user using the input device 40.

Next, the schematic configuration of the face management server 8 will be described. FIG. 6 is a block diagram showing a schematic configuration of the face management server 8.

The face management server 8 includes a communication unit 51, a storage unit 52, and a control unit 53.

The communication unit 51 communicates with the management terminal 2 via the network. In addition, the communication unit 51 communicates with the face recognition device 1 via the network. Further, the communication unit 51 communicates with the group management server 4. Further, the communication unit 51 communicates with the registration device 5. In addition, the communication unit 51 communicates with the face matching server 9. Further, the communication unit 51 can communicate with the mobile terminal 10 on which the application 18 is installed instead of the registration device 5.

The storage unit 52 stores a face information database, a control program executed by a processor constituting the control unit 53, and the like.

In the face information database, face images for registration are accumulated as information about each registrant (each registered user). In addition, the registration information of each registrant is stored in the face information database in association with the face image for registration. The face image for registration and the registration information may be stored in an encrypted state for privacy protection. Further, the face information database may store the face image for registration and a part of the registration information in a state of being replaced with anonymized information.

The control unit 53 includes a registrant management unit 62, a face matching server management unit 65, a face information management unit 66, and a database management unit 67. The control unit 53 is composed of a processor, and each unit of the control unit 53 is realized by executing a program stored in the storage unit 52 by the processor. Each unit of the control unit 53 is configured as a Web API (Web Application Programming Interface).

The registrant management unit 62 manages registrant information such as a face image for registration of each registrant and registration information, and performs necessary processing related to the registrant information in response to a request from the management terminal 2. In the present embodiment, the management terminal 2 makes a request for registration, reference, update, and deletion regarding each registrant, and the registrant management unit 62 performs necessary processing in response to the request.

The face matching server management unit 65 manages information about the face matching server 9, and performs necessary processing related to the face matching server 9 in response to a request from the management terminal 2.

The face information management unit 66 has registrant information (face image, registration information, etc.) stored in its own device and registrant's face information (registrant's face feature amount data) stored in the face matching server 9. Face information is synchronized so that it is maintained in a consistent state.

The database management unit 67 manages the database provided in the own device, and backs up and restores the database.

Next, the schematic configuration of the face matching server 9 will be described. FIG. 7 is a block diagram showing a schematic configuration of the face matching server 9.

The face matching server 9 includes a communication unit 71, a storage unit 72, and a control unit 73.

The communication unit 71 communicates with the face recognition device 1 via the network. In the present embodiment, the photographed image (face image) data or the like is received from the face recognition machine 1. In addition, the user's authentication result and the like are transmitted to the face recognition machine 1. Further, the communication unit 71 communicates with the face management server 8 via the network. In the present embodiment, requests for various processes are received from the face management server 8, and responses and the like corresponding to the requests are transmitted to the face management server 8. Further, the communication unit 71 communicates with the group management server 4 via the network.

The storage unit 72 stores a face matching database, association information between the registrant and the feature amount, a control program executed by the processor constituting the control unit 73, and the like. The face feature amount data of the registrant is registered as the information about the registrant in the face matching database.

The control unit 73 includes an image quality check unit 82, a face image extraction unit 83, a face feature amount generation unit 84, a face feature amount collation unit 86, a database management unit 88, a position information acquisition unit 89, and payment information. The acquisition unit 90 and the like are provided. The control unit 73 is composed of a processor, and each unit of the control unit 73 is realized by executing a program stored in the storage unit 72 by the processor. Each part of the control unit 73 is configured as WebAPI.

The image quality check unit 82 determines whether or not the image of the face region in the captured image satisfies a predetermined quality. Specifically, the presence / absence of wearing a mask and the presence / absence of wearing sunglasses are detected from the target image, and the face recognition suitability (evaluation value based on the orientation and facial expression of the face) is calculated.

The face image extraction unit 83 extracts a face image from the captured image of the user (group) acquired by the registration device 5 at the time of user registration. At this time, if necessary, the face image extraction unit 83 detects the face of the person from the captured image, determines whether the size of the detected face is appropriate, cuts out the face area from the captured image, and faces the person. Get an image. The face image extraction unit 83 can also extract a face image from the photographed image of the user acquired by the face recognition machine 1 at the time of face authentication.

The face feature amount generation unit 84 detects face feature points from the data of the face image for registration and the face image for authentication at the time of user registration and face authentication, and generates face feature amount data, respectively.

The face feature amount collation unit 86 has the face feature amount data of the person to be authenticated generated from the data of the face image for authentication acquired from the face recognition machine 1 at the time of face recognition, and the registrant (registered) stored in the own device. It is determined whether or not the person to be authenticated is a registrant by comparing with the facial feature amount data of the user). At this time, when the face image for authentication includes the faces of a plurality of authentication target persons, the face feature amount collating unit 86 determines whether or not each authentication target person is a registrant.

The face feature amount collation unit 86 calculates the degree of similarity (verification score) between the authentication target person and the registrant. By comparing this similarity with a predetermined threshold value, the success or failure of face recognition can be determined. In addition to notifying the face recognition machine 1 of the success / failure determination result as a collation result, the face recognition machine 1 may be notified of the person ID and the similarity of the registrants having a high degree of similarity as the collation result.

The database management unit 88 manages the database provided in the own device, and backs up and restores the database.

The position information acquisition unit 89 acquires information on the shooting position (for example, the position of the camera 11 or the facility camera 7) and the shooting time of the shot image. The position information acquisition unit 89 acquires the shooting position of the shot image as the position at the shooting time of the registrant included in the shot image (the position at the shooting time of the group to which the registrant belongs). The position information at the shooting time of each registrant is stored in the storage unit 72.

The payment information acquisition unit 90 can acquire the payment processing result by the payment device (external device 16) from the face authentication machine 1 as the payment information of the registrant. The payment information is stored in the storage unit 72.

In the present embodiment, as the main functions of the face matching server 9, each function of face image extraction, face feature amount generation, face feature amount matching, registrant's position information acquisition, and registrant's payment information acquisition is performed. However, these functions can be configured by different information processing devices that are independent of each other. For example, the face image extraction function may be configured by another information processing device independent of other functions such as face feature amount generation and face feature amount matching.

Next, the schematic configuration of the external device 16 will be described. FIG. 8 is a block diagram showing a schematic configuration of the external device 16.

The external device 16 includes a control signal receiving unit 91, a driving unit 92, a power supply unit 93, and a control unit 94.

The control signal receiving unit 91 receives the control signal (operation command from the face authentication machine 1 that cooperates with the external device 16) transmitted from the control signal transmitting unit 17 of the face authentication machine 1.

The drive unit 92 is controlled by the control unit 94 based on the control signal from the face recognition machine 1. For example, when the external device 16 is a gate device, the drive unit 92 supplies power for opening (or closing) the door portion of the gate device. When the external device 16 is a door lock device, the drive unit 92 supplies power for locking (or unlocking) the lock device. If the payment device does not include a portion for driving the external device 16, the drive unit 92 can be omitted.

The power supply unit 93 supplies electric power to each unit of the external device 16. Further, the power supply unit 93 is electrically connected to the power supply unit 97 of the face recognition machine 1, and can supply electric power to the power supply unit 97 as well. The power supply unit 97 supplies electric power to each unit of the face recognition machine 1. Further, by connecting to the power supply unit 97 of the face recognition machine 1 via a LAN cable, power can be supplied to the power supply unit 97 by PoE power supply.

Further, the power supply unit 93 can receive power supply from an uninterruptible power supply (or emergency power supply) independent of the power supply line in the entire facility. As a result, for example, the gate device as the external device 16 can perform normal operation even when a power failure occurs in the facility, and has an advantage that the security of the facility can be maintained.

The control unit 94 controls the operation of each unit of the external device 16. Further, the control unit 94 can execute the necessary processing as the external device 16. The control unit 94 is configured by a processor, and is realized by executing a program stored in a storage unit (memory) (not shown) by the processor.

Next, the schematic configuration of the group management server 4 will be described. FIG. 9 is a block diagram showing a schematic configuration of the group management server 4. FIG. 10 is an explanatory diagram showing an example of group division. FIG. 11 is an explanatory diagram showing a larger group composed of a plurality of groups. FIG. 12 is an explanatory diagram showing an example of a group list. FIG. 13 is an explanatory diagram showing an example of the history of facilities visited by the group. FIG. 14 is an explanatory diagram showing an example of the history of facilities visited by the group.

The group management server 4 includes a communication unit 101, a storage unit 102, and a control unit 103.

The communication unit 101 communicates with the management terminal 2 via the network. In the present embodiment, the communication unit 101 transmits the group action information to the management terminal 2. In addition, the communication unit 101 communicates with the mobile terminal 10 via the network. In the present embodiment, the communication unit 101 transmits information for the group regarding the use of the complex facility to the mobile terminal 10. Further, the communication unit 101 communicates with the camera 11 via the network. In the present embodiment, the communication unit 101 receives the captured image from the camera 11. The communication unit 101 may receive the captured image from the facility camera 7 (see FIG. 1) instead of the camera 11. In addition, the communication unit 101 communicates with the face matching server 9 via the network. In the present embodiment, the communication unit 101 receives the determination information as to whether or not the photographed image includes the registrant from the face matching server 9. In addition, the communication unit 101 communicates with the face management server 8 via the network. In the present embodiment, the communication unit 101 receives the registration information of the registrant included in the captured image from the face management server 8. In addition, the communication unit 101 communicates with the external SNS server 104. The SNS server 104 has a function of providing a social networking service.

The storage unit 102 stores group classification information, group list information, group images, group specific information, group behavior information, related group information, distribution information, and the like. The group identification information includes information for identifying the group (for example, identification information of a registrant belonging to the group). The related group information includes information that associates a plurality of groups. The distribution information includes various information (product information, service information, coupon information, event information, map information, etc.) regarding the use of the complex facility or one or more facilities included therein.

The control unit 103 includes a group determination unit 111, a group division unit 112, a specific person extraction unit 113, a group list creation unit 114, a group action totaling unit 115, and an information distribution unit 116. The control unit 103 is composed of a processor, and each unit of the control unit 103 is realized by executing a program stored in the storage unit 102 by the processor.

The group determination unit 111 determines whether or not the subject of the image captured by the camera 11 includes a group consisting of a plurality of users.

When the subject of the captured image includes a group, the group division unit 112 determines the group division according to the group division information stored in the storage unit 102. The group division is appropriately set by the administrator in order to distinguish the groups according to their different properties. As shown in FIG. 10, the group division includes an adult group, a couple, a family with children, a child group, and one person. These group divisions can be further divided according to the ages of the users who make up the group, such as an adult group (20's, 30's) and an adult group (40's or later).

When the subject of the captured image includes a group, the specific person extraction unit 113 extracts the registrants included in the group based on the face authentication result of the face matching server 9. Further, when the same registrant is extracted from a plurality of groups, the specific person extraction unit 113 sets a plurality of groups to which the same registrant belongs as a larger group (hereinafter, referred to as a “large group”).

For example, as shown in FIG. 11A, group guests 121 including a plurality of groups X, Y, and Z act together as one group at the airport according to the guidance of the guide T. On the other hand, as shown in FIG. 11B, when the group passenger 121 arrives at the complex facility from the airport, each group X, Y, and Z individually starts an action. At this time, since the guide T acts while moving between the groups X, Y, and Z, the specific person extraction unit 113 groups the guide T based on the captured images of a plurality of groups having different shooting times. It may be extracted as a registrant belonging to any of X, Y, and Z. In this way, when the specific person extraction unit 113 extracts the same registrant (here, the guide T) from the plurality of groups X, Y, Z, the plurality of groups are grouped into large groups X, Y, and Z. Set as W. As a result, in the facility management system, the groups X, Y, and Z can be treated as one group W in the aggregation of data related to the group behavior. In addition, there may be a plurality of the same registrants extracted from a plurality of groups. By dealing with multiple people, groups X, Y, and Z can be set as large groups W even when there are the same registrants in groups X and Y and the same registrants in groups Y and Z. can.

When the subject of the captured image includes a group, the group list creation unit 114 creates a group list indicating the members of the group. As shown in FIG. 12, the group list includes information (here, age group, gender) regarding members (unregistered users and registrants) of the group (see groups ac in the figure). Is done. In addition, the group list includes information on each group division determined by the group division unit 112. For example, group a is composed of elementary school students, men in their 40s, and women in their 40s, and the group division is 5 (corresponding to a family with children (elementary school students or younger) in FIG. 10). In addition, the group list includes information on whether or not a member of the group is a registrant. For example, group a includes registrants of a man in his forties and a woman in his forties, and group b includes registrants of a man in his twenties and a man in his thirties. In addition, the group list may include information as to whether the registrant wishes to distribute information regarding the use of the complex. For example, the registrant of a man in his thirties in group b is given an identifier indicating that he does not want to distribute information. The age group and gender of the group members are identified using face recognition technology, but if the person is found to be a registered person by face recognition, the registered information is given priority. Create a group list using the information identified from the image and the registered information.

The group behavior aggregation unit 115 aggregates various data related to the behavior of each group. The aggregated result is stored in the storage unit 102 as group action information. In addition, the group action totaling unit 115 transmits the totaling result of various data to the management terminal 2 in response to the request from the management terminal 2.

The group behavior information includes information on the movement history of a group (registrants belonging to a group) between a plurality of facilities in a complex facility or within each facility. As shown in FIG. 13, the information regarding the movement history includes the identifiers (here, A, D, H) of each facility visited by each group (here, group a) and the date when those facilities were visited. And time are included. The information about the movement history may include a time zone instead of the time.

In addition, the information regarding the movement history may further include the location information (latitude / longitude and coordinates in the map of the complex facility) of each facility visited by each group. The acquisition of information regarding travel history can be identified by admission information or payment information at those facilities. That is, the movement history of each group can be generated based on the location information of each facility where the admission processing or the settlement processing is performed by each group (registrant belonging to the group). The information regarding the movement history may include admission information or payment information (for example, consumption amount at each facility). Further, the information regarding the movement history may include the position information of each group moving between the plurality of facilities (for example, the position based on the GPS function of the mobile terminal 10 at a plurality of times set in a predetermined cycle). good. In addition, the movement history of each group can be generated based on the group division (that is, the information of the group division is included).

Further, as shown in FIG. 14, the payment information includes the payment amount at each facility (here, facilities AC) by the registrants belonging to each group (here, groups a and b). Group a includes the amount of consumption (settlement amount) for men in their 40s and women in their 40s who are registrants. In group b, the consumption amount is included in the registrants of men in their twenties and women in their twenties. The payment information may include the total consumption amount for each group.

The information distribution unit 116 can generate information for a group according to the group classification based on the distribution information stored in the storage unit 102. Alternatively, the information distribution unit 116 may select one or more distribution information determined in advance according to the group classification stored in the storage unit 102 as the information for the group. For example, when restaurant information is provided as information for a group, the information distribution unit 116 generates or selects information on a restaurant having a high utilization rate of a group belonging to the target group category in the past results.

The information distribution unit 116 can acquire information on the shooting position of the shot image (for example, information on the position of the camera 11 or the facility camera 7) and generate or select information for the group according to the shooting position. .. In addition, the information distribution unit 116 can generate or select information about one or more facilities determined according to the group classification as information for the group. In this case, the information distribution unit 116 can determine the one or more facilities according to the information on the usage status of each facility by a plurality of other groups belonging to the same group division. The information on the usage status of each facility includes information on the number of users (number of users) and the usage time zone. That is, the information distribution unit 116 generates or selects information on one or more facilities having a larger number of users as information for the group in the time zone corresponding to the time of information distribution based on the information on the usage status of each facility. can do.

The information distribution unit 116 distributes information for the group to the registrant's mobile terminal 10. The information distribution process for this group is performed by using the information distribution function of the application 18 of the mobile terminal 10.

Next, the face image extraction process will be described. FIG. 15 is an explanatory diagram showing an outline of the face image extraction process.

As described above, the face image extraction unit 22 of the face recognition machine 1 generates face image data from the photographed image data of the user by the face image extraction process, that is, each process of face detection, face size check, and face cutout. Processing is performed. This face image extraction process is also performed in the face image extraction unit 83 of the face matching server 9.

At the time of user registration, the registration device 5 sends the data of the photographed image of the user to the face matching server 9 via the face management server 8, so that the face image extracting unit 83 of the face matching server 9 processes the face image. I do. In some cases, the data of the photographed image of the user may be sent from the management terminal 2 to the face matching server 9 via the face management server 8.

On the other hand, at the time of face authentication, it is preferable that the face image extraction process is performed only by the face authentication machine 1 and the face image extraction process is not performed by the face matching server 9. In this case, the face recognition machine 1 is provided with a highly accurate face detection function. Further, the face authentication machine 1 constantly photographs the photographing area with the camera 11, and sends the photographed image data and the face frame information to the face matching server 9 at the timing when the face is detected. As a result, the load of the face image extraction process is distributed to the plurality of face recognition machines 1, and the load of the face matching server 9 can be reduced. Further, since the amount of communication between the face recognition machine 1 and the face matching server 9 can be reduced, the load on the network can be reduced. Then, since the face recognition machine 1 can perform the face recognition response at high speed, the face recognition of the target person appearing one after another can be efficiently performed.

Next, the registration operation by the user in the registration device 5 will be described. FIG. 16 is an explanatory diagram showing an example of a registration screen displayed on the registration device 5.

When starting to use the complex facility (or when the complex facility is not revisited within a predetermined period after registration), the user can perform a registration operation for authentication by the registration device 5. In the present embodiment, the user can perform the registration operation according to the guidance screen displayed on the display 42 (here, the touch panel) of the registration device 5. In addition, a part of the registration operation for authentication (such as pressing a button) described below may be performed by a receptionist who has obtained the consent of the user.

The display 42 of the registration device 5 displays an initial screen on which the registration start button 131 is displayed (see FIG. 16A). Then, when the user presses (touches) the registration start button 131, an explanation regarding the handling of personal information in the facility management system and a screen asking the user for their consent are displayed (see FIG. 16 (B)).

By pressing the consent button 132 on the screen of FIG. 16B, the user can consent to the handling of personal information in the facility management system. As a result, the registration device 5 starts acquiring the registrant information of the user. Therefore, the display 42 displays a screen showing a method of acquiring the registration information used for identifying the user (see FIG. 16C). The user can cancel the registration operation by pressing the cancel button 133. The screen for asking the user for consent may not require consent depending on the user and can be omitted. Further, the consent may be replaced by the operation of the OK button 135, the shooting button 137, and the registration button 141, which will be described later. In addition, consent may be obtained by a predetermined procedure before using the registration device 5.

In the example shown in FIG. 16C, a screen for inputting user registration information is displayed. Here, as registration information, the user's name, password (PW), e-mail address, selection of whether or not to request the information distribution service regarding the use of the complex, credit card number, credit card expiration date, etc. are entered. Will be done. Then, when the user who has completed the input of the registration information presses the OK button 135, the input of the registration information is completed.

Subsequently, the registration device 5 displays a screen showing a method of shooting a face image for user registration (guidance for positioning the user's face in the shooting area of the face shooting camera 41). (See FIG. 16 (D)). At this time, the registration device 5 can prompt the user to align the face by blinking the shooting frame 136 corresponding to the shooting area. A live image (real-time moving image) taken by the face shooting camera 41 is displayed in the shooting frame 136. Therefore, when the user positions his / her face in the shooting frame and presses the shooting button 137, the registration device 5 takes a picture of the user's face. At this time, the registration device 5 can generate a shutter sound when the shooting button 137 is pressed. The user can stop the shooting of the face by pressing the stop button 138.

Subsequently, the registration device 5 displays a screen for the user to confirm the captured face image (see FIG. 16E). A photographed face image (still image) is displayed on this screen, and the user presses the registration button 141 when it is determined that there is no problem with the photographed face image. On the other hand, the user can press the re-shooting button 142 when it is determined that there is a problem with the captured face image. As a result, the screen shown in FIG. 16D is displayed on the registration device 5, and the face can be re-photographed. The user can stop the shooting of the face by pressing the stop button 143.

When the face shooting is completed, the acquired face image and registration information are transmitted to the face management server 8. As a result, the face management server 8 executes the registration process (see FIG. 17) regarding the user. When the registration process of the face management server 8 is normally completed, the registration device 5 displays a screen indicating that the registration is completed (see FIG. 16F). Therefore, when the user presses the end button 145, the registration operation in the registration device 5 ends.

After the registration operation is completed, the registration device 5 can delete the face image and the registration information stored in the storage unit 44. As a result, it is possible to prevent the personal information from being stored in the registration device 5 used by an unspecified person for a long period of time, and it becomes possible to manage the personal information of the user more safely.

Instead of registering the user by the registration device 5 described above, the administrator prepares a face image file for user registration in advance and then registers the user using the management terminal 2. Can be done. Further, the administrator can also modify the registrant information registered by the registration device 5 or add new information from the management terminal 2 in order to supplement the registration by the registration device 5.

Next, the registration process for the user executed in response to the above-mentioned user registration operation will be described. FIG. 17 is a sequence diagram showing a procedure of registration processing for a user.

When the face management server 8 performs the user registration operation in the registration device 5 as described above, the face management server 8 starts the registration process for the user as shown in FIG. At this time, the face management server 8 receives the photographed face image (face image for registration) and the registration information from the registration device 5 together with the operation information of the user. In this registration process, the face management server 8 first sends a face image extraction request to the face matching server 9. This request includes a photographed image of the user's face acquired from the registration device 5.

When the face matching server 9 receives a request for extracting a face image from the face management server 8, the face matching server 9 performs a face image extraction process. In this face image extraction process, the photographed image of the user's face acquired from the face management server 8 is subjected to processing such as face detection and face cutting to extract the user's face image. Then, the response of the face image extraction is sent to the face management server 8. This response includes a user's face image.

When the face management server 8 receives the response of the face image extraction from the face matching server 9, it sends a face registration request to the face matching server 9. This request can include a user's face image acquired from the face matching server 9.

The face management server 8 omits the above-mentioned request for face image extraction to the face matching server 9, and requests the face matching server 9 to register the photographed image of the user's face acquired together with the operation information from the registration device 5. May be added to.

When the face matching server 9 receives the face registration request from the face management server 8, the face registration process is performed. In this face registration process, a face feature amount is generated from a user's face image, and the face feature amount is registered in a database. At this time, a face registration ID is assigned in association with the face feature amount of the user. Then, the face registration response is sent to the face management server 8. This response includes a result indicating whether or not the face registration is normally completed and a face registration ID. With the completion of registration of the face feature amount in the database, the face image of the user on the face matching server 9 is deleted.

The face management server 8 receives the face registration response from the face matching server 9, and if the process is completed normally, performs the registration process for the user. In this registration process, the user's face image acquired from the registration device 5 is registered in the face information database. Further, the face management server 8 associates the user registration information acquired from the face matching server 9 with the face image and registers it in the face information database. Further, the face management server 8 registers the face registration ID issued by the face matching server 9 in the face information database as user information.

Next, the process of face recognition of the user in the face recognition machine 1 will be described. FIG. 18 is an explanatory diagram showing a first example of face recognition processing for a user. Here, an example is shown in which a gate device (security gate) that manages the passage of the user is used as the external device 16 that cooperates with the face recognition machine 1.

When the face authentication machine 1 detects the face of one or more persons from the captured image of the camera 11 and acquires the face image, the face matching request for each person is transmitted to the face matching server 9. This face matching request includes the device ID of the requesting face recognition machine 1, data of the face image (face image for authentication) of the person to be face-authenticated, face frame information, and matching conditions. ..

When the face matching server 9 receives the face matching request, the face feature amount generation unit 84 first generates the face feature amount of the target person from the face image of each target person acquired from the face recognition machine 1. Next, the face matching server 9 collates the face feature amount of the target person with the face feature amount of the user registered in the face matching database in the face feature amount matching unit 86. When the face matching process is completed, the face matching server 9 transmits the face matching response to the requesting face recognition machine 1. The face matching response includes a matching result (success, failure), a matching score, and a user code.

The face feature amount matching unit 86 calculates a matching score indicating the degree of similarity between the face recognition target person and the registered user, and when this matching score is equal to or higher than a predetermined reference value, the face recognition is performed. The target person is regarded as the registered user, and a matching result indicating that the face matching is successful is generated. On the other hand, if the matching score for all users does not exceed the reference value, it is determined that the target person for face recognition is not a registered user, and a matching result indicating that face matching has failed is generated. ..

The face matching processing conditions (face matching parameters) performed by the face matching server 9 may be added to the matching request. As a result, the face recognition machine 1 can instruct the processing content of the face matching performed by the face matching server 9. For example, as a processing condition for face matching, a threshold value for the matching score is specified, and a matching result having a matching score equal to or higher than a predetermined threshold value is included in the response. In addition, as a processing condition for face matching, the number of matching results is specified, and a predetermined number of matching results are included in the response from the one with the highest matching score.

Further, when the face recognition machine 1 receives the face matching response indicating the success of face recognition from the face matching server 9, the face recognition machine 1 transmits an opening / closing control signal of the door portion to the linked gate device. That is, the face recognition machine 1 functions as a control device (gate opening / closing control device) that converts the face matching response into a door opening / closing control signal, thereby controlling the operation of the door portion of the linked gate device. As a result, the gate device operates to open the door portion (that is, allow the user to pass through) by the drive unit 92. The face recognition machine 1 deletes the user's face image stored in the storage unit 14 when the transmission of the control signal (open / close control signal) is completed. Here, the response of face matching is converted into the opening / closing control signal of the door portion, but when the opening / closing control of the door portion of the gate device is performed by the entrance / exit management system constructed as an external system, the opening / closing control of the door portion is performed. A control signal is transmitted to the entry / exit management system so that the face recognition result is reflected in the opening / closing control of the gate device.

FIG. 19 is an explanatory diagram showing a second example of the face recognition process for the user. FIG. 20 is an explanatory diagram showing an example of an authentication screen (screen at the time of authentication) displayed on the face recognition machine 1. Here, an example is shown in which a lock device provided on the door of a hotel guest room is used as the external device 16 that cooperates with the face recognition machine 1. Regarding the second example of the face recognition process, detailed description of the same items as in the first example described above will be omitted.

The face recognition machine 1 is installed, for example, on the wall of a corridor facing a guest room. When the face recognition machine 1 detects the approach of a person by a motion sensor or the like (that is, detects a user), the face recognition machine 1 displays a standby screen (initial screen) of the authentication screen on the display 12 (see FIG. 20A). Here, the authentication screen (notification screen) displayed on the display 12 includes an image display area 151 that displays a live image (real-time moving image) of a predetermined shooting area taken by the camera 11, and the progress of the face recognition process. Includes an information display area 152 for displaying information indicating the above.

After that, when the face authentication machine 1 detects the face of one or more persons from the captured image of the camera 11 and acquires the face image, the face authentication machine 1 transmits a face matching request for each person to the face matching server 9. At this time, in the information display area 152, characters 153 and a figure 154 indicating that the user's authentication is in progress are displayed (see FIG. 20B). In the information display area 152, the display of one of the characters 153 and the figure 154 may be omitted. Further, in the case of the face authentication machine 1 not provided with a display, for example, an LED lamp is used to blink the LED during authentication and turn on the LED when the authentication is successful to notify the progress of face authentication. You may try to do it. Further, the progress of face recognition may be notified by the display color of the LED. Further, the display color of the LED may be changed depending on the success or failure of the user's face authentication.

When the face matching server 9 receives the face matching request, the face matching server 9 generates a face feature amount and executes the face matching process as in the first example described above. When the face matching process is completed, the face matching server 9 transmits the face matching response to the requesting face recognition machine 1.

Further, when the face recognition machine 1 receives the face matching response indicating the success of the face recognition from the face matching server 9, the character 157 and the graphic indicating that the user authentication is completed are displayed in the information display area 152 of the authentication screen. 158 is displayed (see FIG. 20C). At the same time, the face recognition machine 1 transmits a control signal (command to unlock the door) to the linked door lock device. That is, the face recognition machine 1 functions as a control device that controls the operation of the linked lock device. As a result, the lock device performs an operation of releasing the lock (that is, allowing the registrant to enter the room). The face recognition machine 1 deletes the user's face image stored in the storage unit 14 when the transmission of the control signal is completed.

The face recognition machine 1 can be linked with an external device that only performs information processing without mechanical operation like the above-mentioned gate device and door lock device. FIG. 21 is an explanatory diagram showing a third example of the face recognition process for the user.

Here, an example in which a payment device is used as the external device 16 that cooperates with the face recognition machine 1 is shown. Regarding the third example of the face recognition process, detailed description of the same matters as the above-mentioned first or second example will be omitted.

In the third example of the face authentication process, when the face authentication machine 1 receives the face verification response indicating the success of the face authentication from the face verification server 9, it transmits a control signal to the linked payment device. As a result, the payment device executes a payment process that links the usage fee data for the product purchased by the corresponding registrant and the service used (for example, provision of meals at a restaurant) with the authenticated user. .. As a result, the usage fee is added to the data of the amount billed to the user (for example, the amount billed by the credit card). The payment device transmits the result of the payment process to the face authentication server 3.

FIG. 22 is an explanatory diagram showing an example of information distribution processing to the registrant by the group management server 4. FIG. 23 is an explanatory diagram showing a comparison between information distribution to individual users (comparative example) and information distribution by the group management server 4.

Upon receiving an instruction from the management terminal 2, the group management server 4 acquires a group image (image taken by the camera 11) and transmits a face matching request for each person belonging to the group to the face matching server 9.

When the face matching server 9 receives the face matching request, the face matching server 9 generates a face feature amount and executes the face matching process as in the example shown in FIG. When the face matching process is completed, the face matching server 9 transmits the face matching response to the face recognition machine 1.

Based on the face matching response, the group management server 4 extracts the corresponding group members in the group list information when one or more registrants are included in the group, and the group members are the registrants. Add an identifier indicating that there is.

Subsequently, the group management server 4 generates (or selects) information for the group according to the group division of the group. At the same time, the group management server 4 transmits a request for whether or not to distribute the information to the face management server 8. This request asks whether a registrant belonging to the group wishes to distribute information.

When the face management server 8 receives the request for whether or not to distribute the information, the face management server 8 confirms whether or not the target registrant wants to distribute the information based on the registered information, and responds whether or not the information can be distributed including the result. To the group management server 4.

The group management server 4 transmits information for the group only to the mobile terminal 10 of each registrant who wishes to distribute the information, based on the response of whether or not the information can be distributed received from the face management server 8. When the group includes a plurality of registrants who wish to distribute information, the group management server 4 transmits information for the group only to a predetermined mobile terminal 10 representing the registrants belonging to the group. May be good.

Here, for example, as shown in FIG. 23, an example of distributing information about a restaurant to a group consisting of three generations of family members (1)-(6) will be shown. Group members (1)-(3) are registrants. Group members (4)-(6) are unregistered users or do not have a mobile terminal 10 (the application is not used).

When distributing information about restaurants individually to group members (1)-(6) (see (A)), for example, group members (1) (male in their 40s) are guided to the bar counter. Information for group members (2) (women in their 40s) to guide to casual French restaurants, group members (3) (junior high school girls) to guide to family restaurants Are delivered respectively. Information is not distributed to registrants (4)-(6).

On the other hand, the group management server 4 distributes information about restaurants to the group members on a group basis (that is, according to the group classification) (see (B)), so that the group members (1)-(3). ), Information for guiding a special course for families offered at a high-class sushi restaurant is delivered to the mobile terminal 10. In this way, the group management server 4 can provide desired information to the users who act in the group in a timely manner. In addition, the possibility that the group receiving the information distribution will use the information increases.

FIG. 24 is an explanatory diagram showing an example of group action information generation processing by the group management server 4.

Upon receiving the instruction from the management terminal 2, the group management server 4 sends and receives a face matching request and a face matching response to and from the face matching server 9 as in the case of FIG. 22. Further, based on the face matching response, the group management server 4 adds an identifier indicating that the corresponding person is a registrant in the group list information when one or more registrants are included in the group. ..

Subsequently, the group management server 4 selects a registrant to be observed in the group (here, acquisition of location information and payment information is required). At this time, the group management server 4 can select one or more registrants in the group based on the age and gender of each group member. After that, the group management server 4 transmits a request for location information and payment information regarding the selected registrant to the face matching server 9.

When the face matching server 9 receives the request for the location information and the payment information, the face matching server 9 searches for the location information and the payment information of the target registrant. Subsequently, the face matching server 9 transmits the search result to the group management server 4 as a response of the location information and the payment information.

The group management server 4 records the response of the location information and the payment information as the group action information associated with the target group (that is, generates or updates the group action information).

FIG. 25 is an explanatory diagram showing an example of screen display processing for observing the behavior of the group on the management terminal 2. 26, 27, 28, 29, and 30 are explanatory views showing an example of a display screen of group behavior, respectively.

The management terminal 2 transmits a request for aggregated data related to group behavior to the group management server 4 based on the operation of the administrator (or at a predetermined cycle set in advance).

When the group management server 4 receives the request for the aggregated data, the group management server 4 aggregates the data in response to the request. The group management server 4 transmits the aggregation result to the management terminal 2 as a response of the aggregation data related to the group behavior. At this time, the group management server 4 transmits a part of the group action information already stored in the storage unit 102 to the management terminal 2 as a response of the aggregated data related to the group action, without newly totaling the data. May be good.

When the management terminal 2 receives the response of the aggregated data related to the group action, the management terminal 2 displays the display screen of the group action on the display 32.

As shown in FIG. 26, the group action display screen includes a map 161 showing the usage status of the complex facility. The map 161 shows the icons of each facility constituting the complex facility (see facility AJ included in the facility classification display column 162 in the figure) and the users of each facility AJ displayed near the icon. A pie chart 163 showing the distribution of the number of stays (number of stays) is included. Each pie chart 163 shows the percentage of the number of members of the group corresponding to each group division (see group division display column 164). The size of the pie chart 163 is set according to the number of users using each facility AJ. The group classification also includes a single user (one person).

In addition, the group action display screen includes a facility usage status display column 165 for each group. The facility usage status display column 165 includes a bar graph showing the distribution of the number of users of each facility AJ for each group category.

In addition, the group action display screen includes a time scale setting field 167, a country of residence setting field 168, and a consumption amount setting field 169. The administrator can set the time range of the data to be displayed in the map 161 and the facility usage status display field 165 in the time scale setting field 167. The administrator can set a desired date on the calendar 171 and set a desired time (or time zone) by operating the slider 172. The administrator can also cancel the setting of the time range (at least one of the date and time).

For example, as shown in FIG. 27, when the administrator changes the time by operating the slider 172, the pie chart 163 of the map 161 and each bar graph of the facility usage status display column 165 change according to the time. This allows the manager to observe the behavior of the group in the complex at the desired date and time.

In addition, the manager can set the nationality of the member of the group to be observed in the country of residence setting column 168.

In addition, the manager can set the range of the consumption amount of the group to be observed (at least one of the lower limit and the upper limit of the consumption amount) in the consumption amount setting column 169. For example, the consumption amount of the group is the total settlement amount of the group members who are registrants.

For example, as shown in FIG. 28, when the administrator changes the range of the consumption amount by operating the slider 173, the pie chart 163 of the map 161 and the display column 165 of the facility usage status are displayed according to the range of the consumption amount. Each bar graph changes. This allows the manager to observe the behavior of the group according to the amount of consumption in the complex.

As shown in FIG. 29, the administrator displays a pop-up screen 175 showing the details of the consumption amount of the group at the facility by pressing any of the facilities AJ (here, facility I) on the map 161. Can be made to. This pop-up screen displays the amount of consumption of registrants belonging to a predetermined group. In this way, the manager can grasp the breakdown of the consumption amount in each group (here, the consumption amount of each group member who is a registrant).

Further, by pressing the movement display button 178 on the pop-up screen 175, the administrator can display the movement history of the group on the map 161 as shown in FIG. Here, the dashed arrow in the map 161 indicates that the group has moved from facility B (luxury hotel) to facility E (amusement park) and then from facility E to facility C (luxury sushi restaurant). It is shown. The administrator is not limited to the operation of the movement display button 178, but by selecting a desired group from the group list separately displayed on the group action display screen, the movement history of the group is displayed in the map 161. Can be made to. Further, the display of the movement history of the group is not limited to the display on the map 161 and may be, for example, a table showing the history of facilities visited by the group as shown in FIG.

In FIG. 30, by selecting the destination of the group, the group can confirm the payment amount at the destination, but a list of payment amounts at the nuclear facility by the group may be displayed. For example, as shown in FIG. 14, a table of payment amounts at each facility of the group is displayed. In addition to the payment amount, a list of information related to the group may be displayed. With such a display, detailed information of the group can be analyzed, and the behavior of the group can be easily grasped.

FIG. 31 is an explanatory diagram showing an example of a group integration process in which a plurality of groups are set as a large group.

Upon receiving the instruction from the management terminal 2, the group management server 4 sends and receives a face matching request and a face matching response to and from the face matching server 9 as in the case of FIG. 22. Further, based on the face matching response, the group management server 4 adds an identifier indicating that the corresponding person is a registrant in the group list information when one or more registrants are included in the group. ..

Subsequently, the group management server 4 extracts a plurality of groups to which the same registrant belongs from the plurality of groups in the group list information. After that, the group management server 4 sets the extracted plurality of groups as a large group.

FIG. 32 is an explanatory diagram showing an example of a personal connection chart creation process by the group management server 4. FIG. 33 is an explanatory diagram showing an example of a personal connection chart.

Upon receiving the instruction from the management terminal 2, the group management server 4 sends and receives a face matching request and a face matching response to and from the face matching server 9 as in the case of FIG. 22. Further, based on the face matching response, the group management server 4 adds an identifier indicating that the corresponding person is a registrant in the group list information when one or more registrants are included in the group. ..

Next, the group management server 4 extracts one or more registrants included in the target group. Subsequently, the group management server 4 transmits a request for SNS (social networking service) information regarding each of the extracted registrants to the SNS server 104.

When the SNS server 104 receives the request for the SNS information, the SNS server 104 searches for a connection between each target registrant and another SNS user. Subsequently, the SNS server 104 transmits the search result of the connection with another SNS user to the group management server 4 as a response of the SNS information.

The group management server 4 creates a personal connection chart based on the response of the SNS information. At this time, the group management server 4 recognizes the connection between the registrant who is an SNS user and another SNS user as a personal connection. The administrator can display the personal connection chart on the display 32 of the management terminal 2, for example, as shown in FIG. 33. In FIG. 33, a personal connection chart regarding the registrants 181 and 182 and the user (non-registered person) 183, which are surrounded by a two-dot chain line and are determined to belong to the same group from the face recognition result for the group image, is shown. At this time, the connection with the thick line is the connection determined to be a group from the face recognition result for the group image, and the thin line indicates the connection on the SNS.

In addition, a user (non-registered person) 183 who is an SNS user also belongs to this group. The user 183 and the registrant 182 are determined to belong to the same group based on the face recognition result for the group image and are related to each other. If a user (non-registered person) 183 is found in the SNS-related personal connections of the registrant 182, the SNS account is linked to the user (non-registered person) 183 and saved. As a result, in the facility management system, the relationship between the members 181 to 183 determined to belong to the same group from the captured images becomes clearer. You can also find information about the user (non-registered) 183.

Further, in FIG. 33, the administrator can grasp that both the registrants 181 and 182 are connected to the person 185. Further, in the facility management system, when the person 185 uses the complex facility in the future, the information having the same or similar contents as the information distributed to the registrants 181 and 182 can be distributed to the person 185. ..

In addition, the connection between the registrant 186 and the user (non-registered person) 187 who are determined to belong to the same group from the face recognition result for another group image surrounded by a broken line can be found by using the SNS information. be able to. In other words, it is possible to find connections between groups that could not be found by face recognition grouping alone.

FIG. 34 is an explanatory diagram showing an example of a screen for managing registration information displayed on the management terminal 2.

When managing the registration information, the administrator can display the management screen of the registrant shown in FIG. 34 on the display 32 of the management terminal 2.

The registrant management screen shown in FIG. 34 is for the administrator to perform operations of registering, referencing, updating, deleting, and searching the face image and registration information for authentication relating to the registrant. The information (including images) input to the management screen of the registrant includes the information acquired by the registration device 5.

The registrant management screen includes an operation selection area 191 and a user information area 192.

In the operation selection area 191 the administrator can select items to be executed (information registration, reference, update, deletion, search, etc.).

The user information area 192 includes a registrant's face image, first and last name, and information used for face recognition. Here, in the user information area 192, the face image display unit 201, the first and last name input unit 202, the authenticated person (registrant) ID input unit 203, the identification ID input unit 204, the additional information input unit 207, and the activation date input. A unit 208 and an invalidation date input unit 209 are included.

The face image of the registrant is displayed on the face image display unit 201. The administrator can delete the face image of the face image display unit 201 by pressing the delete button 211. When the face image cannot be acquired by the registration device 5, the administrator can select the prepared face image (image file) as a display target by pressing the image selection button 212. Alternatively, the administrator can display the face image selected by the image selection button 212 on the face image display unit 201 instead of the face image acquired by the registration device 5.

The user's first and last name is input to the first and last name input unit 202. A registrant ID (for example, a driver's license number, my number, etc.) is input to the authenticated person ID input unit 203. In the identification ID input unit 204, an ID that can identify the registrant (for example, a series of unique numbers assigned in the order of registration of the user) is input. The additional information input unit 207 can input additional information. In the activation date input unit 208, a date on which face matching of the person can be enabled is input. In the invalidation date input unit 209, a date on which face matching can be invalidated is input. By setting these activation dates and invalidation dates, it is possible to start and end services for a plurality of users at the same time.

Further, in the present embodiment, when there is an instruction from the management terminal 2 regarding the information distribution process for the registrant, the group action information generation process, the group integration process for setting a plurality of groups as a large group, and the personal connection chart creation process. However, it is not limited to the instruction from the management terminal 2, such as setting at the time and processing when a change in the group is detected.

In the present embodiment, for the information distribution process to the registrant, the group action information generation process, the group integration process for setting a plurality of groups as a large group, and the personal connection chart creation process, a group image is acquired for each process and the group list is displayed. However, it is also possible to create a group list in advance and use the information in the group list. Specifically, a group image is acquired by the first gate device for entering the building of the complex facility where each facility is gathered, and a group list is created. After that, at each facility in the complex facility, it is assumed that the group to which the individual belongs is doing the same behavior by utilizing the image of the individual taken by face payment etc. without acquiring the group image. Then, it is conceivable to save the action history of the group. By doing so, it is not necessary to install a camera for taking a group image at each facility, and the behavior of the group can be tracked from information such as a gate device for one person and a face payment terminal. Alternatively, the complex may be divided into areas and the behavior of the group may be tracked by utilizing a gate device that passes when entering or exiting the area. Specifically, areas such as an amusement park area, a restaurant area, and a casino area are provided, and groups are tracked by utilizing images of gate devices at the entrances and exits of the areas. By doing so, it is possible to grasp in which area in the complex the group acted, and it is possible to analyze the preference of the group.

As described above, embodiments have been described as examples of the techniques disclosed in this application. However, the technique in the present disclosure is not limited to this, and can be applied to embodiments in which changes, replacements, additions, omissions, etc. have been made. It is also possible to combine the components described in the above embodiments to form a new embodiment.

The information distribution system and information distribution method according to the present disclosure have an effect of being able to provide desired information to users of a complex facility acting as a group in a timely manner, and in a complex facility including a plurality of facilities. It is useful as an information distribution system and an information distribution method that distributes information to users who act in a group.

1: Face recognition machine 2: Management terminal 3: Face recognition server (face recognition device, registration management device)
4: Group management server (group judgment device, information distribution device, mobile information generator)
5: Registration device 6: Facility display (display device)
7: Facility camera (shooting device)
8: Face management server 9: Face matching server 10: Mobile terminal 11: Camera (shooting device)
12: Display 13: Communication unit 14: Storage unit 15: Control unit 16: External device 17: Control signal transmission unit 18: App 22: Face image extraction unit 23: Face verification request unit 24: Authentication result notification unit 25: Authentication result Adjustment unit 31: Communication unit 32: Display (display device)
33: Input device 34: Storage unit 35: Control unit 38: GUI control unit 40: Input device 41: Face shooting camera 42: Display 43: Communication unit 44: Storage unit 45: Control unit 46: Face image extraction unit 47: Registration information acquisition unit 48: Registration request unit 49: GUI control unit 51: Communication unit 52: Storage unit 53: Control unit 62: Registrant management unit 65: Face matching server management unit 66: Face information management unit 67: Database management unit 71: Communication unit 72: Storage unit 73: Control unit 82: Image quality check unit 83: Face image extraction unit 84: Face feature amount generation unit 86: Face feature amount collation unit 88: Database management unit 89: Position information acquisition unit 90 : Payment information acquisition unit 91: Control signal reception unit 92: Drive unit 93: Power supply unit 94: Control unit 97: Power supply unit 100: Group 100A: Registrant 101: Communication unit 102: Storage unit 103: Control unit 104: SNS server 111: Group judgment unit 112: Group division unit 113: Specific person extraction unit 114: Group list creation unit 115: Group action totaling unit 116: Information distribution unit 121: Group customer 151: Image display area 152: Information display area 161: Map 162: Facility classification display field 163: Circle graph 164: Group classification display field 165: Display field 167: Time scale setting field 168: Country of residence setting field 169: Consumption amount setting field 171: Calendar 172: Slider 173: Slider 175: Pop-up Screen 178: Move display button 181: Registrant 182: Registrant 183: User

Claims (8)

An information distribution system that provides information to users of complex facilities that act as a group.
A registration management device that acquires a face image for registration of the user registered as a registrant of the complex facility and registration information of the user other than the face image, respectively.
An imaging device that generates a group image by photographing the group, and
A face recognition device that executes face recognition processing based on the face image for authentication of the user and the face image for registration acquired from the group image, and a face recognition device.
Based on the group image, a group determination device for determining the group division for distinguishing the group according to the difference in its properties, and a group determination device.
When the registrant of the complex facility is included in the group image, the information for the group generated or selected according to the determination result of the group classification is transmitted to the mobile terminal of the registrant based on the registration information. Information distribution device to distribute and
With
The registration information includes information on whether or not the registrant wishes to distribute the information.
The information distribution device is an information distribution system that distributes information for the group to the mobile terminal of the registrant only when the registrant desires the information distribution. The information distribution system according to claim 1, wherein the registration management device acquires the face image for registration and the registration information from the mobile terminal. The information distribution system according to claim 1 or 2, wherein the information for the group is generated or selected according to the shooting position of the group image. The information distribution system according to any one of claims 1 to 3, wherein the information for the group is information about one or more facilities included in the complex facility determined according to the group classification. The information distribution system according to claim 4, wherein the one or more facilities are determined according to the usage status of each facility included in the complex facility by a plurality of other groups. The information distribution system according to claim 5, wherein the usage status of each facility includes the number of users of each facility. The information distribution system according to claim 6, wherein the usage status of each facility relates to a predetermined time zone. It is an information distribution method that provides information to users of complex facilities that act as a group.
The face image for registration of the user registered as the registrant of the complex facility and the registration information of the user other than the face image are acquired respectively.
By photographing the group, a group image is generated.
Based on the face image for authentication of the user and the face image for registration acquired from the group image, the face authentication process is executed.
Based on the group image, the group division for distinguishing the group according to the difference in its properties is determined.
When the registrant of the complex facility is included in the group image, the information for the group generated or selected according to the determination result of the group classification is transmitted to the mobile terminal of the registrant based on the registration information. Deliver and
The registration information includes information on whether or not the registrant wishes to distribute the information.
The information for the group is an information distribution method in which the information for the group is distributed to the mobile terminal of the registrant only when the registrant desires the information distribution.

Images