JP2023074751A - Information processing system, information processing device, information processing method, and program - Google Patents

Abstract

To provide an information processing system for suppressing the occurrence of false recognition and reducing processing costs, a device, a method, and a program.SOLUTION: In a recognition device 12, an image acquisition unit 400 receives images captured by an in-store imaging device or a cash register imaging device and imaging data including imaging times of the images, and sequentially transmits the received imaging data to a face detection unit 401. The face detection unit performs face detection processing on the images received by the image acquisition unit, and extracts an area including the face in the images as a boundary frame of the face. In the face detection processing, the face detection unit 401 acquires information indicating feature points of facial organs and the likelihood of the facial organs, along with information on coordinates, width, and height of the boundary frame of the face. A feature quantity extraction unit 402 includes a high-precision processing unit 408 configured to extract a high-precision feature quantity being a second feature quantity through second extraction processing, and a medium-precision processing unit 409 configured to extract a medium-precision feature quantity being a first feature quantity through first extraction processing.SELECTED DRAWING: Figure 7

Description

The present invention relates to an information processing technology that can be used for person verification using person recognition.

Patent Literature 1 discloses a payment system using face recognition. In Patent Literature 1, a list of members currently visiting the store is created by collating face images of customers captured using an in-store camera with face images of all registered member information. Then, at the time of payment at the cash register, matching is performed using the face image of the list of members who are visiting the store and the face image of the customer who has come to the cash register to identify the member. Japanese Patent Laid-Open No. 2002-200002 discloses using a multi-layered neural network called a deep neural network as a technique for performing face detection processing and feature amount extraction processing from image data.

JP 2018-101420 A Japanese Patent Application Laid-Open No. 2020-030480

Generally, in-store cameras are often arranged at high positions, and the image captured by the in-store camera is a bird's-eye view image of the store (bird's-eye view image). However, since the face image of the person in the bird's-eye view image is an image that can be seen obliquely from above, there is a possibility that the matching accuracy will be lower than that of the face image of the person captured from the front, resulting in erroneous recognition. Here, when a member list as disclosed in Patent Document 1 is created in a state where a person is erroneously recognized, an attempt is made to re-verify using the face images of customers who come to the cash register at the time of payment at the cash register. may fail to rematch without finding a matching member from the member list. In addition, depending on the size of the store, the number of in-store cameras and cameras in front of the cash register may increase. As the number of cameras increases, the processing cost for face recognition using face images increases.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to suppress the occurrence of erroneous recognition and to reduce the processing cost.

The present invention is an information processing system including a first information processing device and a second information processing device, wherein the first information processing device extracts face images of a plurality of persons by a first extraction process. Acquiring person information including the extracted first feature information and second feature information extracted by a second extraction process having a larger amount of processing than the first extraction process, and photographing the person a first matching process of comparing the third feature information extracted from the first face image by the first extraction process and the first feature information included in the person information to obtain a degree of similarity; and transmits the person information including the degree of similarity obtained by the first matching process to the second information processing device, and the second information processing device receives the similarity from the first information processing device. a list including the first feature information and the second feature information according to the similarity is created from the person information; and a second facial image of the person is created The person in the second face image is identified by comparing the fourth feature information extracted by the second extraction process with the second feature information in the list in descending order of similarity. It is characterized by performing a second collation process to specify from the information.

According to the present invention, the occurrence of erroneous recognition can be suppressed, and the processing cost can be reduced.

1 is a schematic configuration diagram of an information processing system; FIG. It is an image figure which shows an example of each arrangement|positioning in a store. It is a figure which shows an example of the hardware constitutions of a management apparatus. It is a figure which shows an example of the hardware constitutions of a recognition apparatus. It is a block diagram which shows the functional structure of a management apparatus. It is a figure which shows an example of a member information list. It is a block diagram which shows the functional structure of a recognition apparatus. It is a figure which shows an example of the visitor collation list|wrist which concerns on 1st Embodiment. FIG. 10 is a diagram used for explaining facial organ detection; It is a flow chart of member registration processing. 7 is a flow chart at the time of first face matching according to the first embodiment; 10 is a flow chart for second face collation. FIG. 11 is a flow chart at the time of first face matching according to the second embodiment; FIG. FIG. 13 is a diagram showing an example of a visitor verification list according to the third embodiment; FIG. It is a figure which shows the functional structure of the information processing apparatus which concerns on 4th Embodiment. 14 is a flowchart of information processing according to the fourth embodiment; FIG. 14 is a sequence diagram of information processing according to the fourth embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments according to the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all combinations of features described in the embodiments are essential to the solution of the present invention. The configuration of the embodiment can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (use conditions, use environment, etc.). Also, a part of each embodiment described later may be appropriately combined. In each of the following embodiments, the same configurations are given the same reference numerals.

In this embodiment, an information processing system including an information processing device for performing settlement using face recognition will be described as an example. In an information processing system that uses face recognition to make payments, a member's face image and payment information are linked in advance and registered as personal information (hereinafter referred to as member information), and is installed at the cash register of a store, etc. A customer's face image captured by a camera is collated with a face image registered in member information. Then, when the collation is successful, settlement processing is executed based on the settlement information associated with the face image that has been collated. If there is a large amount of registered member information, it takes a long time to process the checkout at the cash register, and the settlement time at the cash register becomes long. It is considered possible. However, with a bird's-eye view image obtained by a general in-store camera, the accuracy of matching may decrease and misrecognition may occur. As mentioned above, re-matching at the time of payment at the cash register may fail. In addition, as the number of in-store cameras and cameras installed in front of cash registers increases depending on the size of the store, the processing cost for face recognition using face images increases.
Therefore, the information processing system of the present embodiment has the configuration described below as a configuration for performing settlement processing using face recognition. It is possible to suppress the generation and reduce the calculation processing cost.

<First embodiment>
FIG. 1 is a diagram showing a configuration example of an information processing system according to the first embodiment.
In the information processing system of this embodiment, the management device 10 as the first information processing device and the recognition device 12a as one of the second information processing devices are connected via the network 11 . Similarly, the management device 10 and the recognition device 12b, which is one of the second information processing devices, are connected via the network 11. FIG. The recognition device 12a, the in-store imaging device 14a, and the cash register imaging device 15a are connected via a network 13a, and similarly, the recognition device 12b, the in-store imaging device 14b, and the cash register imaging device 15b are connected via a network 13b. . In FIG. 1, two examples are given, such as the recognition devices 12a and 12b, the in-store imaging devices 14a and 14b, the register imaging devices 15a and 15b, and the networks 13a and 13b. or three or more. The network 11 and the networks 13a and 13b are realized by a plurality of routers, switches, cables, etc. complying with communication standards such as Ethernet (registered trademark). Also, the network 11 and the networks 13a and 13b may be realized by the Internet, wireless LAN (WIRELESS LAN), WAN (WIDE AREA NETWORK), or the like. Hereinafter, when the recognition devices 12a and 12b, the in-store imaging devices 14a and 14b, the cashier imaging devices 15a and 15b, and the networks 13a and 13b are not distinguished from each other, the recognition device 12, the in-store imaging device 14, and the cash register imaging device will be referred to as the recognition device 12a and 12b. 15 and network 13 only.

FIG. 2 is a diagram showing an example of an image of a store to which the information processing system of this embodiment is applied.
As shown in FIG. 2, the store 20 is provided with an in-store imaging device 202, a cash register imaging device 204, and a cash register 201. FIG. An in-store imaging device 202 corresponds to the in-store imaging device 14 in FIG. 1, and a cash register imaging device 204 corresponds to the cash register imaging device 15 in FIG. The in-store imaging device 202 is arranged at a position where the inside of the store can be photographed from above, and the cash register imaging device 204 is arranged in front of the cash register 210 . In other words, the in-store imaging device 202 captures a bird's-eye view of a person (customer, etc.) who has come to the store and is present in the store, and the cash register imaging device 204 captures a person (customer about to make a payment) who has come to the cash register 210 from the front. . Although only one in-store imaging device 202 and one cash register imaging device 204 are illustrated in FIG. Further, the in-store imaging device 14 may be installed outside the store, such as near the store entrance, as well as inside the store. The cash register 201 is not limited to one, and may be plural. When a plurality of cash registers 201 are provided, the cash register imaging device 204 is arranged for each cash register.

The in-store imaging device 202 images a person (a visitor 203) in the store 20, and the cash register imaging device 204 images the face of a person 205 who is about to make a payment in front of the cash register 201. The image of the visitor 203 captured by the in-store imaging device 202 is sent to the recognition device 12 via the network 13 of FIG. A face image of a person 205 (person who is about to make a payment) captured by the register imaging device 204 is sent to the recognition device 12 via the network 13 in FIG. As another example, the register imaging device 15 may be provided together with the recognition device 12 in a smart phone, a tablet, or the like instead of being an independent device.

Returning to FIG.
The management device 10 is a device that stores and manages payment information such as credit card information, personal information such as names and ages, face images, and face information used for face recognition for all members. In the present embodiment, member information includes payment information of a certain member, personal information such as name and age, face image, face information used for face recognition, and the like. It is assumed that the management device 10 stores and manages member information of all members as a member information list in a database or the like. Details of the member information list will be described later.

Here, in the present embodiment, face information is feature information representing features of a face extracted from a face image by predetermined extraction processing, and is called a feature amount in the present embodiment. In the present embodiment, the process of extracting a feature amount (face information) from an image is exemplified by a process using a multi-layered neural network called a deep neural network trained using deep learning. Further, in the case of this embodiment, the processing for extracting the feature amount from the image includes the first extraction processing and the second extraction processing. The neural networks used for the first extraction process and the second extraction process have different circuit scales, and the second extraction process uses a neural network with a larger circuit scale than the first extraction process. In general, the circuit scale of a neural network has a corresponding relationship with the amount of processing by the neural network, that is, the processing cost. becomes less. Therefore, in the present embodiment, the second extraction process has a larger processing amount and higher processing cost than the first extraction process. Also, the feature quantity acquired by a neural network with a large circuit scale is generally a feature quantity with higher precision than the feature quantity acquired by a neural network with a smaller circuit scale. For this reason, in the present embodiment, the second feature quantity obtained by the second extraction process using a neural network with a large circuit scale is called a high-precision feature quantity (or high-precision face information). Further, in the present embodiment, the first feature amount obtained in the first extraction process using a neural network having a relatively smaller circuit scale than that used in the second extraction process is used as a medium-precision feature amount ( or medium-precision facial information).

The in-store imaging device 14 and the cashier imaging device 15 are devices (cameras) that respectively capture images, and may be network cameras, module cameras, or the like, and their forms are not particularly defined. The in-store imaging device 14 and the cash register imaging device 15 transmit imaged image data and imaging data including imaging time and the like to the recognition device 12 via the network 13 . In the following description, image data is simply referred to as an image for the sake of simplification.

The recognition device 12 acquires an image captured by an in-store imaging device 14 or a cash register imaging device 15 connected via a network 13, detects a face image of a person from the image, and further detects a face image from the detected face image. Extract features. Although the details will be described later, in the recognition device 12 of the first embodiment, when extracting the feature amount from the face image of the person (visitor 203) appearing in the image of the in-store imaging device 14, the first extraction A first feature amount obtained by processing, that is, a medium-precision feature amount (medium-precision face information) is obtained. Then, the extracted medium-precision feature amount is transmitted to the management device 10 via the network 11 .

When the management device 10 acquires the medium-precision feature quantity extracted by the recognition device 12 from the image captured by the in-store imaging device 14, the management device 10 combines the acquired medium-precision feature quantity and the medium-precision face registered in advance for the registered member. Face matching processing is performed using information (medium-precision feature amount). In the first embodiment, the management device 10 uses medium-precision feature amounts extracted from images captured by the in-store imaging device 14 and medium-precision face information (medium-precision feature amounts) registered as registered members. This face matching process is referred to as first face matching. The management device 10 then sends the result of the first face matching and the member information to the recognition device 12 . Although the details will be described later, in the present embodiment, a similarity score indicating the degree of similarity of facial feature amounts is used as the information indicating the result of the first face matching. Although details will also be described later, the member information sent to the recognition device 12 together with the result of the first face verification is selected from the member information list managed by the management device 10 according to the result of the first face verification. Made with membership information. The result of the first face collation and the member information are used as information held in a visitor collation list, which will be described later, in the recognition device 12 .

The recognition device 12 creates, holds, and manages a visitor verification list including the result of the first face verification by the management device 10 and the member information. Although the details will be described later, the store visitor matching list includes the similarity score obtained as a result of the first face matching and member information corresponding to the result of the first face matching. information, and a list containing at least face information used for face recognition.

After that, when the recognition device 12 acquires an image of the person in front of the cash register (the person 205 in FIG. 2) from the cash register imaging device 15 via the network 13, the recognition device 12 detects a face image from the image of the person, and further detects the face image. Extract features from images. Although the details will be described later, in the recognition device 12 of the present embodiment, when extracting the feature amount from the face image of the person (person 205 who is about to make a payment) in the image of the cash register imaging device 15, the second A second feature amount obtained by extraction processing, that is, a high-precision feature amount (high-precision face information) is obtained. Then, the recognition device 12 performs face recognition using the high-precision feature amount obtained from the face image of the person before the checkout and the high-precision face information (high-precision feature amount) included in the member information of the visitor verification list. . In this embodiment, face matching processing using high-precision feature amounts extracted from the face image of a person before the checkout and high-precision face information (high-precision feature amounts) included in the member information of the visitor verification list is performed by: This is referred to as second face collation. Based on the result of the second face verification, the recognition device 12 identifies whether the person before the checkout is a member of the member information of the visitor verification list.

Note that in the present embodiment, one management device 10 is installed in the data center, but a plurality of management devices may be installed, or may be placed in a store or on the cloud. In the above example, the recognition device 12 is installed in each store, for example, and is connected to the in-store imaging device 14 and the cash register imaging device 15 of each store. and so on. As still another example, a program for realizing each function related to the recognition device 12 may be installed in the cash register terminal, and in this case, the recognition device 12 is included in the cash register terminal. Further, in this embodiment, an example in which the management device 10 and the recognition device 12 are configured separately is given, but the management device 10 and the recognition device 12 may be an integrated device. The management device 10 in this case also executes each process including the second face matching related to the recognition device 12 .

FIG. 3 is a diagram showing an example of the hardware configuration of the management device 10. As shown in FIG.
As shown in FIG. 3 , the management device 10 has a CPU 101 , ROM 102 , RAM 103 , recording device 104 , communication device 105 , input device 106 and display device 107 .

The CPU 101 reads control programs recorded in the ROM 102 and executes various processes. A RAM 103 is used as a temporary storage area such as a main memory or a work area. The recording device 104 is composed of, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like, and is used to store image files and member information.
The communication device 105 is a circuit that performs communication, and the communication may be wireless communication or wired communication. In this embodiment, the communication device 105 is connected to the network 11 . The input device 106 may include a keyboard, a touch panel, and an imaging device (camera). The display device 107 includes a liquid crystal panel and an organic EL panel.

As illustrated in FIG. 3, the hardware configuration of the management device 10 has the same components as the hardware configuration installed in a personal computer (PC). Therefore, various functions realized by the management device 10 can be implemented as software that operates on the PC. The CPU 101 of the management device 10 executes a program, thereby realizing the function of FIG. 5 described later and the processing of the flowcharts of FIGS. 11 and 13 described later.

FIG. 4 is a diagram showing an example of the hardware configuration of the recognition device 12. As shown in FIG.
As shown in FIG. 4 , the recognition device 12 has a CPU 121 , ROM 122 , RAM 123 , recording device 124 , communication device 125 , input device 126 and display device 127 .

The CPU 121 reads control programs recorded in the ROM 122 and executes various processes. A RAM 123 is used as a temporary storage area such as a main memory or a work area. The recording device 124 is composed of, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like, and is used to store image files and member information.
The communication device 125 is a circuit for communication. The communication may be wireless communication or wired communication. In this embodiment, communication device 125 is connected to network 11 and network 13 . The input device 126 has a keyboard and a touch panel. The display device 127 includes a liquid crystal panel and an organic EL panel.

As illustrated in FIG. 4, the hardware configuration of the recognition device 12 has the same components as the hardware configuration installed in a personal computer (PC). Therefore, various functions realized by the recognition device 12 can be implemented as software that operates on a PC. The CPU 121 of the recognition device 12 executes a program, thereby realizing the function of FIG. 7 described later and the processing of flowcharts of FIGS. 11, 12, and 13 described later.

FIG. 5 is a functional block diagram showing an example of the functional configuration of the management device 10. As shown in FIG. The functional configuration of the management device 10 according to this embodiment will be described below with reference to FIG.
The member information registration unit 300 of the management device 10 of this embodiment holds a member information list of all members. In this embodiment, the member information registration unit 300 holds all member information lists. may be obtained.

FIG. 6 is a diagram showing an example of a member information list.
As shown in FIG. 6, the member information list includes member ID 501, payment information 502 such as credit card information, high-precision face information 503 used for face recognition, and medium-precision face information 504 used for face recognition.
The member ID 501 holds identification information for individually identifying each registered member.
The payment information 502 holds information that can be used for payment such as credit card information of each member.
The high-precision face information 503 holds high-precision face information (high-precision feature amount) extracted by using the second extraction process from the face image photographed in advance for each member.
The medium-precision face information 504 holds medium-precision face information (medium-precision feature amount) extracted by the first extraction process from the face image photographed in advance for each member.
Although not shown in FIG. 6, the member information list may further hold attribute information such as the member's name and age, face image, etc., in addition to the above-described information. In this embodiment, information for each member stored in the member information list is called member information. The member information in the member information registration unit 300 is input in advance from the keyboard of the input device 106 or is extracted from the captured image of the camera and stored. If possible, the input method and holding method are not limited.

Returning to FIG.
The face information acquisition unit 301 acquires face information (feature amount) extracted from the face image by the recognition device 12 and sends it to the face matching unit 302 . In the case of the first embodiment, the face information acquired by the face information acquisition unit 301 is obtained by detecting a face image from the image captured by the in-store imaging device 14 by the recognition device 12, and extracting a medium-precision face image by the first extraction process. Information.

The face matching unit 302 performs first face matching using the medium-precision face information acquired by the face information acquisition unit 301 from the recognition device 12 and the medium-precision face information of the member information held in the member information registration unit 300. I do. Then, the face matching unit 302 acquires a similarity score indicating the similarity of the face information as a result of the first face matching.

The information transmission unit 303 acquires from the member information registration unit 300 the member information corresponding to the face information for which the similarity score is obtained in the first face collation. Then, the information transmission unit 303 sends the similarity score, which is the result of the first face matching, and the member information acquired from the member information registration unit 300 corresponding to the face information of the similarity score to the recognition device 12. Send.

Next, the functional configuration of the recognition device 12 according to this embodiment will be described with reference to FIG. FIG. 7 is a functional block diagram showing an example of the functional configuration of the recognition device 12. As shown in FIG.
The image acquisition unit 400 of the recognition device 12 receives the images captured by the in-store imaging device 14 or the cashier imaging device 15 and the imaging data including the imaging times of the images and the like. 401. Note that the captured image may be a still image or a moving image. Hereinafter, in the present embodiment, still images and moving images are referred to as images without distinguishing between them, unless otherwise specified.

The face detection unit 401 performs face detection processing on the image received by the image acquisition unit 400, and extracts an area including a face in the image as a boundary frame of the face. In this embodiment, the so-called background subtraction method is used to extract the boundary frame of the face, but any method may be used as long as it detects the face and the position of the face. In the face detection process, the face detection unit 401 detects the coordinates, width, and height information of the face boundary frame 1500 as well as feature points of facial organs such as the eyes, nose, and mouth, as shown in FIG. and the likelihood indicating the likelihood of these facial features.

The feature amount extraction unit 402 is composed of a high-precision processing unit 408 and a medium-precision processing unit 409 .
A high-precision processing unit 408 extracts a high-precision feature amount (high-precision face information), which is a second feature amount, from the image within the boundary frame of the face extracted by the face detection unit 401 by a second extraction process. . A medium-precision processing unit 409 extracts a medium-precision feature amount (medium-precision face information), which is the first feature amount, from the image within the boundary frame of the face extracted by the face detection unit 401 by the first extraction processing. . In the case of this embodiment, the feature amount extraction unit 402 performs these extraction processes by arithmetic processing using a multi-layered neural network called a deep neural network trained using deep learning. The neural network model used in the high-precision processing unit 408 is larger in scale than the neural network model used in the medium-precision processing unit 409 . Therefore, the high-precision processing unit 408 can generally perform face recognition (face detection, feature amount extraction) with higher precision than the medium-precision processing unit 409, but the processing cost is high.

Here, in the case of this embodiment, in the feature quantity extraction unit 402 , the image acquisition unit 400 acquires the face image from the in-store imaging device 14 and the face image detected by the face detection unit 401 is processed by the medium-precision processing unit 409 . Accuracy feature amount calculation processing is performed. Then, the medium-precision feature amount obtained by the medium-precision feature amount calculation process, that is, the medium-precision feature amount extracted from the face image of the visitor captured by the in-store imaging device 14 is sent from the information transmission unit 403 to the management device 10. .
As a result, the management device 10 performs the above-described first face collation using the middle-precision feature amount (medium-precision face information) of the visitor obtained from the image of the in-store imaging device 14, and the first face matching is performed. The collation result and the member information are sent to the member information acquiring section 404 of the recognition device 12 .

The member information acquisition unit 404 acquires the result of the first face matching by the management device 10 and the member information, and outputs them to the list management unit 405 .
The list management unit 405 retains and manages the result of the first face matching in the management device 10 and the member information as a visitor matching list.

The visitor verification list held and managed by the list management unit 405 is a list having member IDs 801, payment information 802, high-precision face information 805, medium-precision face information 803, and similarity scores 804, as shown in FIG. is made. The store visitor matching list is a list based on the results of the first face matching process performed by the face matching unit 302 of the management device 10 for each store visitor. The similarity score 804 holds the results of the first face matching by the management device 10 in descending order of similarity score. The member ID 801 holds member identification information selected from the member information list shown in FIG. The payment information 802 holds the payment information selected from the member information list in FIG. The high-precision face information 805 holds the high-precision face information (feature amount) selected from the member information list in FIG. be done. The medium-precision face information 803 holds the medium-precision face information (feature amount) selected from the member information list in FIG. be done.

The number of member information included in the visitor verification list may be all the member information in the member information list. may be Also, the visitor verification list is updated by adding the member information of the visitor only when a visitor is newly discovered, and the member information of the visitor who can be tracked in the store is not updated as described later. do. Furthermore, the visitor verification list may be deleted when a visitor becomes untrackable in the store and a certain period of time has elapsed since the visitor was added to the list, or reset at the closing time of the store. good too. The visitor verification list may further include the time when the member information acquisition unit 404 acquired the member information, the image acquired by the image acquisition unit 400, and the like.

In this embodiment, the medium-precision face information calculated by the medium-precision processing unit 409 of the feature amount extraction unit 402 for the face image of the visitor detected by the face detection unit 401 from the image of the in-store imaging device 14 is It is also sent to the face matching unit 406 . At this time, the face matching unit 406 performs processing for tracking the visitor using medium-precision face information of the visitor. In the case of the present embodiment, one visitor verification list is associated with one visitor, so when the visitor being tracked comes in front of the register imaging device 15, the visitor can The second face matching is performed using the store visitor matching list linked to.

Here, in the present embodiment, the second face collation is performed by the high-precision processing unit 408 of the feature amount extraction unit 402 for the face image detected by the face detection unit 401 from the image obtained by the register imaging device 15. This is performed using the feature amount (face information) extracted by performing the high-precision feature amount calculation process. That is, when the tracked visitor comes to the front of the cash register, the face matching unit 406 obtains high-precision face information from the face image of the visitor captured by the cash register imaging device 15 through high-precision feature amount calculation processing. The second face matching is performed using the (feature amount) and the store visitor matching list linked to the store visitor.

At the time of the second face matching, the face matching unit 406 extracts the high-precision face information (feature amount) of the payer detected from the image of the cash register imaging device 15 and acquired by the high-precision processing unit 408, and the store visitor matching list. matching is performed using high-accuracy feature quantities in descending order of similarity scores 804 of the . Then, the face matching unit 406 notifies the payment unit 407 of the member ID 801 and the payment information 802 corresponding to the member information whose similarity score is equal to or higher than a predetermined score threshold as a result of the second face matching.

As another example, the list management unit 405 may create one visitor verification list for a plurality of visitors. In that case, the visitor matching list stores the similarity score between any visitor in the store and the member. Then, in the second face matching, the face matching unit 406 performs matching in order from the member information with the highest similarity score in the visitor matching list, regardless of who the payer is. Further, in the present embodiment, the second face matching is performed in descending order of similarity score in the visitor matching list, but the second face matching is not performed for members whose similarity score is less than a separately defined threshold. may

An example of second face matching will be described with reference to the store visitor matching list shown in FIG. Further, here, an example in which 700 is set as a predetermined score threshold when it is determined that the recognition is successful (matching is successful), that is, if the similarity score is 700 or more, the recognition is determined to be successful. list.

First, the face matching unit 406 selects the member ID 00003 with the highest similarity score (780) among the similarity scores 804 in the store visitor matching list managed based on the result of the first face matching. . Furthermore, the face matching unit 406 uses the high-precision face information 805 associated with the member ID 00003 and the high-precision feature amount (face information) of the person extracted from the image of the cash register imaging device 15 to perform the second perform face matching. Assume that the similarity score obtained as a result of this second face collation is 680. In this case, since the similarity score of 680 obtained by the second face matching is less than the predetermined score threshold of 700, the face matching unit 406 determines that the person in front of the cash register is not the member ID 00003, that is, the payment has not been completed. It is determined that it is not a possible settlement person.

Next, the face matching unit 406 selects the member ID 00001 with the next highest similarity score (660) among the similarity scores 804 of the visitor matching list. Furthermore, the face matching unit 406 uses the high-precision face information 805 associated with the member ID 00001 and the high-precision feature amount (face information) of the person extracted from the image of the cash register imaging device 15 to perform the second perform face matching. Assume that the similarity score obtained as a result of this second face collation is 720. In this case, since the similarity score of 720 obtained by the second face matching exceeds the predetermined score threshold of 700, the face matching unit 406 determines that the person in front of the cash register is the person with the member ID 00001, that is, the payment It is judged that it is a possible settlement person.

In this way, when the member as the payer is specified as a result of the second face matching in the face matching unit 406, the payment unit 407 retrieves the payment information 802 corresponding to the member ID 801 of the specified member. payment processing. In the case of this embodiment, for example, in the settlement process, the settlement amount is received from a purchase amount totaling system (not shown) such as a cash register, and the settlement amount, member ID, and settlement information are transmitted to the settlement system (not shown). will be

FIG. 10 is a flow chart showing the flow of member registration processing executed by the management device 10 according to the first embodiment. In the case of this embodiment, member registration is a process performed before the member makes a payment at the store. It is assumed that the member registration process shown in the flowchart of FIG. 10 is started when the system administrator presses a registration process start button (not shown) on the management device 10 . It should be noted that the symbol S used in each flow chart and sequence diagrams below represents each step of processing.

In S501, high-precision face information and medium-precision face information acquired by performing high-precision feature quantity calculation processing and medium-precision feature quantity calculation processing on a still image of a person's face captured in advance by an imaging device (not shown) and medium-precision feature quantity calculation processing are performed. The accurate facial information is input to the member information registration unit 300 of the management device 10 .
In 501, the member information registration unit 300 acquires the member ID, payment information, and the like input in advance by the system administrator or the like via an input device such as a keyboard (not shown). Then, the member information registration unit 300 saves the high-precision face information and medium-precision face information input in S501 and the member ID, payment information, etc. acquired in S502 as member information in the member information list shown in FIG. do. The member information may be acquired via a network from a mobile terminal such as a mobile phone or smart phone owned by the member, or an information device such as a personal computer.

FIG. 11 is a flow chart showing the flow of the first face matching process using the image of the in-store imaging device 14, which is executed in cooperation by the recognition device 12 and the management device 10 according to this embodiment. It is assumed that the first face matching process shown in the flowchart of FIG. 11 is started by the system administrator pressing a first face matching process start button (not shown) on the recognition device 12, for example. In addition to the instruction from the system administrator, the recognition device 12 is pre-stored with the business start time of the store, and the recognition device 12 starts the first face matching process on condition that the business start time has come. It may be done as follows.

In S600, the image acquisition unit 400 of the recognition device 12 acquires an image of the inside of the store captured by the in-store imaging device 14 .
Next, in S601, the face detection unit 401 of the recognition device 12 performs face detection on the image received from the image acquisition unit 400, and acquires the boundary frame of the face as described with reference to FIG. 9A.
Further, in S602, the medium-precision processing unit 409 of the feature amount extraction unit 402 of the recognition device 12 extracts medium-precision feature amounts (face information) from the image within the boundary frame.
In S<b>603 , the information transmission unit 403 of the recognition device 12 transmits the medium-precision feature amount (face information) extracted in S<b>602 to the face information acquisition unit 301 of the management device 10 .

In the management device 10 , in S<b>604 , the face information acquisition unit 301 receives the medium-precision feature amount (face information) transmitted from the recognition device 12 .
Next, in S605, the face matching unit 302 of the management apparatus 10 combines the received medium-precision feature amount and the medium-precision face information (feature amount ) to obtain a similarity score.
In S606, the information transmission unit 303 of the management device 10 transmits the similarity score acquired in S605 and the member information acquired from the member information registration unit 300 together to the member information acquisition unit 404 of the recognition device 12. .

In the recognition device 12, the member information acquisition unit 404 receives the similarity score and member information transmitted from the management device 10 in S607.
Next, in S608, the list management unit 405 of the recognition device 12 saves the similarity score and member information transmitted from the management device 10 in the visitor verification list.

The recognition device 12 and the management device 10 continue to cooperate and repeat the above-described first face matching process while the in-store imaging device 14 is activated, so that all the people staying in the store Visitors are linked with registered member information.

In the present embodiment, an example has been described in which all visitors are linked with registered member information and registered in the visitor verification list. may be registered in the visitor verification list. As still another example, the management device 10 may execute part or all of the processing executed by the recognition device 12 in the flowchart of the first face matching processing. Alternatively, part or all of the processing executed by the management device may be executed by the recognition device 12 .

FIG. 12 is a flow chart showing the flow of processing in the second face collation executed by the recognition device 12 according to this embodiment using the image of the register imaging device 15 . Assume that the second face matching process shown in the flowchart of FIG. 12 is started by the system administrator pressing a second face matching process start button (not shown) on the recognition device 12 . In addition to the instruction from the system administrator, the recognition device 12 is stored in advance with the business start time of the store, and the recognition device 12 starts the second face matching process on the condition that the business start time has come. It may be made to do.

In S<b>700 , the image acquisition unit 400 acquires, from the cash register imaging device 15 , a front image of the face of a person who is going to pay for a product at the cash register.
Next, in S701, the face detection unit 401 performs face detection on the image received from the image acquisition unit 400, and acquires the boundary frame of the face.
Further, in S702, the feature amount extraction unit 402 causes the high-precision processing unit 408 to extract a high-precision feature amount (face information) from the image of the boundary frame.

Next, in S<b>703 , the face matching unit 406 compares the high-precision feature amount acquired in S<b>702 with the member information with the highest similarity score among member information in the visitor matching list managed by the list management unit 405 . A similarity score is obtained by comparing with the accurate face information (feature amount). That is, in S703, the face matching unit 406 selects the high-precision feature amount (face information) acquired in S702 in order from the high-precision face information of the member information with the similarity score among the high-precision face information in the visitor verification list. perform matching. As a result, it is specified which member information in the store visitor collation list the person appearing in the image of the cash register imaging device 15, that is, the person who is about to make a payment at the cash register.

In the present embodiment, as described above, the image of the in-store imaging device 14 is used to track the visitor, and while the tracking is being performed, the first face matching is performed using the medium-precision feature amount. One store visitor matching list is generated based on one face matching result and linked to the store visitor. Then, in this embodiment, when a person who has been tracked based on the image of the in-store imaging device 14 comes to the front of the cash register, in S703 the face matching unit 406 uses the store visitor matching list linked to the store visitor. A second face collation is performed using the accuracy feature amount to identify which member the visitor belongs to.

Next, in S704, the face matching unit 406 determines whether the similarity score obtained by matching using the high-precision feature amount is equal to or greater than a predetermined score threshold. When the face matching unit 406 determines that the score is equal to or higher than the score threshold, the person corresponding to the member information acquired from the list management unit 405 in S703 is specified as a person who can make a payment. On the other hand, if it is determined to be less than the score threshold, the face matching unit 406 returns the process to S703 and performs comparison using the highly accurate face information in the member information with the next highest similarity score. If the similarity score does not reach or exceed the score threshold even after comparison with the high-precision face information of all member information in the visitor verification list, the face verification unit 406 determines that the person in front of the checkout is an unregistered person. , and instructs the settlement unit 407 to perform settlement such as cash payment (not shown).

In S705, the settlement unit 407 performs settlement processing using the settlement information 802 of the member (settler) when the person who is about to make the settlement is identified as a member in the preceding processing.
As another example, the management device 10 may execute part or all of the processing executed by the recognition device 12 in the flowchart of the second face matching processing.

As described above, in the first embodiment, the first face matching is performed using the medium-precision feature amount (face information) extracted from the face image detected from the image (overhead image) acquired by the in-store imaging device 14. By doing so, candidates for visiting members are narrowed down from among all the members. Then, for the narrowed down candidates, the high-precision face information (feature amount) of the face image registered in the member information is compared with the high-precision feature amount extracted from the face image photographed from the front by the register imaging device 15. A second face match is performed. As a result, even if the image recognition accuracy of the in-store image pickup device 14 is low, it is possible to specify the payer by performing highly accurate face recognition while shortening the recognition time at the cash register. In addition, since face recognition in the in-store imaging device 14 is a process using a medium-precision feature amount, it is possible to reduce the processing cost.

<Second embodiment>
Next, a second embodiment will be described. Here, only the functions and processes different from those of the first embodiment will be described, and the rest are assumed to be the same as those of the first embodiment unless otherwise specified. In the case of the second embodiment, under conditions where high accuracy of face recognition is required for an image acquired by the in-store imaging device 14, recognition processing using high-precision feature amounts (high-precision face information) is performed. do. In this embodiment, a predetermined analysis process is performed on the image of the in-store imaging device 14, and based on the result of the analysis process, it is determined whether or not conditions requiring high face recognition accuracy are satisfied, and high face recognition is satisfied. When it is determined that accuracy is required, recognition processing is performed using high-accuracy feature amounts. Conditions that require high face recognition accuracy for images acquired by the in-store imaging device 14 include, for example, when a part of the face is hidden by an obstacle such as hair, a mask, or sunglasses, or when a person is hidden behind an object. Such cases can be mentioned. In the second embodiment, as a predetermined analysis process, it is analyzed whether a part of a person's face is hidden. extract high-precision features from the image. Details of the predetermined analysis processing in this embodiment will be described later.

Further, in the above-described first embodiment, the visitor verification list is created only when a visitor is newly discovered, and the visitor who is being tracked is not updated. On the other hand, in the second embodiment, the visitor verification list is created only when a visitor is newly discovered, and is updated every time the visitor's face is detected by the in-store camera.

In the case of the second embodiment, when the recognition device 12 extracts the feature amount from the face image of the person (visitor 203) appearing in the image of the in-store imaging device 14, part of the face is hidden by a shield or the like. Perform analysis processing to determine whether or not Then, when the recognition device 12 obtains an analysis result that a part of the face is hidden by a shield or the like, the face image obtained by the in-store imaging device 14 is extracted from the face image by the first extraction process, that is, the high Acquire precision features (high-precision face information). Then, the extracted high-precision feature amount is transmitted to the management device 10 via the network 11 .

In the case of the second embodiment, the management device 10 uses the high-precision feature amount extracted from the image of the in-store imaging device 14 and the high-precision face information (high-precision feature amount) of the registered member to perform the first Perform face matching. The management device 10 then sends the result of the first face matching and the member information according to the second embodiment to the recognition device 12 . In the case of the second embodiment as well, the information indicating the result of the first face collation is a similarity score indicating the degree of similarity of facial feature amounts.

The recognition device 12 creates, holds, and manages a visitor verification list including the result of the first face verification by the management device 10 and member information. The configuration of the store visitor matching list is the same as in the first embodiment, and includes the similarity score obtained by the first face matching, and the identification information, payment information, and face information of the member according to the result of the first face matching. A list containing at least face information used for recognition is created.

FIG. 13 is a flow chart showing the flow of the first face matching process using the image of the in-store imaging device 14, which is executed in cooperation with the recognition device 12 and the management device 10 according to the second embodiment. The difference from FIG. 11 described above is that in the recognition device 12, after the processing of S601, the determination processing of S801 is performed, and depending on the result, the processing of S802 or S803 is performed, then the processing proceeds to S603, and furthermore, The point is that the process of S805 is executed after the process of S607. 11 in that the management apparatus 10 executes the process of S804 after the process of S604, and the process of S606 is executed after the process of S804.

When proceeding to S801, the feature amount extraction unit 402 of the recognition device 12 performs determination processing as predetermined analysis processing. At this time, the feature amount extraction unit 402 performs processing for acquiring the degree of masking of the face in the face image, for example, determines whether the likelihood of the facial features acquired during face detection in S601 is equal to or greater than a predetermined likelihood threshold. do. Here, as shown in FIG. 9A, since the face is not hidden and the likelihood of the facial features is equal to or greater than a predetermined likelihood threshold, there is a possibility that the facial features are plausible and that each facial feature is exposed. can be judged to be high. Therefore, if the likelihood of facial features is greater than or equal to the predetermined likelihood threshold, the feature quantity extraction unit 402 advances the process to S802. On the other hand, as shown in FIG. 8(b), when a part of the face is hidden by some kind of shield 1502, the likelihood of facial features may be less than the predetermined likelihood threshold. However, it is highly probable that part of each facial organ is hidden. Therefore, the feature amount extraction unit 402 advances the process to S803 when the likelihood of the facial feature is less than the predetermined likelihood threshold.

When proceeding to S802, the medium-precision processing unit 409 of the feature amount extraction unit 402 of the recognition apparatus 12 calculates a medium-precision feature amount (medium-precision face information) from the image of the boundary frame of the face.
On the other hand, if the process proceeds to S803, the high-precision processing unit 408 of the feature amount extraction unit 402 of the recognition device 12 calculates a high-precision feature amount (high-precision face information) from the image of the boundary frame of the face.

In the management device 10, when proceeding from S604 to S804, the face matching unit 302 combines the feature amount (face information) sent from the recognition device 12 with the member's face information (feature amount) acquired from the member information registration unit 300. to get a similarity score. That is, when the high-precision feature amount is received from the recognition device 12 , the face matching unit 302 acquires a high-precision similarity score by comparing it with the member's high-precision face information acquired from the member information registration unit 300 . Also, when medium-precision face information is received from the recognition device 12 , the face matching unit 302 acquires a medium-precision similarity score by comparing with the member's medium-precision face information acquired from the member information registration unit 300 .

Also, in the recognition device 12, when proceeding from S607 to S805, the list management unit 405 saves the member's similarity score and member information for each visitor in the visitor collation list. In the case of the second embodiment, the similarity scores saved in the store visitor collation list include the medium precision similarity score or the high precision similarity score acquired by the management device 10 in S804. That is, the list management unit 405 uses the medium-precision similarity score obtained by the management apparatus 10 according to the medium-precision feature amount in S802, or the high-precision similarity score obtained by the management apparatus 10 in accordance with the high-precision feature amount calculated in S803. Store the accuracy similarity score in the visitor matching list.

Note that if the visitor is a visitor whose face has already been detected and a similarity score has been obtained in the past, the list management unit 405 updates the value of the similarity score. In addition, if only one of the high-precision similarity score and the medium-precision similarity score is acquired for the visitor, the list management unit 405 uses the similarity score acquired for the visitor as the evaluation score. do.

On the other hand, if both the high-precision similarity score and the medium-precision similarity score have already been acquired for the visitor, the list management unit 405 considers the weight of each similarity score, calculates and stores the evaluation score. For example, the high-precision similarity score 806 and the medium-precision similarity score are weighted 0.5:0.5, and the sum thereof is used as the evaluation score.

FIG. 14 is a diagram showing an example of a visitor matching list including a high-precision similarity score 1401, a medium-precision similarity score 1402, and an evaluation score 1403. FIG. In FIG. 14, members from member ID 810 to medium-precision face information 803 are the same as in FIG. 8, and description thereof will be omitted. That is, in the case of the second embodiment, in the visitor matching list, the similarity score obtained by the first face matching using the high-precision feature quantity is stored in the high-precision similarity score 1401, and The similarity score from the first face match is saved in medium precision similarity score 1402 . Furthermore, in the store visitor collation list according to the second embodiment, the evaluation score calculated in consideration of the weight is stored in the evaluation score 1403 .

In the second embodiment, when settlement processing is subsequently performed according to the flowchart shown in FIG. 2 face matching is performed. In the second embodiment, processing is performed in consideration of when face detection and feature quantity extraction are performed several times for one visitor.

As described above, in the second embodiment, when the customer's face is not covered and can be confirmed, face recognition is performed using medium-precision feature amounts, thereby reducing the processing cost and reducing the customer's face. is partially hidden, face recognition is performed using high-precision feature amounts. Thus, according to the second embodiment, it is possible to maintain high recognition accuracy while suppressing the processing cost of the system as a whole.

In the second embodiment, as the predetermined analysis processing, the face exposed state, in other words, the hidden state of the face is confirmed using a value such as the likelihood of facial features, but the present invention is not limited to this. . As the predetermined analysis process, for example, a process of detecting shielding objects such as sunglasses or a mask from the image may be performed. The recognition device 12 performs face recognition using the high-precision feature amount when an obstacle is detected.

Further, in face recognition, recognition accuracy is generally higher when the face faces the imaging device. Therefore, as predetermined analysis processing, the face detection unit 401 performs face detection and face orientation detection processing. You can switch the precision of That is, the recognition device 12 performs face recognition using the medium-precision feature amount when the face is facing forward, and performs face recognition using the high-precision feature amount when the face is not facing the face.

In addition, in the present embodiment, the evaluation score based on the high-precision similarity score and the medium-precision similarity score is used, but the present invention is not limited to this. good.

<Third Embodiment>
Next, a third embodiment will be described. Here, only the functions and processes that are different from those of the second embodiment will be described, and the others are the same as those of the first and second embodiments unless otherwise specified.
In the recognition device 12 of the first or second embodiment, the matching process using medium-precision feature amounts is basically performed in the first face matching. The recognition device 12 of the third embodiment basically performs matching processing using high-precision feature amounts for all face recognition, and high-precision recognition results cannot be expected even if high-precision feature amounts are used. In some cases, or when the processing cost increases excessively, matching is performed using medium-precision feature amounts. As a result, in the third embodiment, the processing cost is suppressed while maintaining recognition accuracy.

In the third embodiment, as an example of a case in which a highly accurate recognition result cannot be expected even if a highly accurate feature amount is used, a case where the size of the boundary frame of a face detected from an image is small will be described. That is, in the third embodiment, as predetermined analysis processing, the size of the boundary frame of the face detected from the image is obtained, and depending on the size of the size, matching is performed using medium-precision feature amounts or high-precision feature amounts. Select (determine) whether to perform matching using . For example, when the size of the boundary frame of the face is small, the amount of information obtained from the face image is small, so that it is often impossible to improve the recognition accuracy even if a high-precision feature amount is used. For this reason, in the present embodiment, when the size of the boundary frame of the face is less than a predetermined threshold value, the processing cost is suppressed by using the medium-precision feature amount. On the other hand, when the size of the boundary frame of the face is equal to or larger than the predetermined threshold, matching with high recognition accuracy is possible by using the high-precision feature amount.

In the third embodiment, the processing of the first face matching using the image of the in-store imaging device 14, which is executed by the recognition device 12 in cooperation with the management device 10, is generally similar to the flow chart of FIG. 13 described above. become. However, in the case of the third embodiment, the judgment processing in S801 of the flowchart of FIG. 13 is different.

In the case of the third embodiment, in S801, the feature amount extraction unit 402 of the recognition device 12 determines whether or not the size of the boundary frame detected by face detection in S601 is less than a predetermined size threshold. More specifically, the feature quantity extraction unit 402 determines whether the width of the bounding box is less than a predetermined width threshold or the height of the bounding box is less than the height threshold. If either or both of the width and height of the boundary frame are less than the respective threshold values (width threshold value, height threshold value), the feature amount extraction unit 402 calculates the high-precision feature amount (face information). It is judged that the process using the medium-precision feature quantity, which does not have a sufficient amount of information and has a low processing cost, is sufficient. In other words, if either or both of the width and height of the boundary frame are less than the corresponding threshold value, the feature quantity extraction unit 402 executes medium precision feature quantity calculation processing in S803. On the other hand, when both the width and height of the boundary frame are equal to or greater than the corresponding threshold values (width threshold value or height threshold value), the feature amount extraction unit 402 does not have sufficient information amount to generate a highly accurate feature amount. It is determined that there is, and high-precision feature amount calculation processing in S802 is executed.

As described above, according to the third embodiment, when it is determined that sufficient accuracy cannot be obtained even if a process for extracting high-precision feature amounts (high-precision face information) is performed, medium-precision feature amounts ( By extracting medium-precision face information, it is possible to maintain accuracy and reduce processing costs.

<Modified example of the third embodiment>.
In the third embodiment, the size of the boundary frame of the detected face is used, but as another example of the predetermined analysis processing, processing for detecting the number of faces detected from the captured image is performed, and the analysis is performed. Based on the results, it is possible to select (switch) whether to perform high-precision or medium-precision feature amount calculation processing. That is, when the number of faces detected from an image is large, it is assumed that the processing cost will increase significantly if processing is performed using the large number of face images. Therefore, in the recognition device 12, when performing the first face matching, the number of faces is detected from the image as a predetermined analysis process. Calculation processing is performed. On the other hand, when the number of faces is less than the predetermined number threshold, the recognition device 12 performs high-precision feature amount calculation processing. According to this modified example, when it is determined that there are many faces and the processing cost will increase if the high-precision feature amount calculation is performed, the processing cost can be reduced by performing the medium-precision feature amount calculation. becomes.

<Fourth Embodiment>
In the above-described first to third embodiments, system configuration examples in which the management device and the recognition device are separate have been given, but both the functions of the management device and the recognition device may be realized by one information processing device. The fourth embodiment describes an information processing apparatus having both functions of a management device and a recognition device.

FIG. 15 is a diagram showing a functional configuration example of an information processing apparatus according to the fourth embodiment. Note that the hardware configuration of the information processing apparatus according to the fourth embodiment is substantially the same as the configurations of FIGS. Further, the information processing apparatus of the fourth embodiment can perform the processing described in the management apparatus and the recognition apparatus in any one of the first to third embodiments, but the first embodiment will be described here as an example. A case where the above example is applied to the fourth embodiment will be described.

In the fourth embodiment, the image acquisition unit 141 of the information processing device acquires an image from the in-store imaging device 14 or the cash register imaging device 15, and the face detection unit 142 detects a face from the image.
In the feature amount extraction unit 146, when a face image detected from the image of the in-store imaging device 14 is sent, the medium-precision processing unit 146M extracts the medium-precision feature amount (face information) and performs the first face matching. Send to section 143 . At this time, the member information acquisition unit 140 sends the face information registered in the member information prepared in advance to the first face matching unit 143 . Then, the first face collation unit 143 performs the same first face collation as described above, and the list management unit 144 creates and holds a visitor list based on the result of the first face collation.

Further, in the feature amount extraction unit 146, when a face image detected from the image of the register imaging device 15 is sent, the high-precision processing unit 146H extracts the high-precision feature amount (face information) to obtain a second face image. Send to collation unit 147 . At this time, the second face collation unit 147 acquires high-precision feature amounts in descending order of similarity from the store visitor collation list created and held earlier, and performs the second face collation. Then, when the second face matching unit 147 can identify the person in front of the register as a member, the settlement unit 148 performs settlement processing.

FIG. 16 is a flow chart showing the flow of processing in the information processing apparatus according to the fourth embodiment.
In S<b>160 , the image acquisition unit 141 acquires an image from the in-store imaging device 14 or the cash register imaging device 15 . Then, in S161, if the acquired image is the image from the in-store imaging device 14, the information processing device proceeds with the processing from S162 onwards, and if it is the image from the cash register imaging device 15, the processing proceeds from S166 onwards. proceed.

When proceeding to the process of S<b>162 , the face detection unit 142 performs face detection from the image captured by the in-store imaging device 14 .
Next, in S162, the feature amount extraction unit 146 extracts medium-precision feature amounts (face information) from the face image (boundary frame image) detected by the face detection unit 142 by the medium-precision processing unit 146M.
Furthermore, in S164, the first face matching unit 143 acquires the member information from the member information acquisition unit 140, and uses the medium-precision face information (feature amount) in the member information and the medium-precision feature amount extracted in S163. to obtain a similarity score.
Then, in S165, the list management unit 144 creates and holds a visitor verification list based on the member information obtained from the member information obtaining unit 140 and the similarity score obtained in S164. After that, the information processing device ends the processing of the flowchart of FIG. 16 .

On the other hand, when proceeding to S<b>166 , the face detection unit 142 performs face detection from the image of the register imaging device 15 .
Next, in S167, the feature amount extraction unit 146 extracts a high-precision feature amount (face information) from the face image (boundary frame image) detected by the face detection unit 142 by the high-precision processing unit 146H.
Furthermore, in S168, the second face matching unit 143 uses the high-precision face information (feature amount) of the visitor matching list held by the list management unit 144 and the high-precision feature amount extracted in S167. A second face matching is performed.
Then, when the similarity score equals or exceeds the threshold in the second face collation, the information processing apparatus advances the processing to S170, and the settlement unit 148 performs settlement processing in S170.
On the other hand, if a similarity score less than the threshold value is obtained in S169, the information processing device ends the processing of the flowchart of FIG. In other words, the payer in this case will make the payment in cash or the like.

FIG. 17 shows the processing when the information processing device 1700 in the store performs the first face verification, creation and management of the store visitor verification list, and the second face verification to payment request in the fourth embodiment. is a sequence diagram showing the flow of In this example, it is assumed that a management device 1000 is provided as a device that manages member information of all registered members and performs settlement processing.

In S900, the information processing apparatus 1700 requests the management apparatus 1000 for member information including feature amounts (face information) of all members. Then, in S<b>901 , the information processing device 1700 acquires member information including the feature amount from the management device 1000 .

The information processing apparatus 1700 sends an image acquisition request to the in-store imaging device 14 in S902, and when an image is sent from the in-store imaging device 14, acquires the image from the in-store imaging device 14 in S903.
Next, in S904, the information processing device 1700 detects a face image of a person from the image acquired from the in-store imaging device 14, and extracts a medium precision feature amount (face information) from the face image. Further, in S905, the information processing apparatus 1700 performs first face matching using the medium-precision feature amount extracted in S904 and the medium-precision face information (feature amount) of the member information list, and creates a visitor verification list. create.

Thereafter, the information processing apparatus 1700 sends an image acquisition request to the cash register imaging device 15 in S906, and when an image is sent from the cash register imaging device 15, the image from the cash register imaging device 15 is acquired in S907. do.
Next, in S908, the information processing device 1700 detects a face image of a person from the image captured by the register imaging device 15, and extracts high-precision feature amounts (face information) from the face image. Further, in S909, the information processing apparatus 1700 performs second face collation for comparing the high-accuracy feature amount extracted in S908 with the high-accuracy face information (feature amount) in the visitor collation list.

When the information processing device 1700 identifies the member in S910 as a result of the second face matching, in S911, the information processing device 1700 sends payment information to the management device 1000 to request payment processing. Then, when the settlement process is completed in the management device 1000, a notification of the settlement process result is received in S912.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.
All of the above-described embodiments merely show specific examples for carrying out the present invention, and the technical scope of the present invention should not be construed to be limited by these. That is, the present invention can be embodied in various forms without departing from its technical concept or main features.

10: Management device, 12: Recognition device, 14: In-store imaging device, 15: Cash register imaging device, 300: Member information registration unit, 301: Face information acquisition unit, 302: Face matching unit, 400: Image acquisition unit, 401: Face detection unit 402: Feature amount extraction unit 404: Member information acquisition unit 405: List management unit 406: Face matching unit 407: Payment unit

Claims (27)

An information processing system including a first information processing device and a second information processing device,
The first information processing device is
First feature information extracted by a first extraction process, and second feature information extracted by a second extraction process with a larger amount of processing than the first extraction process, from face images of a plurality of persons. and obtain person information, including
3rd feature information extracted by said 1st extraction process from the 1st face image which image|photographed a person, and said 1st feature information contained in said person information are compared with said 1st feature information, and the similarity is acquired. 1 matching process is performed, and the personal information including the degree of similarity obtained by the first matching process is transmitted to the second information processing device;
The second information processing device is
acquiring personal information including the degree of similarity from the first information processing device, creating a list including the first characteristic information and the second characteristic information according to the degree of similarity from the personal information;
The fourth feature information extracted by the second extraction process from the second facial image of a person is compared with the second feature information in the list in descending order of the degree of similarity, and the performing a second matching process for identifying the person in the face image of 2 from the person information;
An information processing system characterized by: The first extraction process and the second extraction process are processes using neural networks having different circuit scales,
2. The information processing system according to claim 1, wherein the second extraction process is a process using a neural network having a circuit scale larger than that of the first extraction process. The second information processing device transmits to the first information processing device feature information extracted by the first extraction process from the first face image of a person, and
3. The information processing system according to claim 1, wherein said first information processing device acquires said feature information received from said second information processing device as said third feature information. The first collation processing in the first information processing apparatus includes matching feature information extracted from the first face image by the second extraction processing and the second feature information included in the person information. It also includes processing to obtain similarity by comparing
The second information processing apparatus performs the first extraction process and the second extraction process on the first face image based on the result of a predetermined analysis process using a photographed face image of a person. selecting one of the processes, transmitting the feature information extracted by the selected extraction process to the first information processing device;
In the first matching process, the first information processing apparatus performs the feature information received from the second information processing apparatus and the extraction process selected by the second information processing apparatus. 3. The information processing system according to claim 1, wherein the similarity is obtained by comparing the feature information of the person information. The second information processing device weights the degree of similarity depending on which of the first extraction process and the second extraction process is selected based on the result of the predetermined analysis process. 5. The information processing system according to claim 4, wherein: The predetermined analysis process is a process of obtaining the number of face images appearing in the photographed image,
6. The information processing system according to claim 4, wherein the second information processing device selects the first extraction process when the number is equal to or greater than a predetermined number threshold. The predetermined analysis process is a process of acquiring the degree of masking of the face in the photographed face image,
The second information processing apparatus acquires the likelihood of facial organs in the facial image as the degree of masking of the face, and executes the first extraction process when the likelihood is equal to or greater than a likelihood threshold. 6. The information processing system according to claim 4, wherein the selection is made. The predetermined analysis process is a process of detecting an image of a predetermined shield from the photographed face image,
6. The information processing system according to claim 4, wherein the second information processing device selects the second extraction process when the image of the predetermined shielding object is detected. The predetermined analysis process is a process of detecting the orientation of the face in the photographed face image,
6. The information processing system according to claim 4, wherein the second information processing device selects the second extraction process when the face orientation is not the facing orientation. The predetermined analysis process is a process of acquiring the size of the face in the photographed face image,
6. The information processing system according to claim 4, wherein the second information processing device selects the first extraction process when the size of the face is less than a predetermined size threshold. The first facial image is an image detected from an image captured by a first imaging device that captures an image of the interior of the store, and the second facial image is a second imaging that captures an image of a person making a payment at a cash register. 11. The information processing system according to any one of claims 1 to 10, wherein the image is an image captured by a device. 12. The information processing according to any one of claims 1 to 11, wherein the second information processing apparatus does not perform the second matching process for face images with the degree of similarity less than a threshold. system. 13. The information processing system according to any one of claims 1 to 12, wherein the second information processing device performs payment processing based on payment information associated with the specified person. First feature information extracted by a first extraction process, and second feature information extracted by a second extraction process with a larger amount of processing than the first extraction process, from face images of a plurality of persons. and an information acquisition means for acquiring personal information including
A process of comparing third feature information extracted by the first extraction process from a first facial image of a person photographed with the first feature information included in the person information to acquire a degree of similarity. means and
management means for creating and managing a list including the first feature information and the second feature information extracted from the person information according to the similarity and the similarity;
The fourth feature information extracted by the second extraction process from the second face image of a person is compared with the second feature information in the list in descending order of similarity to determine the second face image. identification means for identifying the person in the face image of from the person information;
An information processing device comprising: The first extraction process and the second extraction process are processes using neural networks having different circuit scales,
15. The information processing apparatus according to claim 14, wherein the second extraction process is a process using a neural network having a circuit scale larger than that of the first extraction process. The processing means
selecting either the first extraction process or the second extraction process as the extraction process for the first face image based on the result of a predetermined analysis process using the photographed face image; ,
The feature information extracted from the first face image by the selected extraction process and the feature information of the person information according to the selected extraction process are compared to obtain a degree of similarity. 16. The information processing device according to Item 14 or 15. The processing means weights the degree of similarity depending on whether the first extraction process or the second extraction process is selected based on the result of the predetermined analysis process. 17. The information processing apparatus according to claim 16. The predetermined analysis process is a process of obtaining the number of face images appearing in the photographed image,
18. An information processing apparatus according to claim 16, wherein said processing means selects said first extraction process when said number is greater than or equal to a predetermined number threshold. The predetermined analysis process is a process of acquiring the degree of masking of the face in the photographed face image,
The processing means acquires the likelihood of facial organs in the facial image as the degree of masking of the face, and selects the first extraction process when the likelihood is equal to or greater than a likelihood threshold. 18. The information processing apparatus according to claim 16 or 17. The predetermined analysis process is a process of detecting an image of a predetermined shield from the photographed face image,
18. The information processing apparatus according to claim 16, wherein the processing means selects the second extraction process when the image of the predetermined shielding object is detected. The predetermined analysis process is a process of detecting the orientation of the face in the photographed face image,
18. The information processing apparatus according to claim 16, wherein said processing means selects said second extraction process when said face is not facing forward. The predetermined analysis process is a process of acquiring the size of the face in the photographed face image,
18. The information processing apparatus according to claim 16, wherein said processing means selects said first extraction process when the size of said face is less than a predetermined size threshold. The first facial image is an image detected from an image captured by a first imaging device that captures an image of the interior of the store, and the second facial image is a second imaging that captures an image of a person making a payment at a cash register. 23. The information processing apparatus according to any one of claims 14 to 22, wherein the image is an image captured by the apparatus. 24. The information processing apparatus according to any one of claims 14 to 23, wherein said specifying means does not perform said specifying process for face images whose similarity is less than a threshold. 25. The information processing apparatus according to any one of claims 14 to 24, further comprising payment means for performing payment processing based on payment information associated with the specified person. First feature information extracted by a first extraction process, and second feature information extracted by a second extraction process with a larger amount of processing than the first extraction process, from face images of a plurality of persons. and an information acquisition step of acquiring personal information including
A process of comparing third feature information extracted by the first extraction process from a first facial image of a person photographed with the first feature information included in the person information to acquire a degree of similarity. process and
a management step of creating and managing a list including the first feature information and the second feature information extracted from the person information according to the similarity and the similarity;
The fourth feature information extracted by the second extraction process from the second face image of a person is compared with the second feature information in the list in descending order of similarity to determine the second face image. an identifying step of identifying the person in the face image of from the person information;
An information processing method characterized by having A program for causing a computer to function as the information processing apparatus according to any one of claims 1 to 25.

Images