WO2022176342A1

WO2022176342A1 - Information processing device, information processing system, information processing method, and non-transitory computer-readable medium

Info

Publication number: WO2022176342A1
Application number: PCT/JP2021/045571
Authority: WO
Inventors: 秀典北方
Original assignee: 日本電気株式会社
Priority date: 2021-02-17
Filing date: 2021-12-10
Publication date: 2022-08-25
Also published as: JPWO2022176342A1; JP7468771B2

Abstract

An information processing device (10) comprises a registration unit (11), an authentication control unit (12), a recording control unit (14), an output unit (16), and an evaluation unit (17). After acquiring a registration request, the registration unit (11) registers attribute information on a person in association with biometric information . After acquiring a photographed image, which is obtained by photographing the person at a predetermined location associated with predetermined content, the authentication control unit (12) causes the person to perform biometric authentication by using the photographed image. The recording control unit (14) records, when the biometric authentication is successful, the degree of smile of the person, which is calculated from the photographed image, in association with content information relating to content associated with attribute information for the person and the photography location. The output unit (16) outputs information according to the degree of smile of the person to an information terminal used by the person who has passed the biometric authentication. The evaluation unit (17) evaluates the degree of satisfaction with the content on the basis of the attribute information on the person and the content information.

Description

Information processing device, information processing system, information processing method, and non-transitory computer-readable medium

The present disclosure relates to an information processing device, an information processing system, an information processing method, and a non-temporary computer-readable medium. computer readable medium.

Various applications have been proposed to increase the satisfaction of visitors to the event content (hereinafter referred to as content) provided by the event venue. For example, Patent Document 1 discloses an image selection device that evaluates the degree of smile of a visitor who is experiencing an attraction installed in an amusement park or the like, and selects the still image with the highest degree of smile as an image for sales promotion. disclosed. Further, Patent Document 2 discloses an information processing system that calculates a happiness level from a face image when entering and exiting a space, and calculates an evaluation value for the space based on the difference in the happiness level.

JP 2019-071553 A WO2017/022306

Here, in order to improve the level of content satisfaction, it is effective to first analyze the current level of content satisfaction in detail. However, Patent Literature 1 mentioned above does not describe analyzing the satisfaction level of attractions. Further, as a detailed analysis of the degree of satisfaction with content, analysis of the degree of satisfaction with content for each visitor characteristic can be mentioned. However, Patent Literature 2 mentioned above does not describe analyzing the evaluation value of the space for each visitor's characteristic. In addition, in order to perform such a detailed analysis, it is necessary to collect a large amount of information related to visitors. has not been disclosed.

An object of the present disclosure is to provide an information processing device, an information processing system, an information processing method, and a non-temporary computer-readable medium that suitably analyze the degree of satisfaction with content at an event venue in view of the above-described problems.

An information processing device according to an aspect of the present disclosure includes:
registration means for registering attribute information of a subject in association with biometric information of the subject when a request for use registration is acquired;
Authentication control means for performing biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with a predetermined content is acquired;
recording control means for recording, when the biometric authentication is successful, the degree of smile of the subject calculated from the photographed image in association with content information related to content associated with the subject's attribute information and a photographing location;
Output means for outputting information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication is successful;
evaluation means for summarizing smile levels based on subject's attribute information and content information, and evaluating content satisfaction based on the summarized smile levels.

An information processing system according to one aspect of the present disclosure includes:
an information terminal that generates a captured image of a target person;
and an information processing device communicably connected to the information terminal.
The information processing device is
registration means for registering attribute information of a subject in association with biometric information of the subject when use registration is accepted;
Authentication control means for performing biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with a predetermined content is obtained from the information terminal;
recording control means for recording, when the biometric authentication is successful, the degree of smile of the subject calculated from the photographed image in association with content information related to content associated with the subject's attribute information and a photographing location;
Output means for outputting information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication is successful;
evaluation means for summarizing the degree of smile based on the attribute information of the subject and the content information, and evaluating satisfaction with the content based on the summarized degree of smile.

An information processing method according to an aspect of the present disclosure includes:
When the user registration request is obtained, registering the attribute information of the subject in association with the biometric information of the subject,
When a photographed image of a target person photographed at a predetermined point associated with predetermined content is acquired, biometric authentication is performed using the photographed image,
when the biometric authentication is successful, recording the degree of smile of the subject calculated from the captured image in association with content information related to content associated with the subject's attribute information and the shooting location;
Outputting information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication is successful,
Smile levels are aggregated based on the subject's attribute information and content information, and content satisfaction is evaluated based on the aggregated smile levels.

A non-transitory computer-readable medium according to one aspect of the present disclosure comprises:
a registration process of registering attribute information of a subject in association with biometric information of the subject when a request for use registration is acquired;
Authentication control processing for performing biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with predetermined content is acquired;
recording control processing for recording, when the biometric authentication is successful, the degree of smile of the subject calculated from the captured image in association with content information related to content associated with the subject's attribute information and a shooting location;
An output process for outputting information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication has succeeded;
and an evaluation process of aggregating smile levels based on subject's attribute information and content information and evaluating content satisfaction based on the aggregated smile levels.

According to the present disclosure, it is possible to provide an information processing device, an information processing system, an information processing method, and a non-temporary computer-readable medium that suitably analyze the degree of satisfaction with content at an event venue.

1 is a block diagram showing the configuration of an information processing apparatus according to a first embodiment; FIG. 3 is a diagram showing the flow of an information processing method according to the first embodiment; FIG. 2 is a block diagram showing the overall configuration of an information processing system according to a second embodiment; FIG. 2 is a block diagram showing the configuration of an authentication device according to a second embodiment; FIG. 9 is a flowchart showing the flow of face information registration processing according to the second embodiment; 9 is a flow chart showing the flow of face authentication processing according to the second embodiment; FIG. 10 is a block diagram showing the configuration of a face authentication terminal according to the second embodiment; FIG. FIG. 8 is a block diagram showing the configuration of a user terminal according to the second embodiment; FIG. 2 is a block diagram showing the configuration of an information processing apparatus according to a second embodiment; FIG. FIG. 11 is a diagram illustrating an example of a data structure of aggregated information according to the second embodiment; FIG. 9 is a flowchart showing the flow of usage registration processing according to the second embodiment; 9 is a flowchart showing the flow of image output processing according to the second embodiment; 9 is a flowchart showing the flow of evaluation processing according to the second embodiment; FIG. 11 is a sequence diagram showing the flow of usage registration processing according to the second embodiment; FIG. 10 is a sequence diagram showing the flow of image output processing according to the second embodiment; FIG. 10 is a sequence diagram showing the flow of image output processing according to the second embodiment; FIG. 11 is a diagram showing an example of a synthesized image according to the second embodiment; FIG. FIG. 11 is a diagram showing an example of a terminal screen according to the second embodiment; FIG. FIG. 11 is a diagram showing an example of a terminal screen according to the second embodiment; FIG. FIG. 11 is a diagram showing an example of an overall composite image according to the second embodiment; FIG.

Below, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same reference numerals are given to the same or corresponding elements, and redundant description will be omitted as necessary for clarity of description.

<Embodiment 1>
First, Embodiment 1 of the present disclosure will be described. FIG. 1 is a block diagram showing the configuration of an information processing apparatus 10 according to the first embodiment. The information processing device 10 is an information processing device that analyzes the level of satisfaction with content provided at an event site. Here, the information processing device 10 is connected to a network (not shown). A network may be wired or wireless. The network is also connected to an information terminal (not shown) that takes a picture of a target person who is a visitor at the event venue and generates a taken image. In other words, the information processing device 10 is communicably connected to the information terminal via the network. Here, the information terminal is installed at a predetermined point associated with the predetermined content. The captured image includes at least the subject's face area. The information terminal may be a face authentication terminal, digital signage with a camera, or the like.

The information processing device 10 includes a registration unit 11 , an authentication control unit 12 , a recording control unit 14 , an output unit 16 and an evaluation unit 17 .

The registration unit 11 is also called registration means. When the registration unit 11 acquires a use registration request for a target person, the registration unit 11 registers the target person's attribute information in association with the target person's biometric information. Here, the subject's attribute information is information related to the subject. For example, the subject's attribute information may include at least one of the subject's age, gender, occupation, family structure, hobby, companion information, and means of visit. The companion information indicates the presence or absence of a companion, and if there is a companion, the attributes of the group including the target person and the companion (group attribute). As an example, the group attribute is family (with parent and child), couple, friends, or the like. The companion information may include the identification information (ID) of the companion. A means of visiting is an automobile, a train, or the like. The biometric information may be face, fingerprint, iris or vein feature information, or other biometric information.

The authentication control unit 12 is also called authentication control means. When the authentication control unit 12 acquires a photographed image of the target person photographed at the above-described predetermined point from the above-described information terminal, the biometric authentication is performed using the photographed image.

The recording control unit 14 is also called recording control means. When the biometric authentication is successful, the recording control unit 14 records the subject's smile degree in association with the subject's attribute information and the content information related to the content associated with the shooting location. Incidentally, the degree of smile is calculated from the photographed image.

The output unit 16 is also called output means. The output unit 16 outputs information corresponding to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication has succeeded. The information terminal used by the target person may be an information terminal that is installed at the above-mentioned predetermined spot and takes an image of the target person, or may be a target person terminal owned by the target person.

The evaluation unit 17 is also called evaluation means. The evaluation unit 17 summarizes the degree of smile based on the subject's attribute information and content information. Then, the evaluation unit 17 evaluates the degree of satisfaction with the content based on the aggregated degree of smile.

FIG. 2 is a diagram showing the flow of the information processing method according to the first embodiment. First, the registration unit 11 receives service usage registration from a subject (S10). Then, the registration unit 11 registers the target person's attribute information in association with the target person's biometric information (S11). Subsequently, the authentication control unit 12 acquires from the information terminal a photographed image of the target person photographed by the information terminal at a predetermined point (S12). Then, the authentication control unit 12 performs biometric authentication using the captured image (S13), and if the biometric authentication fails (No in S14), the process ends. On the other hand, if the biometric authentication is successful (Yes in S14), the recording control unit 14 associates the subject's smile degree calculated from the captured image with the subject's attribute information and the content information of the content associated with the shooting location. It is added and recorded (S15). Subsequently, the output unit 16 acquires information corresponding to the degree of smile and outputs it to the information terminal used by the subject (S16). Then, the evaluation unit 17 aggregates the degree of smile based on the associated content information and attribute information of the subject, and evaluates the degree of satisfaction with the content based on the aggregated degree of smile (S17). The evaluation unit 17 then terminates the process.

In this way, the information processing apparatus 10 according to the first embodiment aggregates the degree of smile based on the attribute information of the target person, and performs detailed analysis of the degree of content satisfaction. Therefore, it becomes easy for the operator of the event site to take measures to improve the satisfaction of the content. In addition, since information corresponding to the degree of smile can be obtained as a privilege for the target person, it is easy to work as an incentive to provide the user's attribute information and to permit usage. Thereby, the information processing apparatus 10 can easily acquire the attribute information of the subject, and detailed analysis becomes easy. Therefore, it is possible to more preferably analyze the degree of satisfaction with the content at the event site.

The information processing apparatus 10 includes a processor, a memory, and a storage device (not shown). Further, the storage device stores a computer program in which the processing of the information processing method according to the present embodiment is implemented. Then, the processor loads the computer program from the storage device into the memory and executes the computer program. Thereby, the processor implements the functions of the registration unit 11 , the authentication control unit 12 , the recording control unit 14 , the output unit 16 and the evaluation unit 17 .

Alternatively, the registration unit 11, the authentication control unit 12, the recording control unit 14, the output unit 16, and the evaluation unit 17 may each be realized by dedicated hardware. Also, part or all of each component of each device may be realized by general-purpose or dedicated circuitry, processors, etc., or combinations thereof. These may be composed of a single chip, or may be composed of multiple chips connected via a bus. A part or all of each component of each device may be implemented by a combination of the above-described circuits and the like and programs. Moreover, CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (field-programmable gate array), etc. can be used as a processor.

Further, when part or all of each component of the information processing device 10 is realized by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally arranged, They may be distributed. For example, the information processing device, circuits, and the like may be implemented as a form in which each is connected via a communication network, such as a client-server system, a cloud computing system, or the like. Also, the functions of the information processing device 10 may be provided in a SaaS (Software as a Service) format.

<Embodiment 2>
Next, Embodiment 2 of the present disclosure will be described. FIG. 3 is a block diagram showing the overall configuration of an information processing system 1000 according to the second embodiment. The information processing system 1000 provides a service to a target person (user U) who is a visitor at an event site, collects captured images of the user U, and measures satisfaction with each content provided at the event site. A computer system for analysis. The event venue is a zoo or an amusement park. For example, if the event venue is a zoo, the content may refer to viewing each animal or the animals themselves. For example, if the event venue is an amusement park, the content may refer to experiencing each attraction or the attractions themselves. Moreover, the above-mentioned service provides the user U with privilege information according to the user U's photographed image.

The information processing system 1000 includes an authentication device 100, an information processing device 200, an image storage server 300, face authentication terminals 400-1 to 400-n (n is a natural number of 2 or more), and a user terminal 500. Authentication device 100, information processing device 200, image storage server 300, face authentication terminals 400-1 to 400-n, and user terminal 500 are connected via network N, respectively. Here, the network N is a wired or wireless communication line.

Also, face authentication terminals 400-1, 400-2, 400-3, . . . 400-n are installed at points A1, A2, A3, . Here, points A1 to An are assumed to be different spots in a certain event site A. FIG. For example, point A1 may be the entrance to a zoo, and points A2-An may be animal viewing spots. Incidentally, at the points A2 to An, the face authentication terminal 400 may be configured so as to be able to photograph together the user U and scenery, animals, or the like peculiar to the related content. . . , 400-n may be simply referred to as the face authentication terminal 400 hereinafter when the face authentication terminals 400-1, 400-2, 400-3, . . .

User U registers for use of the service at point A1 and registers his/her own face information. In the usage registration, user U's attribute information is registered. The use registration and face information registration may be performed at any face authentication terminal 400 or user terminal 500 that the user U desires. The user U then visits one or more points. At each point A1 to An, the face authentication terminal 400 takes an image of the visiting user U. FIG. When the face authentication using the captured image is successful, the user U can acquire information according to the degree of smile in a predetermined manner. The degree of smile is generated based on a photographed image for which face recognition has succeeded. In the following description, it is assumed that the information corresponding to the degree of smile is an image corresponding to the degree of smile. The image corresponding to the degree of smile may be a composite image in which the degree of smile of the user U is superimposed on the photographed image. Also, the image corresponding to the degree of smile may correspond to a photographed image in which the user U appears satisfied or happy. For example, the image corresponding to the degree of smile may be a photographed image in which the degree of smile of the user U is equal to or greater than a predetermined value, or the degree of smile of the user U may be superimposed on the photographed image in which the degree of smile of the user U is equal to or greater than a predetermined value. It may be a composite image obtained by The information according to the degree of smile may be information related to content, such as sales promotion information, in addition to or instead of the above-described image. In this case, the content may be content related to the photographing location of the photographed image having a degree of smile equal to or greater than a predetermined value.

Here, the authentication device 100 is an information processing device that stores facial feature information of a plurality of persons. In addition, in response to a face authentication request received from the outside, the authentication device 100 compares the face image or face feature information included in the request with the face feature information of each user, and requests the matching result (authentication result). Reply to original.

FIG. 4 is a block diagram showing the configuration of the authentication device 100 according to the second embodiment. The authentication device 100 includes a face information DB (DataBase) 110 , a face detection section 120 , a feature point extraction section 130 , a registration section 140 and an authentication section 150 . The face information DB 110 associates and stores a user ID 111 and face feature information 112 of the user ID. The facial feature information 112 is a set of feature points extracted from the facial image. Note that the authentication device 100 may delete the facial feature information 112 in the facial feature DB 110 at the request of the registered user of the facial feature information 112 . Alternatively, the authentication device 100 may delete the facial feature information 112 after a certain period of time has passed since it was registered.

The face detection unit 120 detects a face area included in a registration image for registering face information and outputs it to the feature point extraction unit 130 . Feature point extraction section 130 extracts feature points from the face area detected by face detection section 120 and outputs facial feature information to registration section 140 . Further, feature point extraction section 130 extracts feature points included in the facial image received from information processing apparatus 200 and outputs facial feature information to authentication section 150 .

The registration unit 140 newly issues a user ID 111 when registering facial feature information. The registration unit 140 associates the issued user ID 111 with the facial feature information 112 extracted from the registered image and registers them in the facial information DB 110 . The authentication unit 150 performs face authentication using the facial feature information 112 . Specifically, the authentication unit 150 collates the facial feature information extracted from the facial image with the facial feature information 112 in the facial information DB 110 . The authentication unit 150 replies to the information processing apparatus 200 whether or not the facial feature information matches. Whether the facial feature information matches or not corresponds to the success or failure of the authentication. Note that matching of facial feature information (matching) means a case where the degree of matching is equal to or greater than a predetermined value.

FIG. 5 is a flowchart showing the flow of face information registration processing according to the second embodiment. First, the authentication device 100 acquires the registration image included in the face information registration request (S21). For example, the authentication device 100 receives a face information registration request from the face authentication terminal 400, the user terminal 500, or the like via the network N. FIG. Note that the face information registration request source is not limited to this, and may be the information processing apparatus 200 that receives the use registration request from the face authentication terminal 400 or the user terminal 500 . Next, face detection section 120 detects a face area included in the registered image (S22). Next, the feature point extraction unit 130 extracts feature points from the face area detected in step S22, and outputs face feature information to the registration unit 140 (S23). Finally, the registration unit 140 issues the user ID 111, associates the user ID 111 with the facial feature information 112, and registers them in the facial information DB 110 (S24). Note that the authentication device 100 may receive the facial characteristic information 112 from the face information registration requester, associate it with the user ID 111 , and register it in the facial information DB 110 .

FIG. 6 is a flow chart showing the flow of face authentication processing by the authentication device 100 according to the second embodiment. First, the feature point extraction unit 130 acquires facial feature information for authentication (S31). For example, the authentication device 100 receives a face authentication request from the information processing device 200 via the network N, and extracts facial feature information from the face image included in the face authentication request in steps S21 to S23. Alternatively, authentication device 100 may receive facial feature information from information processing device 200 . Next, the authentication unit 150 collates the acquired facial feature information with the facial feature information 112 of the facial information DB 110 (S32). If the facial feature information matches, that is, if the degree of matching of the facial feature information is equal to or greater than a predetermined value (Yes in S33), the authentication unit 150 specifies the user ID 111 of the user whose facial feature information matches (S34). , the fact that the face authentication was successful and the specified user ID 111 are returned to the information processing apparatus 200 (S35). If there is no matching facial feature information (No in S33), the authentication unit 150 returns to the information processing apparatus 200 that the face authentication has failed (S36).

It should be noted that in step S32, the authentication unit 150 does not need to attempt matching with all the facial feature information 112 in the facial information DB 110. For example, the authentication unit 150 may preferentially attempt matching with facial feature information registered during the period from the day the face authentication request is received until several days before. This can improve matching speed. Also, if the preferential collation fails, it is preferable to collate with all the remaining facial feature information.

Return to Figure 3 and continue the explanation. Each of the face authentication terminals 400-1, 400-2, . . . 400-n is an information terminal including a camera and a display device.

The face authentication terminal 400 transmits a usage registration request for the user U to the information processing device 200 via the network N. The face authentication terminal 400 also captures a registration image used for face authentication of the user U at the time of use registration. The face authentication terminal 400 transmits a face information registration request including the registered image to the authentication device 100 via the network N. FIG. The face authentication terminal 400 may transmit a face information registration request including the registered image to the authentication device 100 via the information processing device 200 . Note that the user U may use the user terminal 500 to perform usage registration and face information registration.

The face authentication terminal 400 captures a face image for authentication used for user U's face authentication. For example, the face authentication terminal 400 uses captured images of the user U at each installation location as images for authentication. The face authentication terminal 400 transmits a face authentication request including an image for authentication to the information processing device 200 via the network N. FIG. At this time, the face authentication terminal 400 may include in the face authentication request a location ID that identifies the location where the face authentication terminal 400 is installed. Moreover, the face authentication terminal 400 may include the shooting time in the face authentication request. In addition, the face authentication terminal 400 receives the face authentication result and smile level information from the information processing device 200 via the network N, and displays the information on the screen as necessary. Further, when the face authentication terminal 400 receives input of shooting end information indicating that the user U wishes to end shooting, the face authentication terminal 400 transmits a shooting end request including the shooting end information to the information processing apparatus 200 .

Next, the face authentication terminal 400 will be explained in detail. FIG. 7 is a block diagram showing the configuration of the face authentication terminal 400 according to the second embodiment. The face authentication terminal 400 includes a camera 410 , a storage section 420 , a communication section 430 , a display section 440 , a control section 450 and an input section 460 .

The camera 410 is a photographing device that takes pictures under the control of the control unit 450 . Storage unit 420 is a storage device that stores a program for realizing each function of face authentication terminal 400 . A communication unit 430 is a communication interface with the network N. FIG. Display unit 440 is a display device. The input unit 460 is an input device that receives an input from the user U. The display unit 440 and the input unit 460 may be configured integrally. As an example, the display unit 440 and the input unit 460 are touch panels. The control unit 450 controls hardware included in the face authentication terminal 400 . The control unit 450 includes an imaging control unit 451 , a registration unit 452 , an authentication control unit 453 and a display control unit 454 . Note that the authentication control unit 453 is not essential in the face authentication terminal 400-1 installed at the point A1. Also, the registration unit 452 is not essential in the face authentication terminals 400-2 to 400-n installed from the point A2 to An.

The imaging control unit 451 controls the camera 410 to capture the user U's registration image or authentication image. The registered image and the authentication image are images including at least the user's face area. Note that the photographed image (authentication image) at the point A2 or the like may include scenery, animals, or the like unique to the point in the background. The imaging control section 451 outputs the registration image to the registration section 452 . The imaging control unit 451 also outputs the authentication image to the authentication control unit 453 .

The registration unit 452 transmits a face information registration request including the registration image to the authentication device 100 via the network N. Note that the registration unit 452 may transmit the face information registration request to the information processing apparatus 200 via the network N. FIG. Further, the registration unit 452 transmits a usage registration request to the information processing apparatus 200 via the network N. FIG. Note that the registration unit 452 may transmit the user attribute information received by the input unit 460 to the information processing apparatus 200 via the network N at the time of registration for use. The user attribute information may be included in the usage registration request. Authentication control unit 453 transmits a face authentication request including an authentication image to information processing apparatus 200 via network N, receives the result of face authentication, and outputs the result to display control unit 454 . In addition, the authentication control unit 453 outputs an input screen for information on completion of photographing to the display control unit 454 and prompts the user U to input. The authentication control unit 453 transmits a shooting end request to the information processing apparatus 200 in response to receiving the shooting end information from the user U. FIG.

The display control unit 454 displays on the display unit 440 the display content according to the face authentication result and the degree of smile.

Return to Figure 3 and continue the explanation. A user terminal 500 is an information terminal owned by a user U. FIG. The user terminal 500 is, for example, a mobile phone terminal, a smart phone, a tablet terminal, a PC (Personal Computer) equipped with or connected to a camera, or the like. The user terminal 500 is associated with the user U's user ID or facial feature information. In other words, the user terminal 500 is a display terminal that can be specified by the user ID or facial feature information in the information processing apparatus 200 . For example, the user terminal 500 is a terminal logged in by the user U with his own user ID.

The user terminal 500 transmits a service usage registration request to the information processing device 200 via the network N. Also, the user terminal 500 transmits a registration image used for face authentication of the user U to the authentication device 100, and makes a face information registration request. Note that the user terminal 500 may transmit the facial feature information extracted from the registered image to the authentication device 100 to make a facial information registration request. The user terminal 500 may transmit the registered image and facial feature information to the authentication device 100 via the information processing device 200 . Also, the user terminal 500 acquires an image corresponding to the degree of smile from the information processing apparatus 200 via the network N. FIG.

Next, the user terminal 500 will be described in detail. FIG. 8 is a block diagram showing the configuration of the user terminal 500 according to the second embodiment. User terminal 500 includes camera 510 , storage unit 520 , communication unit 530 , display unit 540 , control unit 550 , and input unit 560 .

The camera 510 is an image capturing device that performs image capturing under the control of the control unit 550 . The storage unit 520 is a storage device in which programs for realizing each function of the user terminal 500 are stored. A communication unit 530 is a communication interface with the network N. FIG. Display unit 540 is a display device. Input unit 560 is an input device that receives an input. The display unit 540 and the input unit 560 may be configured integrally. As an example, the display unit 540 and the input unit 560 are touch panels. The control unit 550 controls hardware of the user terminal 500 . The control unit 550 includes an imaging control unit 551 , a registration unit 552 , an acquisition unit 553 and a display control unit 554 .

The photographing control unit 551 controls the camera 510 to photograph a registered image of the user U. The imaging control section 551 outputs the registration image to the registration section 552 .

The registration unit 552 transmits a face information registration request including the registered image to the authentication device 100 via the network N. Note that the registration unit 552 may transmit the face information registration request to the information processing apparatus 200 via the network N. FIG. Further, the registration unit 552 transmits a service usage registration request to the information processing apparatus 200 via the network N. FIG. Note that the registration unit 552 may transmit the user attribute information received by the input unit 560 to the information processing apparatus 200 via the network N at the time of registration for use. The user attribute information may be included in the usage registration request. Acquisition unit 553 acquires an image corresponding to the degree of smile from information processing apparatus 200 via network N. FIG. Acquisition unit 553 also outputs an image corresponding to the degree of smile to display control unit 554 . The display control unit 554 displays an image corresponding to the degree of smile on the display unit 540 .

Return to Figure 3 and continue the explanation. The information processing apparatus 200 is an information processing apparatus that provides an image corresponding to the degree of smile to the user U using an image captured at a point A1 or the like of the user U, and analyzes the degree of satisfaction with content using collected information. be. The information processing apparatus 200 may be made redundant by a plurality of servers, and each functional block may be realized by a plurality of computers.

The image storage server 300 is one or more file servers for storing images generated by the information processing device 200 according to the degree of smile. Note that the image storage server 300 may store a photographed image for authentication. Upon receiving the image acquisition request, the image storage server 300 provides the requester with an image corresponding to the degree of smile or a photographed image.

Next, the information processing device 200 will be described in detail. FIG. 9 is a block diagram showing the configuration of an information processing apparatus 200 according to the second embodiment. Information processing apparatus 200 includes storage unit 210 , memory 220 , communication unit 230 , and control unit 240 . The storage unit 210 is a storage device such as a hard disk or flash memory. Storage unit 210 stores program 211 , user information 212 , history information 213 , content information 214 , and consolidated information 215 . The program 211 is a computer program in which the processing of the information processing method according to the second embodiment is implemented.

The user information 212 is basic information related to the user. That is, the user information 212 is user attribute information and the like. Specifically, the user information 212 is information in which a user ID 2121 and user attribute information 2122 are associated with each other. The user ID 2121 is information for identifying the user U, and is a user ID notified when face information is registered in the authentication device 100 . The user attribute information 2122 indicates the attribute information of the user U, and corresponds to the "subject's attribute information" according to the first embodiment.

The history information 213 is the shooting history of the user U using the face authentication terminal 400 at each location. Specifically, the history information 213 is information in which a user ID 2131, a spot ID 2132, a date and time 2133, a smile level 2134, and access information 2135 are associated with each other. The user ID 2131 is information for identifying the user U, and is the user ID included in the face authentication result when the face authentication is successful. The point ID 2132 is information for identifying the point where the face authentication terminal 400 that captured the captured image for face authentication is installed. The date and time 2133 is the date and time when the captured image for face authentication was taken or the date and time when face authentication was performed. The degree of smile 2134 is the degree of smile calculated based on the photographed image for face authentication. The access information 2135 is access information of a storage destination of a captured image or a composite image generated based on the captured image. The access information is, for example, link information to the WEB information corresponding to the image, and is specifically a URL or the like.

The content information 214 is information in which the point ID 2141, the content ID 2142, and the content attribute information 2143 are associated with each other. The content ID 2142 is information identifying content associated with the point ID 2141 . For example, the content ID 2142 is the ID of the content when the content providing place exists within a predetermined distance from the location ID 2141 . The content attribute information 2143 is attribute information of the content of the content ID 2142 . For example, the content attribute information 2143 may be the type of content. Examples of types of content may be lion/panda, carnivore/herbivore, or indoor/outdoor. In addition, the content attribute information 2143 may be configured including the location ID 2141 .

The aggregated information 215 is information obtained by aggregating smile degrees based on user U's attribute information and content attribute information. FIG. 10 is a diagram illustrating an example of the data structure of aggregated information 215 according to the second embodiment. In this figure, aggregated information 215 associates user attribute information, content attribute information, smile level, and content satisfaction level. In this figure, age groups (infants, junior high school students, . . . ) are recorded as user attribute information, and animal types (lion, panda, . . . ) are recorded as content attributes.

In this figure, the degree of smile is aggregated for each combination of user attribute information and content attribute information. Content satisfaction is recorded for each combination of user attribute information and content attribute information. The content satisfaction may be an aggregated smileiness statistic (eg, mean). Note that the aggregation information 215 may include a content ID instead of the content attribute information. In this case, the degree of smile may be aggregated for each combination of user attribute information and content ID, and content satisfaction may be recorded for each combination of user attribute information and content ID. From this figure, it can be seen that pandas are more satisfied with content than lions, and that lions are more satisfied with content than pandas are junior high school students.

Return to Fig. 9 and continue the explanation. The memory 220 is a volatile storage device such as a RAM (Random Access Memory), and is a storage area for temporarily holding information when the control unit 240 operates. The communication unit 230 is a communication interface with the network N. FIG.

The control unit 240 is a processor that controls each component of the information processing device 200, that is, a control device. The control unit 240 loads the program 211 from the storage unit 210 into the memory 220 and executes the program 211 . Thereby, the control unit 240 realizes the functions of the registration unit 241 , the authentication control unit 242 , the calculation unit 243 , the recording control unit 244 , the image generation unit 245 , the output unit 246 and the evaluation unit 247 .

The registration unit 241 is an example of the registration unit 11 described above. The registration unit 241 registers the user ID 2121 and the user attribute information 2122 in the storage unit 210 when the face information of the user U is registered by the authentication device 100 at the time of service usage registration and the user ID is notified. Thereby, the attribute information of the user U is associated with the facial feature information 112 stored in the authentication device 100 via the user ID.

Here, the user attribute information 2122 may be information that the information processing apparatus 200 receives via the network N the input data that the user U has input to the face authentication terminal 400 or the user terminal 500 . In this case, the registration unit 241 registers the information provided by the user U in association with the user ID as the user U attribute information.

Also, the user attribute information 2122 may be information generated by the registration unit 241 based on input data from the user U. For example, when the input data from the user U includes schedule information including flight time and itinerary, the registration unit 241 estimates the means of arrival and registers the user attribute information 2122 including the means of arrival in the storage unit 210. you can Note that the registration unit 241 may acquire the schedule information from the schedule application of the user terminal 500 instead of the input data from the user U. This saves the user U the trouble of inputting.

Also, the user attribute information 2122 may be information generated by the registration unit 241 based on a captured image of the user U. The captured image may be a captured image captured for face authentication or face registration. For example, the registration unit 241 may estimate the age and gender of the user U from the captured image, and generate the user attribute information 2122 based on the estimated information. Further, for example, when a companion is detected together with the user U from the photographed image, the registration unit 241 may generate companion information and include it in the user attribute information 2122 . Specifically, the registration unit 241 may determine that the user U has a companion when a plurality of person areas are detected from the captured image. Here, whether or not the person area other than the image area of the user U is the image area of the companion of the user U is determined from the distance between the image area of the user U and other person areas and the sizes of both. You can However, not limited to this, the registration unit 241 can use an existing method. Further, the registration unit 241 may estimate the relationship (group attribute) between the user U and the companions from the captured image, and generate companion information including the group attribute. Note that when estimating various types of attribute information of the user U from the photographed image, the subject to be estimated is not limited to the registration unit 241 and may be the authentication device 100 . In this case, the registration unit 241 receives the attribute information of the user U together with the user ID from the authentication device 100 . In this way, by acquiring the attribute information of the user U from the captured image, it is possible to save the effort of the user U to input.

It should be noted that receiving a service usage registration request may mean obtaining user U's attribute information, as well as permission to utilize data of user U's attribute information, that is, permission to use.

The authentication control unit 242 is an example of the authentication control unit 12 described above. The authentication control unit 242 controls face authentication for the face area of the user U included in the captured image. That is, the authentication control unit 242 controls face authentication for the face area of the user U included in each photographed image photographed at each location. That is, the authentication control unit 242 causes the authentication device 100 to perform face authentication on the captured image acquired from the face authentication terminal 400 . For example, the authentication control unit 242 transmits a face authentication request including the acquired captured image, location ID, and shooting date and time to the authentication device 100 via the network N, and receives the face authentication result from the authentication device 100 . Note that the authentication control unit 242 may detect the face area of the user U from the captured image and include the image of the face area in the face authentication request. Also, the authentication control unit 242 may extract facial feature information from the face area and include the facial feature information in the face authentication request. The authentication control unit 242 then supplies the captured image to the calculation unit 243 and the image generation unit 245 and supplies the face authentication result to the recording control unit 244 .

The calculation unit 243 is also called calculation means. The calculation unit 243 calculates the degree of smile of the user U based on the face area of the user U included in the captured image. Here, a specific example of calculating the degree of smile of the user U will be described. In general, when a person smiles, features such as raised corners of the mouth, lowered corners of the eyes, narrowed eyes, and wrinkles around the mouth appear in the face area. Therefore, the calculation unit 243 may extract feature points of facial features such as both eyes and mouth in the face region of the user U, and calculate the degree of smile based on the coordinates of each extracted feature point. Further, the calculation unit 243 may calculate the degree of smile using a learned discriminator. The discriminator may be a support vector machine (SVM) that takes as input the feature amount of facial organs and outputs whether or not the degree of smile is greater than or equal to a predetermined value. Also, the discriminator may be a convolutional neural network (CNN) that receives the face area of the captured image as input and outputs the degree of smile. Note that the method for calculating the degree of smile is not limited to the above, and existing methods may be used.

Further, when the photographed image includes the face regions of the user U and the companion, the calculation unit 243 calculates the degree of smile based on the degree of smile of the face region of the companion in addition to the degree of smile of the face region of the user U. may be calculated. For example, the calculation unit 243 may calculate an average value of the smile degrees of the user U's face region and the smile degrees of the companion's face regions as the smile degree.

The calculation unit 243 supplies the smile degree information to the recording control unit 244 and the image generation unit 245 .

The recording control unit 244 is an example of the recording control unit 14 described above. When the face authentication is successful, the recording control unit 244 associates the user ID, the location ID, and the date and time included in the face authentication result with the degree of smile calculated by the calculation unit 243, and stores the result in the storage unit 210 as the history information 213. Record. In addition to this, the recording control unit 244 may record the position information of the photographing location by including it in the history information 213 .

The image generation unit 245 is also called image generation means. The image generator 245 uses the degree of smile and the photographed image to identify or generate an image corresponding to the degree of smile. For example, the image generation unit 245 may generate a composite image in which the degree of smile is superimposed on the photographed image as an image corresponding to the degree of smile. Further, when the degree of smile is equal to or greater than a predetermined value, the image generation unit 245 may specify the photographed image as an image corresponding to the degree of smile. Further, when the degree of smile is equal to or greater than a predetermined value, the image generation unit 245 may generate a composite image in which the degree of smile is superimposed on the photographed image as an image corresponding to the degree of smile. The image generator 245 may synthesize the generated or specified images using a predetermined template, and use the synthesized image as an image corresponding to the degree of smile. The predetermined template may be a template related to content associated with the shooting location. For example, if the shooting location is associated with a panda, the template may include an image of a panda. The image generation unit 245 saves the image corresponding to the degree of smile in the image saving server 300 and records the access information of the saving destination in the storage unit 210 as the history information 213 .

The output unit 246 is an example of the output unit 16 described above. The output unit 246 outputs an image corresponding to the degree of smile to the user terminal 500 via the network N when a predetermined provision condition is satisfied. The predetermined provision condition may be that the communication unit 230 receives a photographing end request from the face authentication terminal 400 after successful face authentication based on the captured image. Here, images corresponding to smile degrees are stored in the image storage server 300 . Therefore, in this case, the output unit 246 may acquire an image corresponding to the degree of smile from the image storage server 300 using the access information corresponding to the user ID included in the history information 213 and transmit the image to the user terminal 500 . . The images corresponding to the degree of smile acquired at this time may be a plurality of images corresponding to images taken at different shooting dates and times. Accordingly, when there are a plurality of images corresponding to the degree of smile, the user U can acquire the images collectively. Note that the predetermined provision condition may simply be that face authentication based on the captured image has succeeded. In this case, the output unit 246 transmits information according to the degree of smile to the user terminal 500 when the face authentication based on the captured image is successful even if the request to end the photographing is not received.

Here, the output unit 246 may determine information corresponding to the degree of smile to be output to the user terminal 500 according to the amount of attribute information provided by the user U. The attribute information provided by the user U may be attribute information input by the user U or attribute information licensed by the user U. For example, the output unit 246 may determine the number of images to be output according to the amount of attribute information provided by the user U. FIG. As a result, the incentives for providing attribute information and permitting use are more likely to work, and the acquisition of attribute information is further facilitated.

The evaluation unit 247 is an example of the evaluation unit 17 described above. The evaluation unit 247 uses the user information 212 , the history information 213 and the content information 214 to generate aggregate information 215 . Specifically, the evaluation unit 247 acquires the user attribute information of the user information 212 using the user ID corresponding to the degree of smile of the history information 213 . Also, the evaluation unit 247 acquires the content attribute information or the content ID of the content information 214 using the point ID corresponding to the degree of smile of the history information 213 . Then, the evaluation unit 247 associates the degree of smile, the user attribute information, and the content attribute information or the content ID, and records them as aggregated information 215 . Then, the evaluation unit 247 aggregates the degree of smile for each combination of user attribute information and content attribute information and for each combination of user attribute information and content ID, and calculates content satisfaction based on the aggregated degree of smile. As a result, content IDs and content attributes with high satisfaction for each user attribute can be determined, for example, "Pandas are more popular than lions and carnivores are more popular than herbivores among junior high school students." It is possible to comprehend.

In addition to or instead of the user attribute information and the content information, the evaluation unit 247 aggregates the degree of smile based on at least one of the position information of the shooting location, the shooting time period, and the order of visits to the shooting location. good too. Here, the location information of the shooting location indicates the location information of the content, the shooting time zone indicates the time zone in which the content is experienced, and the visit order of the shooting location indicates the order in which the content is experienced. Then, the evaluation unit 247 calculates the degree of content satisfaction based on the degree of smile aggregated based on various parameters. As a result, the evaluation unit 247 can evaluate the degree of satisfaction of the content from various evaluation axes, enabling detailed analysis. For example, it is possible to analyze that ``when the panda viewing spot is closer to the entrance of the zoo, the elderly people are more satisfied with the overall content than when it is farther away''.

FIG. 11 is a flowchart showing the flow of usage registration processing according to the second embodiment. First, the registration unit 241 receives a usage registration request via the network N (S401). Here, the request source is assumed to be either the face authentication terminal 400 or the user terminal 500 . It should be noted that, at this time, the registration unit 241 requests the authentication device 100 via the network N to register the face information in the face information DB 110 when the photographed image is received in addition to the use registration request.

Then, the registration unit 241 acquires the user ID from the authentication device 100 via the network N (S402). Then, the registration unit 241 acquires the attribute information of the user U via the network N from the face authentication terminal 400 or the user terminal 500 that made the request (S403). As described above, instead of this, the registration unit 241 may generate attribute information of the user U based on the input data of the user U or the captured image. Then, the registration unit 241 registers the attribute information of the user U as the user attribute information 2122 of the user information 212 in association with the user ID 2121 (S404). Then, the registration unit 241 terminates the processing.

FIG. 12 is a flowchart showing the flow of image output processing according to the second embodiment. First, the authentication control unit 242 acquires a captured image from the face authentication terminal 400 via the network N (S411). Next, the authentication control unit 242 transmits a face authentication request to the authentication device 100 via the network N (S412). At this time, the authentication control unit 242 includes at least one of the photographed image acquired in step S411, the face area extracted from the photographed image, or the facial feature information extracted from the face area in the face authentication request. The authentication control unit 242 then receives the face authentication result from the authentication device 100 via the network N (S413). The face authentication result includes the fact and the user ID when the face authentication is successful, and includes the fact when the face authentication is unsuccessful.

The authentication control unit 242 determines whether face authentication has succeeded (S414). When it is determined that the face authentication has failed (No in S414), the authentication control unit 242 outputs that the face authentication has failed (S415). Specifically, the authentication control unit 242 transmits a message to the face authentication terminal 400 that provides the image via the network N to the effect that the face authentication has failed. The authentication control unit 242 then terminates the process.

On the other hand, if it is determined that face authentication has succeeded (Yes in S414), the authentication control unit 242 identifies the user ID for which face authentication has succeeded (S416). Specifically, the authentication control unit 242 extracts the user ID included in the face authentication result. At this time, the authentication control unit 242 may output that the face authentication was successful. Specifically, the authentication control unit 242 may transmit a message to the face authentication terminal 400 that provides the image via the network N to the effect that the face authentication has succeeded.

Next, the calculation unit 243 calculates the degree of smile based on the captured image (S417). Subsequently, the image generation unit 245 generates an image corresponding to the degree of smile based on the photographed image and the degree of smile (S418). The image generation unit 245 stores the generated image corresponding to the degree of smile in the image storage server 300 via the network N (S419). The image generation unit 245 then acquires the access information of the image storage destination from the image storage server 300 . Subsequently, the recording control unit 244 registers the history information 213 (S420). Specifically, the recording control unit 244 associates the user ID, the spot ID, the date and time, the degree of smile, and the access information, and records them in the storage unit 210 as the history information 213 . The image generation unit 245 also outputs an image corresponding to the degree of smile to the face authentication terminal 400 via the network N (S421). As a result, the face authentication terminal 400 displays an image corresponding to the degree of smile on the display unit 440 . Subsequently, the output unit 246 determines whether or not a predetermined provision condition is satisfied (S422). For example, the output unit 246 determines whether or not a predetermined provision condition is satisfied by determining whether or not the communication unit 230 has received a shooting end request from the face authentication terminal 400 . When the output unit 246 determines that the predetermined provision condition is not satisfied (No in S422), the process ends. On the other hand, when the output unit 246 determines that the predetermined provision condition is satisfied (Yes in S422), the output unit 246 acquires an image corresponding to the degree of smile from the image storage server 300 using the access information of the history information 213 ( S423). The output unit 246 then transmits the image corresponding to the degree of smile to the user terminal 500 via the network N (S424).

Note that the image corresponding to the degree of smile may be generated after the output unit 246 determines that the provision condition is satisfied. In this case, specifically, step S418 is omitted, and instead of step S419, the image generator 245 stores the captured image in the image storage server 300 via the network N. The image generation unit 245 then acquires the access information of the image storage destination from the image storage server 300 . In step S<b>421 , the image generator 245 may output only the smile level information to the face authentication terminal 400 via the network N. Then, instead of step S423, the output unit 246 acquires the captured image corresponding to the user U from the image storage server 300 using the access information of the history information 213. FIG. Then, the image generation unit 245 generates an image corresponding to the degree of smile based on the captured image and the degree of smile that have been acquired. In step S<b>424 , the output unit 246 transmits the generated image corresponding to the degree of smile to the user terminal 500 via the network N.

FIG. 13 is a flowchart showing the flow of evaluation processing according to the second embodiment. This evaluation process may be executed periodically, or may be executed when a predetermined execution condition is satisfied, such as when the history information 213 is updated. First, the evaluation unit 247 determines whether or not to start the evaluation process (S431). If it is determined to start (Yes in S431), a smile The degrees are aggregated (S432), and aggregated information 215 is recorded. The evaluation unit 247 calculates the degree of content satisfaction based on the aggregated degree of smile (S433). The evaluation unit 247 then outputs the calculated content satisfaction level to the outside in a predetermined format together with the user attribute information and content attribute information (or content ID) (S434).

FIG. 14 is a sequence diagram showing an example of the flow of usage registration processing according to the second embodiment. First, at point A1, face authentication terminal 400-1 transmits a service use registration request to information processing apparatus 200 (S500). Further, the face authentication terminal 400-1 photographs the user U (S501), and transmits a face information registration request including the photographed image to the authentication device 100 via the network N (S502). Then, the authentication device 100 registers face information (face feature information) of the user U based on the captured image included in the received face information registration request (S503). The authentication device 100 then notifies the information processing device 200 of the user ID via the network N (S504). The face authentication terminal 400-1 also transmits the attribute information of the user U to the information processing apparatus 200 via the network N (S505). The information processing apparatus 200 associates the notified user ID with the attribute information of the user U and registers them in the user information 212 (S506). Then, the user U moves from the point A1 to the point A2.

15 and 16 are sequence diagrams showing an example of the flow of image output processing according to the second embodiment. At the point A2, the face authentication terminal 400-2 takes an image of the user U for whom usage registration has been completed (S510), and transmits the taken image and the point ID to the information processing apparatus 200 via the network N (S511). The information processing apparatus 200 transmits a face authentication request for the face area of the user U in the captured image included in the received face authentication request to the authentication apparatus 100 via the network N (S512). Then, the authentication device 100 performs face authentication on the face area of the user U in the captured image included in the received face authentication request (S513). Here, it is assumed that face authentication has succeeded. The authentication device 100 transmits the face authentication result including the fact that the face authentication was successful and the user ID to the information processing device 200 via the network N (S514).

Subsequently, the information processing apparatus 200 calculates the degree of smile from the photographed image (S515). The information processing apparatus 200 then generates a composite image by superimposing the degree of smile on the photographed image (S516). The information processing apparatus 200 saves the generated composite image in the image saving server 300 (S517). Then, the information processing apparatus 200 registers the user ID, the spot ID, the date and time, the degree of smile, and the access information of the save destination as the history information 213 (S518). The information processing device 200 also transmits the generated composite image to the face authentication terminal 400-2 via the network N (S519). Face authentication terminal 400-2 that has received the synthesized image displays the synthesized image (S520). The face authentication terminal 400-2 displays an input screen for shooting end information and accepts input from the user U. FIG. Here, it is assumed that the face authentication terminal 400-2 does not accept an input operation of shooting end information from the user U, but accepts an input operation of continuation information indicating that the user U does not wish to end (S521). The face authentication terminal 400-2 transmits a continuation request including continuation information to the information processing device 200 via the network N (S522). Then, the user U moves from the point A2 to the point A3.

At the point A3, the face authentication terminal 400-3 takes an image of the user U (S530), and transmits the taken image and the point ID to the information processing device 200 via the network N (S531). Then, processing similar to steps S512 to S520 is executed. The face authentication terminal 400-3 displays an input screen for the shooting end information and accepts input from the user U. FIG. Here, it is assumed that the face authentication terminal 400-3 receives an input operation of shooting end information from the user U (S532). The face authentication terminal 400-3 transmits a photographing end request including photographing end information to the information processing apparatus 200 via the network N (S533). The information processing apparatus 200 that has received the shooting end request accesses the image storage server 300 using the access information corresponding to the user ID (S534), and acquires the composite image (S535). The information processing apparatus 200 then transmits the acquired composite image to the user terminal 500 corresponding to the user ID (S536).

FIG. 17 is a diagram showing an example of a composite image 620 according to the second embodiment. A composite image 620 shown in the figure is a composite image generated in step S516 of FIG. 15, displayed by the face authentication terminal 400 in step S520, and acquired by the user terminal 500 in step S536 of FIG. A composite image 620 is an image obtained by superimposing a degree of smile on a photographed image. The composite image 620 includes in its background an image area of an animal representing the features of the content associated with the shooting location.

As described above, the information processing apparatus 200 according to the second embodiment aggregates smile levels based on user attribute information and other information related to the user U, and performs detailed analysis of content satisfaction levels. Therefore, it becomes easier for the operator of the event site to take measures to improve the satisfaction of the content. In addition, for the user U, since information corresponding to the degree of smile can be obtained as a benefit, incentives to provide the user's attribute information and permit usage are likely to work. As a result, the information processing apparatus 200 can easily acquire the attribute information and facilitate detailed analysis. Therefore, it is possible to more preferably analyze the degree of satisfaction with the content at the event site.

Note that the output unit 246 may transmit an image corresponding to the degree of smile to the user terminal 500 only when the user U desires. In this case, the face authentication terminal 400 that has received the input operation of the shooting end information in step S532 of FIG. 16 may display an input screen for receiving input as to whether or not to obtain an image corresponding to the degree of smile.

FIG. 18 is a diagram showing an example of a terminal screen 610 according to the second embodiment. A terminal screen 610 shown in the figure is displayed after step S532 in FIG. 16 is executed. The terminal screen 610 includes an input area for receiving an input as to whether or not the user U wishes to transmit the displayed composite image 620 along with a message "Can I send this picture?" Then, instead of step S533 in FIG. 16, the face authentication terminal 400 transmits to the information processing apparatus 200 information indicating whether or not acquisition is desired together with a request to end shooting. Note that the terminal screen 610 shown in this figure may be displayed in step S520 of FIG. In this case, the information processing apparatus 200 may record whether or not each photographed image is desired to be acquired. 16, the information processing apparatus 200 may acquire the desired image from the image storage server 300 and transmit it to the user terminal 500. FIG.

In the above description, the output unit 246 transmits an image corresponding to the degree of smile to the user terminal 500 in order to provide it to the user U, but it may transmit access information. For example, instead of steps S534 to S536 in FIG. 16, the output unit 246 generates an image download guidance screen including access information, and transmits the generated guidance screen information. The destination is not limited to the user terminal 500 and may be the face authentication terminal 400 .

FIG. 19 is a diagram showing an example of a terminal screen 610 according to the second embodiment. Terminal screen 610 includes access information 611 . A terminal screen 610 shown in this figure is displayed when the face authentication terminal 400 receives the information of the download guidance screen. The access information 611 in this figure is shown as a QR code (registered trademark). In other words, the display of the access information 611 indicates that the access information 611 has been presented to the user U. FIG. Here, the user terminal 500 reads the access information 611 in the terminal screen 610 displayed on the face authentication terminal 400 according to the user U's operation. Then, the user terminal 500 analyzes the read access information 611 and transmits an image acquisition request to the image storage server 300 via the network N based on the analysis result. The image acquisition request may simply be an access (request message) from the user terminal 500 to a predetermined storage destination of the image storage server 300 . The image storage server 300 reads the synthetic image stored in the storage destination indicated by the image acquisition request from the user terminal 500 and transmits a response including the synthetic image to the user terminal 500 via the network N.

The image corresponding to the degree of smile provided to the user U in step S536 of FIG. good.
FIG. 20 is a diagram showing an example of an overall synthetic image 630 according to the second embodiment. The overall composite image 630 includes a plurality of images (

composite images

631, 632, 633, 634, and 635) corresponding to the degree of smile of the user U whose face authentication has succeeded, and which correspond to the images taken at different shooting dates and times. The overall composite image 630 is composited so that each of the composite images 631 to 635 can be individually identified.

When a predetermined provision condition is satisfied, the image generation unit 245 uses the access information corresponding to the user ID included in the history information 213 to generate an image (a photographed image or an image corresponding to the degree of smile) from the image storage server 300. get. When acquiring a plurality of images, the image generation unit 245 synthesizes the plurality of images so as to be individually identifiable to generate an overall composite image. At this time, the image generation unit 245 may select images shot at different shooting locations from among the plurality of images, and generate an overall composite image from the selected images. Further, the image generation unit 245 may select an image having a degree of smile equal to or greater than a predetermined value from among the plurality of images, and generate an overall composite image from the selected images. In addition, the image generation unit 245 may select images in descending order of the degree of smile from among the plurality of images, and generate an overall composite image from the selected images. For example, the image generation unit 245 selects a predetermined number of images with the highest degree of smile from among the plurality of images, and generates an overall composite image from the selected images. The image generator 245 may also select an image with a degree of smile equal to or greater than a predetermined value or with the highest degree of smile for each shooting location, and generate an overall composite image from the selected images. The output unit 246 then provides the user U with the generated overall composite image. At this time, the output unit 246 may cause the user terminal 500 to display the generated overall composite image. The output unit 246 may cause the user terminal 500 to display individual images included in the overall composite image in an identifiable manner in addition to the overall composite image.

In this way, the user U who has visited each location can acquire an overall composite image that aggregates the captured images taken when face authentication was performed at each location by performing face authentication at a specific location. Therefore, the user U can easily comprehend the content that he/she likes out of the content that he or she has actually experienced. Furthermore, if it is an overall composite image, the user U can easily publish it on an SNS (Social Networking Service) or the like. Therefore, the recognition of the content by many people can contribute to the revitalization of the event site.

It should be noted that the present disclosure is not limited to the above embodiments, and can be modified as appropriate without departing from the scope. For example, in the second embodiment described above, the authentication device 100 is connected to the information processing device 200 via the network N. FIG. However, instead of this, the functions of the face detection unit 120 , the feature point extraction unit 130 , the registration unit 140 and the authentication unit 150 of the authentication device 100 may be included in the control unit 240 of the information processing device 200 . Moreover, the face information DB 110 of the authentication device 100 may be included in the storage unit 210 of the information processing device 200 .

Also, in the second embodiment described above, the image storage server 300 is connected to the information processing apparatus 200 via the network N. However, instead of this, the functions of image storage server 300 may be included in storage unit 210 of information processing apparatus 200 .

Also, in the second embodiment described above, the calculation unit 243 of the information processing apparatus 200 calculates the degree of smile based on the photographed image. However, instead of this, authentication device 100 may calculate the degree of smile based on the photographed image. In this case, the calculation unit 243 of the information processing device 200 may supply the smile level information to the recording control unit 244 and the image generation unit 245 in response to receiving the smile level information from the authentication device 100 .

In addition, in the above-described embodiment, the hardware configuration has been described, but the configuration is not limited to this. The present disclosure can also implement arbitrary processing by causing a CPU to execute a computer program.

In the above examples, the program includes instructions (or software code) that, when read into a computer, cause the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or tangible storage medium. By way of example, and not limitation, computer readable media or tangible storage media may include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSD) or other memory technology, CDs - ROM, digital versatile disc (DVD), Blu-ray disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disc storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example, and not limitation, transitory computer readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

Some or all of the above-described embodiments can also be described in the following supplementary remarks, but are not limited to the following.
(Appendix 1)
a registration unit that registers attribute information of a subject in association with biometric information of the subject when a request for use registration is acquired;
an authentication control unit for performing biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with a predetermined content is acquired;
a recording control unit that, when the biometric authentication is successful, records the degree of smile of the subject calculated from the captured image in association with attribute information of the subject and content information related to content associated with a shooting location;
an output unit that outputs information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication has succeeded;
An information processing apparatus comprising: an evaluation unit that aggregates smile levels based on subject's attribute information and content information, and evaluates content satisfaction based on the aggregated smile levels.
(Appendix 2)
The subject's attribute information includes at least one of the subject's age, gender, occupation, family structure, hobby, companion information indicating the presence or absence of companions or group attributes, and means of visit. Information processing equipment.
(Appendix 3)
The registration unit registers information provided by the subject in association with the subject as attribute information of the subject,
Information according to appendix 1 or 2, wherein the output unit determines information according to the degree of smile to be output to an information terminal used by the subject, according to the amount of information provided by the subject. processing equipment.
(Appendix 4)
The output unit
outputting information corresponding to the degree of smile of the subject to the information terminal in response to the success of the biometric authentication based on the acquired photographed image;
or
any of Supplementary Notes 1 to 3, wherein information corresponding to a plurality of smile levels corresponding to images taken at different shooting dates and times of the subject is output to the information terminal when a photographing end request indicating the end of photographing is acquired. or the information processing device according to claim 1.
(Appendix 5)
the information corresponding to the degree of smile is an image corresponding to the degree of smile generated based on a photographed image for which biometric authentication is successful;
The image corresponding to the degree of smile is a composite image in which the degree of smile of the subject is superimposed on the photographed image, the photographed image in which the degree of smile of the subject is a predetermined value or more, or the degree of smile of the subject is a predetermined value. 5. The information processing apparatus according to any one of appendices 1 to 4, wherein the composite image corresponds to a photographed image having a value equal to or greater than the value.
(Appendix 6)
Further comprising an image generation unit that generates the overall composite image of the subject,
The overall composite image includes a plurality of images corresponding to the degree of smile corresponding to images taken at different shooting dates and times of the subject,
The output unit outputs the composite image to the information terminal used by the subject when the biometric authentication based on the acquired photographed image is successful and the photographing end request indicating the end of photographing is acquired. The information processing device according to .
(Appendix 7)
The recording control unit records the degree of smile in association with at least one of position information of the shooting location and shooting date and time,
7. The information processing apparatus according to any one of appendices 1 to 6, wherein the evaluation unit aggregates the degree of smile based on at least one of position information of shooting locations, shooting time period, and visit order of shooting locations.
(Appendix 8)
an information terminal that generates a captured image of a target person;
an information processing device communicably connected to the information terminal,
The information processing device is
a registration unit that registers attribute information of a subject in association with biometric information of the subject when use registration is accepted;
an authentication control unit that performs biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with a predetermined content is obtained from the information terminal;
a recording control unit that, when the biometric authentication is successful, records the degree of smile of the subject calculated from the captured image in association with attribute information of the subject and content information related to content associated with a shooting location;
an output unit that outputs information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication has succeeded;
An information processing system, comprising: an evaluation unit that aggregates smile levels based on target person attribute information and content information, and evaluates content satisfaction based on the aggregated smile levels.
(Appendix 9)
The subject's attribute information includes at least one of the subject's age, gender, occupation, family structure, hobby, companion information indicating the presence or absence of companions or group attributes, and means of visit. Information processing system.
(Appendix 10)
registering attribute information of a subject in association with biometric information of the subject when a request for use registration is obtained;
a step of performing biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with a predetermined content is obtained;
a step of recording, when the biometric authentication is successful, the degree of smile of the subject calculated from the captured image in association with content information related to content associated with the subject's attribute information and a shooting location;
a step of outputting information according to the degree of smile of the subject to an information terminal used by the subject whose biometric authentication is successful;
An information processing method comprising: aggregating smile levels based on subject's attribute information and content information; and evaluating content satisfaction based on the aggregated smile levels.
(Appendix 11)
to the computer,
a registration process of registering attribute information of a subject in association with biometric information of the subject when a request for use registration is acquired;
Authentication control processing for performing biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with predetermined content is acquired;
recording control processing for recording, when the biometric authentication is successful, the degree of smile of the subject calculated from the captured image in association with content information related to content associated with the subject's attribute information and a shooting location;
An output process for outputting information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication has succeeded;
A program for executing an evaluation process of aggregating smile levels based on subject's attribute information and content information, and evaluating content satisfaction based on the aggregated smile levels.

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the invention.

This application claims priority based on Japanese Patent Application No. 2021-023729 filed on February 17, 2021, and the entire disclosure thereof is incorporated herein.

The information processing device and information processing system according to this embodiment can be used, for example, to operate an event venue.

REFERENCE SIGNS LIST 10 information processing device 11 registration unit 12 authentication control unit 14 recording control unit 16 output unit 17 evaluation unit 100 authentication device 110 face information DB
111 User ID
112 facial feature information 120 face detection unit 130 feature point extraction unit 140 registration unit 150 authentication unit 200 information processing device 300 image storage server 400 face authentication terminal 410 camera 420 storage unit 430 communication unit 440 display unit 450 control unit 451 photographing control unit 452 Registration unit 453 Authentication control unit 454 Display control unit 460 Input unit 500 User terminal 510 Camera 520 Storage unit 530 Communication unit 540 Display unit 550 Control unit 551 Shooting control unit 552 Registration unit 553 Acquisition unit 554 Display control unit 560 Input unit 210 Storage unit 211 program 212 user information 2121 user ID
2122 User attribute information 213 History information 2131 User ID
2132 Point ID
2133 Date and time 2134 Degree of smile 2135 Access information 214 Content information 2141 Spot ID
2142 Content ID
2143 content attribute information 215 aggregation information 220 memory 230 communication unit 240 control unit 241 registration unit 242 authentication control unit 243 calculation unit 244 recording control unit 245 image generation unit 246 output unit 247 evaluation unit 610 terminal screen 611 access information 620 synthetic image 630 whole

Synthetic image

631, 632, 633, 634, 635 Synthetic image 1000 Information processing system A1, A2, A3, An Point N Network U User

Claims

registration means for registering attribute information of a subject in association with biometric information of the subject when a request for use registration is acquired;
Authentication control means for performing biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with a predetermined content is acquired;
recording control means for recording, when the biometric authentication is successful, the degree of smile of the subject calculated from the photographed image in association with content information related to content associated with the subject's attribute information and a photographing location;
Output means for outputting information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication is successful;
An information processing apparatus comprising: evaluation means for aggregating smile levels based on subject's attribute information and content information, and evaluating content satisfaction based on the aggregated smile levels.
2. The target person's attribute information includes at least one of age, gender, occupation, family structure, hobby, companion information indicating the presence or absence of companions or group attributes of the target person, and means of visiting. information processing equipment.
The registration means registers information provided by the subject in association with the subject as attribute information of the subject,
3. The output unit according to claim 1 or 2, wherein the output means determines the information according to the degree of smile to be output to the information terminal used by the target person according to the amount of information provided by the target person. Information processing equipment.
The output means is
outputting information corresponding to the degree of smile of the subject to the information terminal in response to the success of the biometric authentication based on the acquired photographed image;
or
4. Outputs to the information terminal information according to a plurality of smile levels corresponding to images taken at different shooting dates and times of the subject, when a shooting end request indicating the end of shooting is acquired. The information processing device according to any one of the items.
the information corresponding to the degree of smile is an image corresponding to the degree of smile generated based on a photographed image for which biometric authentication is successful;
The image corresponding to the degree of smile is a composite image in which the degree of smile of the subject is superimposed on the photographed image, the photographed image in which the degree of smile of the subject is a predetermined value or more, or the degree of smile of the subject is a predetermined value. The information processing apparatus according to any one of claims 1 to 4, wherein the composite image corresponds to a photographed image having a value equal to or greater than the value.
further comprising an image generating means for generating the overall composite image of the subject;
The overall composite image includes a plurality of images corresponding to the degree of smile corresponding to images taken at different shooting dates and times of the subject,
The output means outputs the composite image to the information terminal used by the subject when the biometric authentication based on the acquired photographed image is successful and the photographing end request indicating the end of photographing is acquired. 6. The information processing device according to 5.
The recording control means records the degree of smile in association with at least one of position information of the shooting location and shooting date and time;
7. The information processing apparatus according to any one of claims 1 to 6, wherein the evaluation unit aggregates smile levels based on at least one of location information of shooting locations, shooting time zones, and order of visits to shooting locations. .
an information terminal that generates a captured image of a target person;
an information processing device communicably connected to the information terminal,
The information processing device is
registration means for registering attribute information of a subject in association with biometric information of the subject when use registration is accepted;
Authentication control means for performing biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with a predetermined content is obtained from the information terminal;
recording control means for recording, when the biometric authentication is successful, the degree of smile of the subject calculated from the photographed image in association with content information related to content associated with the subject's attribute information and a photographing location;
Output means for outputting information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication is successful;
An information processing system, comprising: evaluation means for aggregating smile levels based on subject's attribute information and content information, and evaluating content satisfaction based on the aggregated smile levels.
9. The subject's attribute information includes at least one of the subject's age, gender, occupation, family structure, hobbies, companion information indicating the presence or absence of companions or group attributes, and means of visit. information processing system.
When the user registration request is obtained, registering the attribute information of the subject in association with the biometric information of the subject,
When a photographed image of a target person photographed at a predetermined point associated with predetermined content is acquired, biometric authentication is performed using the photographed image,
when the biometric authentication is successful, recording the degree of smile of the subject calculated from the captured image in association with content information related to content associated with the subject's attribute information and the shooting location;
Outputting information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication is successful,
An information processing method, comprising aggregating smile levels based on target person attribute information and content information, and evaluating content satisfaction based on the aggregated smile levels.
to the computer,
a registration process of registering attribute information of a subject in association with biometric information of the subject when a request for use registration is acquired;
Authentication control processing for performing biometric authentication using the captured image when a captured image of a target person captured at a predetermined point associated with predetermined content is acquired;
recording control processing for recording, when the biometric authentication is successful, the degree of smile of the subject calculated from the captured image in association with content information related to content associated with the subject's attribute information and a shooting location;
An output process for outputting information according to the degree of smile of the subject to the information terminal used by the subject whose biometric authentication has succeeded;
A non-temporary computer storing a program for executing an evaluation process of aggregating smile levels based on attribute information of a subject and content information, and evaluating content satisfaction based on the aggregated smile levels. readable medium.