JP2014042235A - Audience rating survey system, facial expression information generation device, and facial expression information generation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an audience rating survey system, a facial expression information generation device, and a facial expression information generation program that are capable of taking statistics of facial expressions of persons and the number of the persons.SOLUTION: A facial expression information generation device comprises: acquisition means that acquires moving image data; person detection means that detects persons from the moving image data; facial expression detection means that detects facial expressions from the faces of the detected persons; storage means that stores the number of the persons, the facial expressions, and dates and times when the facial expressions are detected in an associated manner; and statistics means that takes statistics of the facial expressions and the number of the persons at each specific time. Thereby, facial expression information useful for marketing uses is obtained.

Description

  The present invention relates to an audience rating survey system, a facial expression information generation device, and a facial expression information generation program.

  2. Description of the Related Art Conventionally, a number determination device has been proposed in which a person is detected from shooting data shot by a video camera and the number of people is counted (see, for example, Patent Document 2).

JP-A-8-272878

  However, there is a demand not only to count the number of photographed persons but also to acquire the facial expressions of the persons as information.

The present invention has been made in view of the above points, and an object of the present invention is to provide a facial expression information generation method and facial expression information generation program capable of generating useful information regarding a facial expression of a person.

  In order to solve such a problem, the expression information generation apparatus of the present invention detects an expression from an acquisition unit that acquires moving image data, a person detection unit that detects a person from the moving image data, and a detected human face. Expression means detecting means, storage means for storing the number of persons, expression and date and time when the expression is detected in association with each other, and statistical means for statistically calculating the number of expressions and the number of persons for each specific time I made it.

  As a result, the facial expression information generating means can statistic the facial expressions and the number of persons at specific times, so that various information can be obtained from the facial expressions of the persons at the shooting location.

  Furthermore, the expression information generation program of the present invention includes an acquisition step of acquiring moving image data, a person detection step of detecting a person from the moving image data, an expression detection step of detecting an expression from the detected face of the person, A storing step for storing the number of persons, an expression, and the date and time when the expression is detected in association with each other, and a statistical step for statistically calculating the number of expressions and the number of persons at a specific time are provided.

  As a result, the facial expression information generating apparatus can statistic the facial expressions and the number of persons at specific time intervals, so that various information can be obtained from the facial expressions of the persons at the shooting location.

  Furthermore, the expression information generation apparatus of the present invention detects a person from moving image data, detects expression from the detected person's face, and detection information determination that determines the age and sex of the person as detection information. And screen data generating means for generating screen data so that the color-coded frame and the degree of expression corresponding to the detection information correspond to the movement of the person in the moving image data.

  Thereby, the facial expression information generating device can display the frame color-coded according to the age and sex of the person so that the user can recognize the age and sex of the person at a glance, and the relationship between the age and the facial expression. Can be easily seen.

  Further, the expression generation program of the present invention detects a person from moving image data, detects an expression from the detected person's face, and a detection information determination step that determines the age and sex of the person as detection information And a screen data generation step for generating screen data so that the color-coded frame and the degree of expression corresponding to the detection information correspond to the movement of the person in the moving image data.

  Thereby, the facial expression information generation program can make the user recognize the age and gender of the person at a glance by displaying a color-coded frame according to the age and sex of the person, and the relationship between the age and the facial expression. Can be easily seen.

Furthermore, the audience rating survey system of the present invention includes display means for displaying video data, photographing means for photographing a person who views the video data, and transmitting means for transmitting moving image data photographed by the photographing means. A terminal device; an acquisition unit that acquires moving image data; a person detection unit that detects a person from the moving image data; an expression detection unit that detects an expression from the detected face of the person; the number of persons; The facial expression information generating device includes a storage unit that stores the date and time when a facial expression is detected in association with each other and a statistical unit that statistically counts the number of facial expressions and the number of persons at a specific time.

  Thus, the audience rating survey system can detect not only the number of viewers who are viewing the video data but also the facial expressions of the viewers, so that the viewer's reaction to the video data can be known.

  The audience rating survey system according to the present invention includes video data display means, photographing means for photographing a person who views the video data, person detecting means for detecting a person from moving image data photographed by the photographing means, A terminal apparatus having facial expression detection means for detecting facial expressions from the detected human faces, a transmission means for associating and transmitting the number of persons, facial expressions, and time information of video data, and corresponding to the video data And a statistical device having statistical means for statistically counting facial expressions and the number of persons.

  Thus, the audience rating survey system can detect not only the number of viewers who are viewing the video data but also the facial expressions of the viewers, so that the viewer's reaction to the video data can be known.

  In addition, since the facial expression information generation apparatus and the facial expression information generation program of the present invention can statistic the facial expression and the number of persons at specific times, various information can be obtained from the facial expressions of the person at the shooting location. Can be obtained. For this reason, useful information regarding the facial expression of a person can be generated.

  Furthermore, the facial expression information generation device and facial expression information generation program of the present invention can display a frame color-coded according to a person's age and gender so that the user can recognize the person's age and gender at a glance. , Make it easy to see the relationship between age and facial expression. For this reason, it is possible to convey the age and facial expression to the user in an easy-to-understand manner.

Furthermore, since the audience rating survey system of the present invention can detect not only the number of viewers who are viewing video data but also the facial expressions of viewers, the viewer's reaction to the video data can be known. Information useful for the user can be obtained.

It is a basic diagram which shows the structure of a database creation system. It is a block diagram of a computer in a database creation system. It is a basic diagram which shows a face display screen. It is a flowchart which shows a moving image data acquisition process procedure. It is a flowchart which shows an apparent information determination process procedure. It is a basic diagram which shows the structure of a facial expression information generation system. It is a block diagram of the computer in a facial expression information generation system. It is a basic diagram which shows a smile degree display screen. It is a flowchart which shows a facial expression information generation process procedure. It is a basic diagram with which it uses for description of the detection of a moving direction. It is a basic diagram with which it uses for description of the detection of the moving direction to right and left. It is a basic diagram with which it uses for description of the detection (1) of the movement direction to the front and back. It is a basic diagram with which it uses for description of the detection of a back-and-forth motion direction (2). It is a basic diagram which shows the structure of the database creation system by 3rd Embodiment. It is a basic diagram which shows the structure of the facial expression information generation system by 3rd Embodiment. It is a flowchart which shows the facial expression information generation process procedure by 3rd Embodiment. It is a basic diagram which shows the structure of the facial expression information generation system by 4th Embodiment. It is a flowchart which shows the facial expression information generation process procedure by 4th Embodiment. It is a basic diagram which shows the smile degree display screen in other embodiment. It is a flowchart which shows the correction value determination processing procedure by 5th Embodiment. It is a basic diagram which shows the structure of the audience rating investigation system by 7th Embodiment. It is a basic diagram which shows the investigation result display process screen (1) by 7th Embodiment. It is a basic diagram which shows the investigation result display process screen (2) in 7th Embodiment.

  Next, modes for carrying out the present invention will be described with reference to the drawings.

<First Embodiment>
[1-1. Create database]

  2. Description of the Related Art Conventionally, there has been proposed an apparatus configured to estimate an actual age of a person based on a plurality of feature amounts in the person's face and their distribution.

JP 2004-318632 A

  However, the actual age does not always match the user's preference, and the estimated actual age information is not optimal for marketing use.

  The present invention has been made to solve such problems, and an object of the present invention is to provide a database creation apparatus and a database creation program that can provide useful information as marketing information.

  FIG. 1 shows an example of a database creation system 1 according to the present invention. The database creation system 1 includes a video camera 2 as a face image acquisition unit and a computer 3 as a database creation device.

  As the video camera 2, for example, a simple moving image photographing device such as a WEB camera is used in addition to a camcorder. The video camera 2 is installed with permission in a company employee cafeteria, for example, and moving image data including a human face image is taken.

The computer 3 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access) (not shown).
A system controller 11 composed of a memory) controls the entire computer 3 in an integrated manner. The system controller 11 expands a basic program or a database creation program stored in the storage unit 12 such as a flash memory, a ROM, or a hard disk drive in the RAM, thereby performing various processes based on these programs and creating a database described later. It is designed to execute processing.

  The computer 3 is generated in response to various requests from the user by operating the external operation unit 14 connected via the external interface 13 such as a memory card slot, a cable terminal, or a wireless communication receiving unit. The request signal is supplied to the system controller 11 via the external interface 13. Further, the system controller 11 generates screen data based on various programs and supplies the screen data to the display unit 4 connected via the external interface 13 to display various screens on the display unit 4. Yes.

  For example, when an operation signal representing a database creation program activation request is supplied from the external interface 13, the system controller 11 executes database creation processing.

  Specifically, the system controller 11 of the computer 3 displays a start screen (not shown) on the display unit 4 in response to a user operation on the operation unit. On the start screen, a face image data acquisition button and an appearance information input button are displayed. When the system controller 11 recognizes that the face image data acquisition button has been operated, it displays a moving image input screen (not shown). In the moving image input screen, an input field for history information of input moving image data and a start button for starting input of moving image data are provided.

  In the history information input field, it is assumed that the shooting location / year / month / month when the moving image data was shot is input. The storage unit 11 stores in advance a history information database that associates shooting locations, regions, and occupations. For example, when the shooting location is input as the employee cafeteria of Toranomon branch of XX bank, the system controller 11 reads the history information database, and the area “Tokyo” where the workplace is located, the occupation “bank”, the date “2012” “July” is recognized as history information.

  When the system controller 11 recognizes that the start button has been operated in response to a user operation, the system controller 11 acquires moving image data from an external storage unit such as a memory card or the video camera 2. When acquiring the moving image data, the system controller 11 detects a human face from the moving image data, cuts out only the face image area, and generates face image data. The detection of the human face may be performed for all frames, but it is preferable to perform the detection at regular intervals because the processing load on the system controller 11 can be reduced.

The system controller 11 selects only one face image data for the same person by extracting and comparing feature amounts from the face image data, and stores them in the storage unit 12 in association with the history information.

  As described above, the computer 3 acquires the face image data from the input moving image data, and stores the face image data in the storage unit 12 in association with the history information.

  On the other hand, when the system controller 11 recognizes that the appearance information input button has been operated on the start screen, the system controller 11 reads the face image data and history information stored in the storage unit 12 and generates face display screen data. As a result, as shown in FIG. 3, the face display screen 21 is displayed on the display unit 4. The face display screen 21 is provided with a human face image 22, history information 23, an apparent age input field 24, an apparent gender input field 25, and a decision button 26.

  When the system controller 11 recognizes that the apparent age and the apparent gender are input in the apparent age input field 24 and the apparent gender input field 25, for example, the determination button 26 is displayed in a blinking manner or a focus display, thereby displaying the determination button 26. Encourage operation. The system controller 11 associates the displayed face image data with the input apparent age and apparent gender (hereinafter referred to as input apparent information) and stores them in the storage unit 12 in accordance with the user's operation on the decision button 26. To do.

  Then, the system controller 11 reads out the next face image data from the storage unit 12, displays the face display screen 21 on which the face image data is displayed on the display unit 4, and displays input appearance information for the next face image data. Let the user input.

  When the input appearance information for the face image data is input a predetermined number of times (for example, 5 times), the system controller 11 determines the registered apparent age and the registered apparent sex as the registered apparent information. The system controller 11 calculates the average value of the input apparent ages as the registered apparent ages.

The average value is calculated, for example, as an average value obtained by dividing the total value of the input apparent ages by the number of inputs, or an average value excluding the maximum value and the minimum value. The average value may be calculated after weighting according to the input person.

The system controller 11 determines the registered apparent gender by, for example, the majority of the input apparent gender. This majority vote may also be performed after weighting according to the input person. These weightings are performed so as to be heavy for an input person living in an area using the apparent information database and light for an input person living in a different area, for example.

  Thereby, priority can be given to the input apparent age and the input apparent gender seen from the eyes of the person who lives in the area where the apparent information database is used, so that the accuracy of the registered apparent age and the registered apparent gender can be improved.

  The system controller 11 extracts a feature amount from the face image data, and registers the feature amount as a registered face feature amount in the appearance information database together with the registered appearance information and history information. At this time, the system controller 11 deletes the face image data and all information associated with the face image data from the storage unit 12. As a result, the system controller 11 cannot display the face image data again, so that it is possible to protect the privacy of the person who is the source of the face image data.

  As described above, the computer 3 is configured to create an appearance information database in which the feature amount of the person's face and the registered appearance information are associated with each other. This makes it possible to detect the apparent age and gender from the feature amount of the person's face.

  Here, a person who is actually a man, even if she is a man, purchases women's clothing. Also, a person with a young apparent age tends to prefer young clothes, and a person with a large apparent age tends to prefer older clothes. Therefore, by determining the apparent age and sex, it is possible to obtain information that is more marketingally useful than the actual age or actual sex. Therefore, the apparent information database can provide useful information for marketing.

As described above, the database creation apparatus of the present invention includes an acquisition unit that acquires image data including a person's face image, an input unit that causes a plurality of people to input an apparent age and an apparent gender for the person's face image, Apparent age determination means for determining the apparent age for facial images based on the average apparent age input from multiple people, and the apparent gender for facial images based on the apparent gender input by multiple people An apparent gender determining means, a feature quantity extracting means for extracting a facial feature quantity that is a facial image feature quantity from the image data, and a storage means for storing the apparent age and the apparent gender in association with the facial feature quantity. I did it.

  As a result, the database creation device can associate the facial feature quantity of the person's face with the integrated apparent age and gender of the person's face, so that the apparent age and gender without bias can be determined. It can be associated with facial feature quantities. For this reason, by using this database, it is possible to detect an apparent age and sex without any bias.

  The database creation program of the present invention includes an acquisition step of acquiring image data including a person's face image, an input step for causing a plurality of persons to input an apparent age and an apparent gender for the person's face image, and a plurality of persons The apparent age determination step for determining the apparent age for the face image based on the average value of the apparent age input from, and the apparent sex for determining the facial image based on the apparent gender input by multiple persons A gender determination step, a feature amount extraction step for extracting a face feature amount that is a feature amount of a face image from the image data, and a storage step for storing the apparent age and the apparent gender in association with the face feature amount. did.

  As a result, the database creation program can associate the facial feature quantity of the person's face with the integrated apparent age and gender of the person's face, so that the apparent age and gender without bias can be determined. It can be associated with facial feature quantities. For this reason, by using this database, it is possible to detect an apparent age and sex without any bias.

  The database creation apparatus and database creation program according to the present invention can associate a facial feature amount of a person's face with an integrated apparent age and gender of a plurality of persons on the face of the person. Age and gender can be associated with facial features. For this reason, by using this database, it becomes possible to detect the apparent age and sex without any bias, and it is possible to supply information useful for marketing purposes.

[1-2. Database creation process]
Next, database creation processing executed according to the database creation program will be described with reference to the flowcharts shown in FIGS.

  The computer 3 activates a database creation program in response to a user operation. When the computer 3 recognizes that the moving image data acquisition process has been selected in accordance with the user operation, the computer 3 starts the moving image data acquisition process procedure RT1 and proceeds to the next step SP1.

  In step SP1, when acquiring the history information and the moving image data, the computer 3 proceeds to the next step SP2.

  In step SP2, the computer 3 detects the face area of the person from the moving image data, and extracts the face image data, the process proceeds to the next step SP3.

  In step SP3, when the computer 3 stores the history information acquired in step SP1 and the face image data extracted in step SP2, the computer 3 moves to an end step and ends the process.

  On the other hand, when recognizing that the apparent information determination process is selected in accordance with the user operation, the computer 3 starts the apparent age information processing procedure RT2 and proceeds to the next step SP11.

  In step SP11, the computer 3 displays the history information and the face image data, and when the user inputs the apparent age and sex, the process proceeds to the next step SP12.

  In step SP12, the computer 3 determines whether or not the apparent age and sex for the face image data have been input a predetermined number of times. If a negative result is obtained here, the process proceeds to the next step SP13.

  In step SP13, when the computer 3 stores the input apparent information in association with the face image data, the computer 3 proceeds to an end step and ends the process.

On the other hand, when a positive result is obtained in step SP12, the process proceeds to the next step SP14. In step SP14, the computer 3 extracts a registered face feature amount from the face image data, and proceeds to the next step SP15.

  In step SP15, the computer 3 calculates the average value of the input apparent ages as the registered apparent ages, and determines the registered apparent sexuality by majority decision of the input apparent sexualities. When the computer 3 registers the registered appearance information in the appearance information database together with the registered face feature amount and history information, the computer 3 proceeds to the next step SP16.

  In step SP16, when the computer 3 performs a process of prohibiting the display of the face image data, for example, by deleting the face image data, the process proceeds to an end step and ends the process.

[1-3. Generation of facial expression information]
2. Description of the Related Art Conventionally, there has been proposed a number determination device that detects a person from shooting data shot by a video camera and counts the number of persons (for example, see Patent Document 1).

  However, there is a demand not only to count the number of photographed persons but also to acquire the facial expressions of the persons as information.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a facial expression information generation method and facial expression information generation program capable of generating useful information regarding a facial expression of a person.

  FIG. 6 shows an example of a facial expression information generation system 51 according to the present invention. The facial expression information generation system 51 includes a video camera 52 as a person image acquisition unit and a computer 53 as an facial expression information generation device.

  As the video camera 52, for example, a simple moving image photographing device such as a WEB camera is used in addition to a camcorder. The video camera 52 is installed at an entrance of a restaurant or the like, for example, and moving image data including a human face image is taken.

The computer 53 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access) (not shown).
A system controller 61 composed of a memory) controls the entire computer 53 in an integrated manner. The system controller 61 expands a basic program and marketing information acquisition processing stored in the storage unit 62 such as a flash memory, a ROM, or a hard disk drive in the RAM, thereby performing various processes and facial expressions to be described later based on these programs. An information generation process is executed.

  The computer 53 supplies a request signal from the external operation unit 55 to the system controller 61 via the external interface 63 when various requests are made by the user by operating the external operation unit 55 connected via the external interface 63. It is made to do. Further, the system controller 61 generates display screen data based on various programs and supplies the screen data to the display unit 54 connected via the external interface 63 so that various screens are displayed on the display unit 54. Has been made.

  For example, when an operation signal representing an activation request for the facial expression information generation program is supplied from the external interface 63, the system controller 61 executes facial expression information generation processing.

  Specifically, the system controller 61 of the computer 53 starts facial expression information generation processing in response to a user operation on the external operation unit 55, and supplies moving image data to the video camera 52 connected wirelessly or by wire. A data request signal is sent to that effect. When the moving image data is supplied in response to the data request signal, the system controller 61 detects a person from the moving image data.

  When the system controller 61 detects a shape corresponding to the upper body of a person (ie, a human figure) by edge detection, for example, the system controller 61 detects the face of the upper body head. The face detection is performed by detecting skin color, eyes, mouth, and the like, for example. If the upper body and face can be detected, the system controller 61 recognizes that the detected upper body is a person and counts it as the number of persons.

  The system controller 61 extracts a feature amount from the detected face (hereinafter referred to as a detected face feature amount). The system controller 61 reads the apparent information database stored in the storage unit 62 in advance.

The detected face feature quantity is compared with the registered face feature quantity registered in the appearance information database, and the closest registered face feature quantity is selected. The system controller 61 determines the registered apparent information associated with the selected registered facial feature quantity as the detected apparent age and detected apparent gender of the detected person (hereinafter collectively referred to as detected apparent information). Further, the registered apparent information may be corrected by an age specifying factor that determines the age such as wrinkles and skin condition, and the corrected value may be used as detected apparent information.

  Here, the system controller 61 allows the user to set a collation condition to be collated in advance, inquires history information registered in the apparent information database, and only the registered face feature amount that matches the collation condition is to be collated. It can be.

  Apparent age varies depending on the region and occupation. The system controller 61 can set only the registered face feature amount corresponding to the detected photographing location (that is, the residence or occupation) of the person to be collated. For this reason, the system controller 61 can improve the accuracy of the detected apparent information, and can reduce the number of registered face feature quantities to be collated, thereby reducing the collation processing load.

  The system controller 61 detects the facial expression of the person while confirming the identity of the person by tracking the upper body. The system controller 61 calculates the smile level based on the eyebrow angle, the corner of the eye corner, the angle of the mouth corner, the wrinkle of the corner of the eye, the degree of exposure of the teeth, the size of the mouth, etc. every few frames. The smile level is expressed by being divided into a plurality of levels such as 5 levels, 10 levels, or by a percentage.

  For example, the system controller 61 detects the angle of the eyebrows, the corner of the corner of the eye, and the angle of the mouth corner, calculates the average value after weighting each, and calculates the smile level by percentage.

At this time, for example, the system controller 61 surrounds the detected person's face with a frame, and detects the detected image data so as to display the smile level detected for each frame as a smile level display next to the frame with a numerical value and a graph. Is generated and superimposed on the moving image data to generate display screen data.

  Here, the system controller 61 may group detected apparent ages and detected apparent genders, and generate detected image data that is color-coded. For example, a group of men in their 20s, women in their 20s, men in their 30s, women in their 30s, men in their 40s, women in their 40s, men over 50, women over 50 it can. This grouping is preferably performed appropriately according to the usefulness as marketing information.

  The color classification is preferable because it is easy to recognize a group of persons by systematizing, for example, blue for men and red for women. Further, by performing a specific color arrangement only for an important group as marketing information, it is possible to display only those persons belonging to the important group.

  As a result, as shown in FIG. 8, the display unit 54 displays a frame 71 indicating that the face is detected as a person's face and a smile level display 72 indicating the smile level of the detected person. Thus, the user can know the customer satisfaction level in real time.

The system controller 61 calculates the average smile level of each person (hereinafter referred to as smile average value), and whether the person is smiling depending on whether the smile average value is equal to or greater than a predetermined threshold. Determine whether or not. Then, the system controller 61 stores the time when the human face is detected, the detected appearance information, and the presence or absence of a smile in association with each other.

  As described above, the computer 53 counts the number of persons from the captured moving image data by the facial expression information generation process, and stores the detected detected person information and the presence / absence of a smile in association with each other.

  The system controller 61 of the computer 53 starts a marketing information display process in response to a user operation on the operation unit.

  Specifically, the system controller 61 aggregates the number of persons, detected appearance information, and the presence / absence of a smile in a designated time zone on a day designated by the user, generates display screen data, and outputs via the external interface 63 It supplies to the display part 54. As a result, the total result is displayed on the display unit 54 by, for example, graph display or list display.

  As a result, the user can see how many people have passed the spot on the day specified by the user, the detected apparent age and the detected gender of the passed person, and how much of the person is smiling You can easily know if it was.

[1-4. Database creation process]
Next, facial expression information generation processing executed in accordance with the facial expression information generation program will be described using the flowchart shown in FIG.

  The computer 53 activates a facial expression information generation program in response to a user operation. When the computer 53 recognizes that the moving image data acquisition process has been selected in accordance with the user's operation, it starts the marketing information processing procedure RT3, and proceeds to the next step SP31.

  In step SP31, when the computer 53 detects a person, the process proceeds to the next step SP32.

  In step SP32, when the computer 53 adds “1” to the counter for counting the number of persons, the process proceeds to the next step SP33.

  In step SP33, the computer 53 compares the detected face feature amount of the detected person with the registered face feature amount registered in the appearance information database, and detects the detected apparent age and detected appearance from the closest registered face feature amount. If sex is determined, it will move to following step SP34.

  In step SP34, after calculating the smile average value for the detected person, the computer 53 proceeds to the next step SP35.

  In step SP35, the computer 53 determines whether or not the smile average value calculated in step SP34 is greater than or equal to a threshold value. If an affirmative result is obtained, the computer 53 proceeds to the next step SP36, determines that the detected person is a smile, and proceeds to the next step SP38.

  On the other hand, if a negative result is obtained in step SP35, the computer 53 proceeds to the next step SP37, determines that the detected person is not smiling, and proceeds to the next step SP38.

  In step SP38, when the computer 53 stores the detected apparent age and the detected apparent gender determined in step SP33 and the presence or absence of the smile determined in steps SP36 and 37, the computer 53 proceeds to the next step SP39.

  In step SP39, the computer 53 determines whether or not the end of the process is requested according to the user's operation. If a negative result is obtained, the computer 53 returns to step SP31 and continues the process.

  On the other hand, when a positive result is obtained in step SP39, the process proceeds to an end step and the marketing information processing procedure RT3 is ended.

[1-5. Operation and effect]
According to the above configuration, the computer 3 serving as the database creation device determines the apparent age and the apparent gender of the person's face image via the acquisition unit that acquires image data including the person's face image and the external operation unit 14. An external interface 13 is provided as input means for allowing a plurality of people to input.

The computer 3 also includes an apparent age determining unit that determines an apparent age for the face image based on an average value of the apparent age input from a plurality of persons, and a facial image based on the apparent gender input from the plurality of persons. And a system controller 11 as a feature quantity extracting means for extracting a face feature quantity, which is a feature quantity of a face image, as a registered face feature quantity from the image data.

Further, the computer 3 includes a storage unit 12 as a storage unit that stores the apparent age and the apparent gender as the registered apparent age and the registered apparent gender in association with the registered face feature amount.

  As a result, the computer 3 can determine the apparent age and the apparent gender of the person's face image by fair judgment by a plurality of persons, and therefore stores the apparent age and the apparent gender in association with the registered face feature amount. The apparent information database can be created.

  In addition, the system controller 11 of the computer 3 acquires history information about a person. Thereby, the computer 3 can create an apparent information database by associating the factors that influence the apparent age and the apparent gender, such as the residence and occupation, with the apparent information and the registered face feature amount. For this reason, when the user uses the apparent information database, it is possible to determine the apparent age and the apparent gender for each piece of history information, and it is possible to increase the discrimination accuracy of the apparent age and the apparent gender.

  Further, the system controller 11 deletes the face image data after inputting the apparent age and the apparent gender for the plurality of persons, that is, after the input appearance information is input by the number of times of input, so that the face image data that is the face image of the person The display of is prohibited.

  As a result, the computer 3 does not need to save the face image data after registration in the appearance information database, so that the storage capacity of the storage unit 12 can be saved and the photographed person can be saved by saving unnecessary face image data. Can be surely protected.

  The computer 3 acquires GPS information as history information. Thereby, it is not necessary for the user to perform input work, and highly reliable information can be obtained.

  The computer 3 acquires image data and history information via the Internet. Thus, the computer 3 can easily collect image data and history information from an unspecified large number of adults.

  Further, the system controller 11 apparently stores only the shooting date in the information database as history information. Thereby, since the computer 3 does not leave the detailed date and time when the person existed, the privacy of the photographed person can be reliably protected.

Further, the computer 53 as a facial expression information generation device has an external interface 63 as an acquisition means for acquiring moving image data.

  Further, the computer 53 includes a person detection unit that detects a person from the moving image data, a facial expression detection unit that detects a facial expression from the detected person's face, and the presence / absence of a smile level as a facial expression at a specific time, A system controller 61 is provided as statistical means for statistically counting the number of persons.

  Further, the computer 53 has a storage unit 62 as storage means for storing the number of persons, facial expressions, and the date and time when the facial expressions were detected in association with each other.

  Thereby, since the computer 53 can statistic the presence / absence of a smile and the number of persons at specific times according to a user's request, it is very useful information such as customer satisfaction and customer service attitude. Can be given to the user.

The system controller 61 extracts the detected face feature value from the face image data that is the detected face of the person, compares the detected face feature value with the registered face feature value registered in the apparent information database, and determines the age and gender. The detected apparent age and the detected apparent gender are discriminated. Then, the system controller 61 statistics the presence / absence of a smile and the number of persons for each detected apparent age and detected apparent gender, or for each detected apparent age and detected apparent gender, in accordance with a user request.

  Thereby, the system controller 61 can statistic the customer satisfaction according to age and sex, and can give very useful information to the user as marketing information.

  The system controller 61 calculates the smile level at predetermined intervals, and determines the presence or absence of a smile, which is the smile level of the person, based on the smile average value, which is the average value of the smile levels.

  As a result, the system controller 61 can simplify the statistics of the smile level and provide easy-to-understand information to the user.

Further, the computer 53 detects a person from the moving image data, detects expression from the detected person's face, and detection information determination that determines the age and sex of the person as detection apparent information as detection information. A system controller 61 as screen data generating means for generating screen data so that the means, the color-coded frames corresponding to the detected apparent information, and the smile level as the expression level correspond to the movement of the person in the moving image data. have.

  Thereby, since the computer 53 can display the color-coded frame 71 corresponding to the detected apparent information, the user can visually recognize the age and gender of the person at a glance. In addition, the relationship between gender and smile level can be easily recognized.

  Further, the computer 53 displays a smile level display 72 representing a smile level next to the frame 71 in a graph. Thereby, the computer 53 can make the user visually recognize the smile level, and can make the user more easily understand the relationship between the age and sex of the person and the smile level.

  The system controller 61 determines the apparent age and the apparent gender as the detection information, so that the apparent age and the apparent gender that are directly related to the customer's desire to purchase can be made the person's age and gender. Thereby, the computer 53 can further increase the usefulness of the screen data as marketing information.

  The system controller 61 generates screen data by superimposing the frame 71 and the smile level display 72 on the moving image data. Accordingly, the computer 53 can display the relationship between the age and sex of the person and the smile level together with the actual face image data on the display unit 54, thereby further enhancing the usefulness of the screen data as marketing information. Can do.

<Second Embodiment>
In the second embodiment shown in FIGS. 10 to 13, portions corresponding to those in the first embodiment shown in FIGS. 1 to 9 are denoted by the same reference numerals, and description of the same portions is omitted. In the second embodiment, the position information using GPS (Global Positioning System) as the history information, the point of detecting the moving direction of the person, and the appearance information using the face image data obtained in the expression information generation process The point of updating the database is different from the first embodiment.

[2-1. Database creation process]
In the database creation process in the second embodiment, history information is automatically acquired using position information by GPS.

  Specifically, the video camera 102 is installed in an arbitrary place such as a street corner, and moving image data including a human face image is captured. The video camera 102 is equipped with a GPS function and can automatically acquire position information of the place where the image was taken. An external GPS terminal may be connected to the video camera.

  In the moving image data acquisition process, the system controller 111 of the computer 103 displays a start screen (not shown) on the display unit 4 in response to a user operation on the operation unit. The system controller 111 identifies an area where the shooting location exists from the position information supplied from the video camera 102, and recognizes the shooting location and shooting date and time as history information. This area includes, for example, “Tokyo”, “Osaka”, “Chuo-ku, Tokyo”, “Hiroshima City”, “Chugoku Region”, “Kanto Region”, “Japan”, “Europe”, etc. It is preferable to become a predetermined group such as a global region.

  As a result, the system controller 111 can automatically recognize the history information, so that it is not necessary for the user to input the history information, and the user's hand is not bothered.

  As described above, the computer 103 uses position information acquired by GPS as history information.

  On the other hand, in the appearance information determination process, when the input appearance information is input a predetermined number of times for the face image data, the system controller 111 sets a flag indicating display prohibition for the face image data. Thereby, since the system controller 11 cannot display the face image data, it is possible to protect the privacy of the person who is the source of the face image data. Further, by setting the face image data to be displayable only when the number of times of input is changed, it is possible to cope with a case where the setting of the number of times of input is changed.

[2-2. Facial information generation processing]
In the facial expression information generation process according to the second embodiment, as shown in FIG. 10, the moving direction of a person is detected from moving image data acquired from the video camera 52.

  When the system controller 161 detects a shape corresponding to the upper body of a person, the system controller 161 detects the face of the head of the upper body. If the upper body and face can be detected, the system controller 61 recognizes that the detected upper body is a person and counts it as the number of persons.

  The system controller 161 performs tracking of the upper body of the person, and detects the direction connecting the initial detection point where the upper body of the person is first detected on the screen and the end detection point detected last as the movement direction. Specifically, as shown in FIG. 11, the system controller 161 calculates the center point (hereinafter referred to as a motion reference point) SP of the upper body of the person and detects the motion direction from the movement vector of the center point. To do.

This movement direction is recognized as, for example, four directions of front, rear, left and right depending on the angle. Note that the motion reference point SP may be any point that can be fixed in a person. For example, the center point of the face or the eyes / mouth can be used as the motion reference point. The movement directions do not necessarily have to be four directions, and may be recognized in two or more directions according to the angle.

  When the video camera 52 is arranged above the person's height, the shooting angle changes according to the distance between the video camera 52 and the person, and thus moves up and down the screen as shown in FIG. . For this reason, the system controller 161 moves the person forward when the movement reference point SP moves from the top to the bottom, and conversely moves the person backward when the person moves from the bottom to the top. Can be recognized.

  On the other hand, when the video camera 52 is installed at the same height as the person, as shown in FIG. 13, the movement reference point SP hardly changes when the person moves in the front-rear direction. Therefore, when the movement reference point SP hardly changes, the system controller determines the movement direction based on the size of the person.

  Specifically, the system controller 161 detects the interval between the eyeballs of the detected person's movement direction, and determines whether the person has moved forward or backward based on the change rate of the interval. Similarly, it is possible to determine whether the person has moved forward or backward based on the rate of change of the size of the person's upper body, face, mouth, and the like.

  The system controller 161 stores the detected direction information of the person and the presence / absence of a smile in association with the direction of movement.

  When the appearance information display process is started in response to a user request, the system controller 161 aggregates detected person appearance information and the presence / absence of a smile for each movement direction, and generates screen data.

  As a result, the user can know the apparent age and apparent gender of the person for each direction of movement and the presence / absence of a smile, for example, at a branch point of a shopping center. I can know.

[2-3. Database update process]
In the facial expression information generation process, the system controller 161 stores the acquired moving image data for a predetermined period (for example, one week).

  Then, the system controller 161 reads the moving image data stored in the storage unit 62 in accordance with a user operation on the operation unit, and executes database creation processing.

  As a result, the system controller 161 can sequentially update the apparent information database and can determine the detected apparent age and gender based on more registered face feature quantities, thus improving the accuracy of the detected apparent age and gender determination. can do.

  In general, the apparent age depends on the image of the age of the person to be judged, and thus varies greatly depending on the characteristics of the region and place. For example, in Ginza, Tokyo, there are many young people who care about fashion and makeup, and their apparent age is relatively large. On the other hand, the apparent age is smaller in rural areas in Tohoku.

In the database update process, the system controller 161 uses the face image data of the customer who actually visited the shopping center to determine the apparent age from the viewpoint of an employee living near the shopping center, and the input apparent Can be age. For this reason, the system controller 161 can create an apparent information database that matches the characteristics of the region, and can improve the accuracy of discrimination of the detected apparent age and sex.

Note that the system controller 161 can store only face image data obtained by cutting out a face detected from a person, instead of the moving image data acquired in the facial expression information generation process. Only one face image data is selected for the same person from the viewpoint of data capacity. The system controller 161 can also detect the face angle (front, sideways, etc.) and store a plurality of face image data with different angles for the same person.

  As a result, the system controller 161 can significantly save the storage capacity compared to the case of storing moving image data.

[2-4. Operation and effect]
According to the above configuration, the system controller 161 of the computer 153 acquires GPS information as history information via the external interface 63.

Thereby, the computer 153 can automatically register the history information in the apparent information database without allowing the user to input the history information.

  In addition, the system controller 161 detects the direction in which the person moves, and statistics the presence / absence of a smile and the number of persons for each moving direction and for a specific time.

  As a result, the computer 153 can statistically calculate the presence / absence of smiles and the number of persons for each person heading in a specific direction, such as a person heading to the exit direction or a person heading to a specific store on the food court. Can be further enhanced.

  Further, the system controller 161 executes a database generation process on the face image data acquired by the facial expression information generation process, and updates the apparent information database.

  Accordingly, the computer 153 can update the apparent information database based on the input apparent information input from the viewpoint of the user who uses the marketing information, based on the face of a person living in an area where the apparent information database is used. it can. For this reason, the computer 153 can reflect the feature of the apparent information in each region in the apparent information database, and can improve the accuracy of the apparent information discrimination.

<Third Embodiment>
In the third embodiment shown in FIGS. 14 to 15, the same reference numerals are given to the portions corresponding to those in the first embodiment shown in FIGS. 1 to 9, and description of the same parts is omitted. In the third embodiment, image data that is the basis of face image data is acquired via the Internet in database creation processing, and person recognition is performed by installing a plurality of video cameras 52 in facial expression information generation processing. However, this is different from the first embodiment.

[3-1. Database creation process]
As shown in FIG. 14, in the database creation system 201, a computer 203 is connected to the Internet 220. In the database creation system 201, for example, by opening a questionnaire response site, a homepage recruitment site, or distributing an application, moving image data or still image data that is the source of face image data from the personal information terminal 221, and history information To get.

  The personal information terminal 221 is not limited as long as it is a terminal that can be connected to a network such as the Internet, such as a personal computer, a smartphone, a mobile phone, and a tablet terminal. The personal information terminal 221 supplies moving image data or still image data photographed using a photographing device such as a video camera or a photographing function provided in advance to the personal information terminal 221 together with history information to a questionnaire response site.

  When the computer 203 acquires moving image data or still image data and history information from a questionnaire response site via the Internet 220, the computer 203 generates face image data and stores it in the storage unit 62.

  As described above, the computer 203 acquires the image data that is the source of the face image data via the Internet. Thereby, the computer 203 can acquire face image data of people living in various places in the world without bothering to install the video camera 52.

[3-2. Facial information generation processing]
As shown in FIG. 15, in the facial expression information generation system 251, a plurality of (for example, four) video cameras 52 (52 </ b> A to 52 </ b> D) and a computer 253 are connected via a network 270.

  The video camera 52 is installed, for example, at the exit of each attraction in the amusement park, and photographs the customer who came out after finishing the attraction.

  When the system controller 261 of the computer 253 detects a person from the moving image data supplied from the four video cameras 52, the system controller 261 extracts a detected face feature and determines detected appearance information for the person to be processed. Then, the presence or absence of a smile is determined. When the system controller 261 ends these processes, the system controller 261 determines whether or not the person to be processed has already been shot by another video camera 52.

  The system controller 261 collates the extracted detected face feature quantity with the detected face feature quantity stored in the storage unit 62 during the day (hereinafter referred to as the same day detected face feature quantity). At this time, the system controller 261 determines that the person who is the basis of the detected face feature amount is the person who is the basis of the detected face feature amount if the matching rate between the extracted detected face feature amount and the detected face feature amount on the day is high Is recognized as the same.

The system controller 261 reads the identification ID assigned to the face feature amount detected on the same day that is recognized as the same person, apparently identifies the identification ID and age, sex, whether or not there is a smile, and the storage unit 62.

  On the other hand, the system controller 261 recognizes that the person who is the source of the extracted detected face feature amount is detected for the first time on the day when there is no detected face feature amount on the same day recognized as the same person. Then, the system controller 261 adds a new identification ID to the extracted detected face feature quantity, and stores the extracted detected face feature quantity in the storage unit 12 together with the identification ID as a detected face feature quantity on the day. Furthermore, the system controller 261 stores the identification ID in the storage unit 12 in association with the apparent age, sex, and determination of whether or not there is a smile.

  The system controller 261 of the computer 253 starts marketing information display processing in response to a user operation on the external operation unit 55.

  Specifically, the system controller 261 aggregates the number of persons, the detected apparent age and detected apparent gender of the person, and the presence or absence of a smile in the specified time zone on the day specified by the user, and generates display screen data. , And supplied to the display unit 54 via the external interface 63. As a result, the total result is displayed on the display unit 54 by, for example, graph display or list display.

  Furthermore, the system controller 261 totals the results for each identification ID in response to a user request. For example, the system controller 261 adds up the time of attraction from the shooting time for a person who is all smiling in the moving image data shot by the four video cameras 52, and generates display screen data. Of course, the same calculation is possible for a person with a small smile.

  Thereby, the user can know the marketing information which shows the relationship between the participating attraction and the turn order, and the presence or absence of a smile, and can acquire more marketing information.

[3-2. Facial information generation processing procedure]
Next, facial expression information generation processing executed in accordance with the facial expression information generation program will be described using the flowchart shown in FIG.

  The computer 253 activates a facial expression information generation program in response to a user operation. When the computer 253 recognizes that the moving image data acquisition process has been selected in accordance with the user operation, the computer 253 starts the marketing information processing procedure RT4, and proceeds to the next step SP31.

  In steps SP31 to 38, when the computer 253 performs the same processing as the marketing information processing procedure RT3, the computer 253 proceeds to the next step SP51.

  In step SP51, the computer 253 collates the detected face feature amount extracted in step SP34 with the detected face feature amount on the day, and based on the matching rate with the detected face feature amount, becomes the source of the extracted detected face feature amount. It is determined whether or not a person has already been detected on the day.

  If a positive result is obtained here, this means that the person who is the source of the extracted detected face feature quantity already has an identification ID. At this time, the computer 253 Control goes to the next step SP52.

  In step SP52, the computer 253 reads out the identification ID associated with the detected face feature amount of the person determined to match, and stores it in the storage unit 62 in association with the presence / absence of the smile stored in step SP38 and the apparent age and sex. Then, the process proceeds to the next step SP54.

  On the other hand, if a negative result is obtained in step SP51, this means that the person who is the source of the extracted detected face feature quantity does not have an identification ID. The computer 253 proceeds to the next step SP53.

  In step SP53, the computer 253 stores the extracted detected face feature quantity as the detected face feature quantity on the day, adds a new identification ID to the detected face feature quantity on the day, and associates it with the detected face feature quantity on the day. If it memorize | stores in the memory | storage part 62, it will move to following step SP54.

  In step SP54, the computer 253 determines whether or not the user has requested to end the process. If a negative result is obtained, the computer 253 returns to step SP31 and continues the facial expression information generation processing procedure RT4.

  On the other hand, if a negative result is obtained in step SP54, the computer 253 moves to an end step and ends the facial expression information generation processing procedure RT4.

[3-4. Operation and effect]
In the above configuration, the external interface 63 of the computer 253 acquires image data and history information via the Internet.

  As a result, the computer 253 can acquire face image data for which the user's intention to use has been confirmed without performing the shooting using the video camera 52. In addition, since no photographing is performed, face image data can be acquired very easily from various races, ethnicities, and regions around the world.

  The system controller 261 determines the identity of a detected person among a plurality of moving image data shot by a plurality of video cameras 52.

  As a result, the computer 253 can grasp at what timing the same person was photographed in a plurality of moving image data, and thus can further increase the usefulness as marketing information.

<Fourth embodiment>
In the fourth embodiment shown in FIG. 17, portions corresponding to those in the first embodiment shown in FIGS. 1 to 9 are denoted by the same reference numerals, and description of the same portions is omitted. The fourth embodiment is different from the first embodiment in that marketing information is acquired only for a specific customer in the facial expression information generation process.

[4-1. Facial information generation processing]
In the fourth embodiment, the video camera 52 is installed in an inn or hotel. For example, the video camera 52 is installed at the exit of an inn. In addition, a video camera 69 for identifying a customer to be checked out is installed at a hotel checkout counter.

  The computer 353 is connected to an information processing terminal 68 (not shown) that performs a checkout process, and video cameras 52 and 69. The system controller 361 of the computer 353 generates marketing information only for the customer who has performed the checkout process.

Specifically, the system controller 361 of the computer 353 performs face detection processing on the moving image data supplied from the video camera 69 when a communication signal for performing the checkout processing is supplied from the information processing terminal 68. The specific face image data is acquired.

  The system controller 361 extracts a feature amount from the specific face image data (hereinafter referred to as a specific face feature amount) and stores it in the storage unit 62.

The system controller 361 performs person detection and face detection processing on the moving image data supplied from the video camera 52 installed at the exit, and collates the detected face feature quantity with the specific face feature quantity. .

When the system controller 361 recognizes that the person who is the source of the detected face feature value is the customer who performed the checkout process because the matching rate between the detected face feature value and the specific face feature value is equal to or greater than the threshold value, Generate marketing information.

The system controller 361 determines the apparent age and gender, determines the presence or absence of a smile, and stores them in the storage unit 62.

  As described above, the system controller 361 of the computer 353 can generate only the marketing information that is really necessary by generating the marketing information only for a specific person.

[4-2. Operation and effect]
In the above configuration, the system controller 361 of the computer 353 determines whether or not the detected person is a specific person, and only when the detected person is the specific person, The presence or absence of a smile is detected from face image data that is a face.

  As a result, the system controller 361 only needs to detect the presence / absence of a smile for a specific person required as marketing information, and thus can reduce the processing load while eliminating unnecessary information.

<Fifth embodiment>
[5-1. Brightness adjustment processing]
In the fifth embodiment shown in FIG. 20, portions corresponding to those in the third embodiment shown in FIG. 15 are denoted by the same reference numerals, and description of the same portions is omitted. The fifth embodiment is different from the fourth embodiment in that, at the time of installation, the luminance adjustment processing is performed for each video camera 52A to 52D to suppress the difference in age determination due to the luminance change as much as possible. .

The system controller 261 of the computer 253 shifts to the camera installation mode in response to a user operation such as clicking a camera installation button on the menu screen, for example. In the camera installation mode, when the video camera 52A, for example, is selected as the video camera to be subjected to the brightness adjustment process, the system controller 261 displays an image being shot by the video camera 52A.

At this time, a specific shooting reference target is shot in front of the video camera 52A. In addition to images such as a board with a specific color such as white, a board with various colors drawn in a grid, a board with a plurality of faces printed on it, the shooting reference target is a face type or egg. An object having a three-dimensional shape, such as a sphere having a shape or unevenness, and a combination of an image and an object are assumed.

When the system controller 261 acquires an image of the shooting reference target in response to a user operation, the system controller 261 corrects the size of the shot shooting reference target based on, for example, a shooting size correction mark provided in advance on the shooting reference target. To do.

This specific range for detecting the luminance in the specific range of the photographing reference target as the specific range luminance may be all or a part of the photographing reference target, and may be singular or plural. It is desirable to provide a plurality for this purpose.

Next, in the system controller 261, the brightness correction value is stored in advance in the storage unit 262 of the computer 253. The system controller 261 specifies the luminance correction value according to the luminance in the specific range. Further, the system controller 261 may calculate the brightness correction value by calculation based on, for example, a difference from the reference value of the specific range brightness.

When the system controller 261 is requested to execute the brightness adjustment process for the next camera 52B according to the user's operation, the system controller 261 executes the same process for the camera 52B. When the system controller 261 is requested to end the camera installation mode in response to a user operation, the system controller 261 ends the camera installation mode.

Thereafter, in the facial expression information generation process, the system controller 261 corrects the detected apparent age value by, for example, multiplying or adding / subtracting the brightness correction value with respect to the detected detected apparent age to obtain the final detected apparent age. Determine. The brightness correction value may be a constant value, or may be a variable that changes according to the age value.

As described above, the computer 253 can correct the detected apparent age value according to the shooting situation by correcting the age value based on the luminance value of the shooting reference object that has been shot. It is possible to suppress as much as possible that the value of the detected apparent age is changed depending on the degree of hitting.

[5-2. Brightness adjustment processing procedure]
Next, luminance adjustment processing executed according to the luminance adjustment program will be described with reference to the flowchart shown in FIG.

When the computer 253 shifts to the camera installation mode according to the user's operation, the computer 253 starts the brightness adjustment processing, and proceeds to the next step SP101.

In step SP101, when the video camera 52 to be subjected to brightness adjustment processing is selected by the user, the computer 253 proceeds to the next step SP102.

In step SP102, when the computer 253 displays the photographed image of the selected video camera 52, the computer 253 proceeds to the next step SP102.

In step SP103, the computer 253 determines whether or not an image as a photographing reference target has been acquired. If it has been acquired, the computer 253 proceeds to the next step SP104, and otherwise waits until it is acquired.

In step SP104, when the computer 253 adjusts the size of the photographing reference target in the image data, the computer 253 proceeds to the next step SP104.

In step SP105, when the computer 253 detects the specific range luminance, the computer 253 proceeds to the next step SP106.

  In step SP106, when the computer 253 determines the luminance correction value based on the specific range luminance, the computer 253 proceeds to the next step SP107.

  In step SP107, the computer 253 determines whether or not an operation request for further executing the brightness adjustment processing has been made in accordance with the user's operation. If the operation request is affirmed, the computer 253 returns to step SP101 and proceeds to the next video camera. The brightness adjustment process for 52 is executed.

  On the other hand, if a negative result is obtained in step SP107, the computer 253 proceeds to an end step, ends the brightness adjustment process, and cancels the camera installation mode.

  As described above, the computer 253 determines the brightness correction value based on the specific range brightness in the specific range of the predetermined image data of the imaging reference target, and corrects the age value by using the brightness correction value. The apparent age was determined based on the specific range luminance.

  Thereby, the computer 253 can suppress the apparent age from fluctuating as much as possible depending on the location where the video camera 52 is installed, the performance of the video camera 52 itself, and the like, and can improve the detection accuracy of the apparent age.

<Sixth Embodiment>
In the sixth embodiment, parts corresponding to those in the third embodiment are denoted by the same reference numerals, and description of the same parts is omitted. The sixth embodiment is different from the third embodiment in that an environmental element according to the environment in which the video camera 52 is installed, such as the current date and weather, is determined and the age value is corrected.

  In the facial expression information generation process, the system controller 261 of the computer 253 obtains current weather information via a network such as the Internet, and determines the sunshine intensity and seasonal sunshine intensity from the current date and time. Then, the system controller 261 calculates an environmental correction value corresponding to the current environmental element from the weather information, the date and time sunshine intensity, and the weather information.

  For example, in the storage unit 262 of the computer 253, environmental correction values corresponding to three environmental elements of date, time, and weather are stored in advance. In the facial expression information generation process, the system controller 261 determines an environment correction value based on the date, time, and weather, corrects the detected apparent age value, and determines the final detected apparent age. The environmental correction value is preferably changed every predetermined time, for example, every 30 minutes or every hour. Further, the environmental correction value may be calculated by calculation according to an environmental element.

As described above, in the facial expression information generation system according to the present embodiment, by correcting the age value according to the environmental element and determining the detected apparent age, regardless of the situation where each of the video cameras 52A to 52D is installed, An almost constant apparent age can be detected, and the apparent age detection accuracy can be improved.

  The computer 253 determines an apparent correction age based on the environmental correction value by determining an environmental correction value from an environmental element corresponding to the current environment at the shooting location and correcting the age value using the environmental correction value.

  Thereby, the computer 253 can minimize the influence of the environment at the shooting location where the video camera 52 is installed, and can improve the apparent age discrimination accuracy. In particular, when the video camera 52 is installed outdoors, the influence of the environment can be minimized, which is very effective.

<Seventh embodiment>
In the seventh embodiment, portions corresponding to those in the first embodiment are denoted by the same reference numerals, and description of the same portions is omitted. The seventh embodiment is different from the first embodiment in that an audience rating survey is performed using facial expression information generation processing.

  As shown in FIG. 21, in the audience rating survey system 751, a computer 756 is connected to the Internet 720. A terminal 752 with a camera as a terminal device has a camera as a photographing unit, a display unit as a display unit, and a communication unit as a reception and transmission unit, and can receive video data and view it via the display unit. Has been made. As video data, for example, in addition to receiving radio waves such as terrestrial television broadcasting and satellite broadcasting, on-demand streaming broadcasting, Internet television broadcasting, DVD (Digital Versatile Disc), USB (universal serial bus) and other fixed media Are stored in a built-in device such as a built-in HDD (Hard Disc Drive).

The camera-equipped terminal 752 is not limited as long as it can be connected to a network such as the Internet, such as a personal computer, a smartphone, a mobile phone, a tablet terminal, or a television receiver.

The audience rating survey program is stored in advance in a storage unit (not shown) of the camera-equipped terminal 752. When receiving the video data, the camera-equipped terminal 752 displays a broadcast screen based on the video data on the display unit, and performs shooting of the viewer using the attached camera. It is desirable for the viewer to download the audience rating survey program after obtaining permission for the audience rating survey system in advance.

Then, the camera-equipped terminal 752 executes facial expression information generation processing on the captured data. The facial expression information generation process is preferably executed at a predetermined interval such as every second.

The camera-equipped terminal 752 associates the detected apparent age, gender, and smile level with time information indicating which part of the video data, and transmits it to the computer 756 via the Internet 720.

Note that the camera-equipped terminal 752 may link the time information of the video data and transmit the shooting data to the computer 756, and the computer 756 may execute facial expression information generation processing.

As a result, the detected apparent age, gender, and smile level of the viewer are stored in the storage unit of the computer 756 in a state linked to the time information of the video data. The system controller (not shown) of the computer 756 detects the viewer's detected apparent age, sex, and smile level for each time information, thereby detecting the audience rating for each time information and each video data. Calculate for each apparent gender.

As illustrated in FIG. 22, the computer 756 displays a display screen 780 on the display unit 54 in accordance with a user operation. As shown in FIGS. 22 and 23, the display screen 780 includes a video display frame 781 for displaying video, a time display frame 782 indicating the playback position of video data, and a total total audience rating graph 783a of all video data. And a total display frame 783 representing the total smile degree graph 783b, a video reputation frame 784 representing the audience rating and smile level corresponding to the currently displayed video data, and the audience rating and smile level corresponding to the currently displayed video data. And a detailed display frame 785 for displaying details for each gender and age.

Items displayed in the detailed display frame 785 can be changed by setting. For example, it is possible to display in increments of 5 years or to display items for both men and women.

On this display screen 780, by moving the cursor 786 on the time bar 782a displayed in the time display frame 781, the playback position is moved, video data corresponding to the playback position is displayed, and the video data is also displayed. A corresponding total display frame 783, a video reputation frame 784, and a detailed display frame 785 are displayed. In addition, the reproduction position line 783c indicating the reproduction position in the total display frame 783 is also moved.

As a result, it is possible to easily specify the playback position and quickly recognize the audience rating and smile level at the specified playback position.

For example, by placing the cursor 786 in the total display frame 783 and double-clicking, the positions and sizes of the total display frame 783 and the video display frame 781 on the display screen 780 can be switched as shown in FIG. Further, in the detailed display frame 785, details of the audience rating and the smile level for each gender and age are displayed in a graph.

In the display screen 780 of FIG. 23, by moving the cursor 786 on the playback position line 783c, the playback position is moved, video data corresponding to the playback position is displayed, and a total display frame 783 corresponding to the video data is displayed. A video reputation frame 784 and a detailed display frame 785 are displayed.

Thereby, for example, the playback position with the highest audience rating can be easily specified, and the video data at the designated playback position, and the detailed situation of the audience rating and smile level can be recognized quickly.

As described above, in the audience rating survey system 751 according to the present embodiment, the camera-equipped terminal 752 displays the video data and photographs a person who views the video data. Furthermore, the camera-equipped terminal 752 detects a person from the captured moving image data, detects a facial expression from the detected face of the person, and associates the number of persons, the degree of smile, and time information of video data. To the computer 756 for transmission. The computer 756 statistics the facial expression and the number of persons in association with the video data.

Accordingly, the computer 756 can calculate the audience rating and the smile level for each time information of the video data, and can easily associate each screen of the video data with the audience rate and the smile level. As a result, the computer 756 can statistically evaluate the audience rating corresponding to the video data, and can easily recognize the viewer's reaction to each scene of the video data.

<Other embodiments>
In the database creation process of the first to seventh embodiments described above, the case where the input apparent information is input via the external operation unit 14 connected to the external interface 13 as the input means has been described. The present invention is not limited to this. For example, the input apparent information may be input from an internal input unit such as a keyboard or a touch panel of a notebook computer.

  Furthermore, in the database creation processing of the first to seventh embodiments described above, the case where history information is registered in the apparent information database has been described. The present invention is not limited to this, and history information is not necessarily essential. In addition, race information, ethnicity, and the like can be included as history information. History information such as race and ethnicity may be input from the appearance, or may be automatically determined from the color and characteristics of the face.

  Furthermore, in the database creation processing of the first to seventh embodiments described above, the case where the apparent age and the apparent sex are input as the apparent information has been described. The present invention is not limited to this, and race or ethnicity may be input as apparent information.

  Furthermore, in the database creation process of the first to seventh embodiments described above, the case where the display of the face image data is prohibited immediately after the number of times of input has been described has been described. The present invention is not limited to this, and face image data may be deleted after a predetermined period (for example, one month) has elapsed, or display of face image data may be prohibited by setting a flag.

  Furthermore, in the database creation process of the third embodiment described above, a case has been described in which image data is acquired via the Internet through a questionnaire recruitment site or the like. The present invention is not limited to this, and image data may be acquired via the Internet, for example, by a computer automatically acquiring face image data published on a homepage or the like.

Furthermore, in the database creation process of the first to seventh embodiments described above, the case where the face detection process is executed after the human figure is detected to track the upper body has been described. The present invention is not limited to this. For example, the upper body or clothes are tracked after the face is detected and the person is counted, or the face is tracked as it is after the face is detected and the person is counted. Also good.

  Further, the case where one face image data is generated from moving image data in the database creation processing of the first to seventh embodiments described above has been described. The present invention is not limited to this, and two or more face image data may be generated. For example, the system controller 11 stores a plurality of face image data having greatly different feature amounts in the storage unit 12 while recognizing that they are the same person by detecting and tracking a human figure. As a result, a plurality of face image data useful for collation, such as a side face or a face when facing down, can be obtained from one person.

  Furthermore, the case where the computer acquires the image data acquired from the video camera 52 in the expression information generation processing of the first to seventh embodiments described above has been described. The present invention is not limited to this, and image data supplied from other information communication terminals such as smartphones, tablet terminals, and other computers may be acquired.

  Specifically, the computer 553 of the facial expression information generation system is connected to another information terminal by a cloud system via a network such as the Internet. The computer 553 acquires moving image data from another information terminal, executes display information generation processing, and supplies screen data to the other information terminal. Other information terminals only perform image data supply and screen data display processing. As a result, the facial expression information generation system can remarkably reduce the processing load on the computer 553, so that, for example, a personal computer or a small processing unit can be used as the computer 553.

  In this case, the computer 553 executes database update processing. As a result, the computer 553 can acquire a huge number of face image data from moving image data supplied from other information terminals. In this case, when the computer 553 acquires new face image data, the computer 553 extracts the face feature amount, confirms the matching rate with the registered face feature amount, and for face image data with a high matching rate with the registered face feature amount. The display of face image data is omitted. Accordingly, the computer 553 can prevent the collation processing load from becoming excessive while maintaining the diversity of the face image data.

  Further, in the facial expression information generation processing of the first to seventh embodiments described above, the computer selects a registered face feature amount having the highest coincidence with the detected face feature amount, and is associated with the registered face feature amount. The case where the apparent age and gender are the apparent age and gender of the person who is the basis of the detected face feature amount has been described. The present invention is not limited to this. For example, based on registered face feature values, model feature values are created for each age and gender, and the apparent age and sex associated with the model feature value having the highest matching rate are detected. The apparent age and gender of the person who is the basis of the facial feature quantity may be used. Only one model feature quantity may be created for each apparent age and sex, or two or more model feature quantities may be created.

  Thereby, in the facial expression information generation system, the number of collation targets can be significantly reduced. For this reason, in the facial expression information generation system, the processing load on the computer can be significantly reduced. For example, it is possible to use a general personal computer for personal use or a small processing unit as the computer.

  Furthermore, in the facial expression information generation processing of the first to seventh embodiments described above, the detected apparent information and the presence / absence of a smile are statistically calculated for each specific time (for example, 1 hour, 1 day, day of the week, etc.). Said about the case. The present invention is not limited to this, and the actual age, the actual sex, and the detected facial expression (presence / absence of a smile) may be statistically calculated using an actual age database and an actual sex determination method.

  Furthermore, in the facial expression information generation process of the fourth embodiment described above, the case where the appearance information and facial expression are detected only for a specific person has been described. The present invention is not limited to this. For example, appearance information and facial expression may be detected for all persons. In this case, it is preferable that the statistics are divided into a specific person and an unspecified person according to a user request by setting a flag only for the specific person. Thereby, since the expression of a specific person with respect to the whole becomes clear, the usefulness of marketing information can be enhanced.

  Further, the statistical method may be changed between a specific person and an unspecified person. For example, the computer 653 can acquire the smile level of the store clerk and the smile level of the customer at the same time by identifying only the store clerk and detecting a smile in an environment where the store clerk and the customer are confused, such as a store. is there. When acquiring the smile level of the store clerk, the computer 653 stats the relationship between the smile level and the time by, for example, dividing the smile level into a plurality of levels and statistically analyzing the relationship between the smile level and time. On the other hand, the computer 653 calculates the average value of smile levels and the presence or absence of smiles for customers.

Furthermore, in the facial expression information generation processing of the first to seventh embodiments described above, the case where the presence / absence of a smile is determined from the average value of smile levels has been described. The present invention is not limited to this. For example, a smile value such as large / medium / small smiles, 0-20%, 20-40%, 40-60%, 60-80%, 80-100%. Depending on, it may be classified into three or more stages. It is also possible to use the smile level as it is. As a result, for example, a natural smile is required in the hospitality industry, and a smile that is required according to the purpose can be flexibly dealt with so that a full smile is required for the customer. .

  Furthermore, in the above-described portable facial expression information generation processes of the first to seventh embodiments, the case where a person is recognized when a human figure and a face are detected has been described. The present invention is not limited to this. For example, the computer may count the number of people only by detecting the human type, or may count the number of people according to the detection of the face or the detection of the facial expression.

  Further, in the above-described portable facial expression information generation processing of the first to seventh embodiments, the case where a human figure is detected has been described. The present invention is not limited to this, and the computer does not necessarily need to detect a human figure. For example, the number of persons may be counted by detecting the face, and the facial expression may be detected as it is.

  Furthermore, in the facial expression information generation processing of the first to seventh embodiments described above, the case where the presence or absence of a person's smile is determined has been described. The present invention is not limited to this. For example, the presence or absence of the facial expression of other faces such as an angry face and a sad face may be determined. In this case, the presence / absence of the facial expression and its degree can be determined by detecting the movement of the hand, the opening ratio of the eyes, the wrinkles between the eyebrows, etc. in addition to the angles of the corners of the eyes, the eyebrows, and the mouth corners.

  In this case, for example, as shown in FIG. 19, the computer 753 displays a frame 471 that moves in accordance with the movement of the person in a color-coded manner according to the detected apparent information, and also displays a facial expression that expresses an expression inside the frame 471. 472 is displayed. The color of the expression display 472 may be the same as that of the frame 471. However, the expression display 472 can be easily recognized by color-coding each expression.

The expression display 472 may be displayed outside the frame 471, and the degree of expression can be displayed in a graph. It is also possible to display facial expressions in colors, for example, anger = red, sadness = blue, and laughter = yellow. Further, the inside of the frame 471 may be filled.

In this case, only the frame 471 corresponding to the movement of the computer 753 person and the degree of expression are displayed, and the moving image data does not necessarily have to be superimposed. As a result, the computer 753 can display information on the movement and facial expression of the person that is really necessary, and can avoid unnecessary display of the photographed face image, thereby protecting the privacy of the photographed person. It becomes possible.

  Further, the computer 753 does not necessarily display such a screen in real time, and can display moving image data stored in the storage unit 62 in response to a request from the user. In this case, only the moving point of the person (for example, the center point of the face) is stored in the storage unit 62 instead of the moving image data, so that the data capacity can be greatly reduced as compared with the case of storing the moving image data. Is possible.

  Furthermore, in the facial expression information generation processing of the first to seventh embodiments described above, the case where marketing information is generated from moving image data acquired from the video camera 52 has been described. The present invention is not limited to this. For example, a recording function such as an IC recorder may be used at the same time.

  Thus, for example, when a conversation between a salesperson and a customer is filmed / recorded, the computer designates and plays back when the customer's facial expression changes or when the smile level is high. It can be traced very easily later whether these situations have occurred in the part.

  Furthermore, in the first to seventh embodiments described above, a case is described in which moving image data is captured by a video camera 52 installed in a restaurant, shopping center, amusement park, or hotel in the facial expression information generation process. It was. The present invention is not limited to this. For example, the moving image data may be captured by the video camera 52 installed in various places such as an inn, a bank, and a home sales office.

  Furthermore, in the fifth embodiment described above, a case has been described in which the image of the photographing reference target is acquired in accordance with the user operation in the brightness adjustment processing. The present invention is not limited to this, and the system controller 261 automatically detects the photographing reference object and automatically photographs the photographing reference object by attaching a mark indicating that the photographing reference object is the photographing reference object. Is possible. In this case, it is not always necessary to display an image being picked up by the selected video camera 52.

Further, in the above-described fifth embodiment, the case where one video camera 52 is selected in the brightness adjustment processing has been described. The present invention is not limited to this. For example, when only one video camera 52 is installed, selection of the video camera 52 is unnecessary. Further, it is not always necessary to adjust the size of the photographing reference object in the image data.

  Further, in the fifth and sixth embodiments described above, the case where the detected apparent age value is directly corrected in the facial expression information generation processing has been described. The present invention is not limited to this. For example, the detected apparent age value is indirectly set by performing image processing such as adjusting the contrast and sharpness on the detected face image and the detected face feature amount. It may be corrected.

Further, in the above-described sixth embodiment, the case has been described in which the environmental correction value corresponding to the three environmental elements of date, time, and weather is determined in the facial expression information generation process. The present invention is not limited to this. For example, the environmental correction value may be determined by combining only one element or two elements. Further, data obtained by measuring the intensity of sunlight with an illuminometer or the like may be acquired as an environmental element, or directly measured, and an environmental correction value may be determined based on this.

Furthermore, in the seventh embodiment described above, the case where the terminal with camera 752 executes the facial expression information generation process in the audience rating survey system 751 has been described. The present invention is not limited to this, and the camera-equipped terminal 752 may transmit moving image data to the computer 756, and the computer 756 may execute facial expression information generation processing. Thereby, the processing load of the camera-equipped terminal 752 can be reduced.

Furthermore, in the seventh embodiment described above, the case where the audience rating survey system 751 displays the audience rating and smile level corresponding to the video data has been described. The present invention is not limited to this, and a value obtained as a result of statistics of at least the number of detected persons and the expression level, such as the number of viewers and the expression level, may be displayed.

Further, in the audience rating survey system 751 in the seventh embodiment described above, the case where the camera-equipped terminal 752 executes the facial expression information generation process has been described. The present invention is not limited to this, and the moving image data photographed by the camera-equipped terminal 752 and the time information of the video data are supplied in association with each other, and the computer 756 executes the facial expression information generation process. good.

  That is, in this audience rating survey system, a terminal device having display means for displaying video data, photographing means for photographing a person who views the video data, and transmission means for transmitting moving image data photographed by the photographing means. An acquisition means for acquiring moving image data, a person detection means for detecting a person from the moving image data, an expression detection means for detecting an expression from the face of the detected person, the number of persons, an expression, and an expression A facial expression information generating device having storage means for associating and storing the date and time when the facial expression is detected, and statistical means for statistically analyzing the number of facial expressions and the number of persons for each specific time.

  Thereby, the audience rating survey system can confirm the facial expression of a person in real time as well as statistically associating the facial expression and the number of persons with the video data.

Furthermore, the audience rating survey system 751 in the seventh embodiment described above can use a so-called signage terminal that displays advertisement information on the screen, for example, on a street as the camera-equipped terminal 752 that displays video data. In this case, as facial expression information generation processing, it is preferable that the identity of a person is specified by tracking processing or addition of a personal ID, and the residence time for each person is detected.

Furthermore, it is possible to appropriately combine the elements of the first to seventh embodiments described above.

Further, in the above-described embodiment, the case where the database creation program, the facial expression information generation program, and the like are stored in advance in a ROM or a hard disk drive has been described. The present invention is not limited to this, and it may be installed from an external storage medium such as a memory card into a flash memory or a hard disk drive in the computer. In addition, the database creation program and the like can be downloaded from USB (Universal Serial Bus) or Ethernet (registered trademark)
of Electrical and Electronics Engineers) 802.11a / b / g or other wireless LAN (Local Area Network) acquired from the outside, and further distributed by terrestrial digital television broadcast or BS digital television broadcast May be.

  Furthermore, in the database creation process of the first embodiment described above, the external interface 13 as an image acquisition means and an input means, the apparent age determination means, the apparent sex determination means, and the system controller 11 as a feature quantity extraction means, The case where the computer 3 as the database creation apparatus of the present invention is configured by the storage unit 12 as the storage means has been described. The present invention is not limited to this, and the database creation apparatus of the present invention may be configured by image acquisition means, input means, apparent age determination means, apparent sex determination means, feature amount extraction means, and storage means having various configurations. good.

  Furthermore, in the facial expression information generation process of the first embodiment described above, the external interface 63 as an acquisition unit, the person detection unit, the facial expression detection unit, the system controller 61 as a statistical unit, and the storage unit 62 as a storage unit. The case where the computer 53 as the facial expression information generating apparatus is configured is described. The present invention is not limited to this, and the facial expression information generating apparatus of the present invention may be configured by acquisition means, person detection means, facial expression detection means, statistical means, and storage means having various configurations.

  Furthermore, in the facial expression information generation process of the first embodiment described above, the computer 53 as the facial expression information generation device is configured by the system controller 61 as the facial expression detection means, detection information determination means, and screen data generation means. Said about the case. The present invention is not limited to this, and the facial expression information generation apparatus of the present invention may be configured by facial expression detection means, detection information determination means, and screen data generation means having various configurations.

  The present invention can be applied to a program for acquiring marketing information, for example.

1 ... Database creation system 2, 52 ... Video camera 3, 5 ... Computer 4 ... Display 11, 61 ... System Controllers 12, 62 ... Storage unit 13, 63 ... External interface 14, 55 ... External operation unit 51 ... Display information generation system

Claims (18)

Acquisition means for acquiring moving image data;
Person detecting means for detecting a person from the moving image data;
Facial expression detection means for detecting facial expressions from the detected human face;
Storage means for associating and storing the number of persons, the facial expression, and the date and time when the facial expression was detected, and statistical means for statistically calculating the facial expression and the number of persons at a specific time. Characteristic facial expression information generation device. Furthermore, it has an age and gender detection means for discriminating age and gender from the detected face of the person,
The statistical means is
The facial expression information generating apparatus according to claim 1, wherein the facial expression and the number of persons are statistically determined according to a user's request, for age and sex, or for each age or sex. The age and gender are
The expression information generation device according to claim 2, wherein the expression information is an apparent age and an apparent gender. A moving direction detecting means for detecting a moving direction of the person,
The statistical means is
The facial expression information generating apparatus according to any one of claims 1 to 3, wherein the facial expression and the number of persons are statistics for each of the movement direction and the specific time. 5. The apparatus according to claim 1, further comprising identity determining means for determining identity of a detected person among a plurality of moving image data acquired by the moving image data acquiring means. Facial information generator. Specific person determining means for determining whether or not the detected person is a specific person,
The facial expression detection means includes
The facial expression information generating apparatus according to claim 1, wherein an facial expression is detected from the detected face of the person only when the detected person is the specific person. Specific person determining means for determining whether or not the detected person is a specific person,
The statistical means is
The facial expression information generating apparatus according to claim 1, wherein a statistical method is changed according to whether or not the person is the specific person. Correction value determining means for determining a correction value based on a specific range luminance in a specific range of image data of a predetermined photographing reference object acquired by the acquiring means,
The age gender detecting means is
The facial expression information generating apparatus according to claim 2, wherein the age is determined based on the correction value. An environmental correction value determining means for determining an environmental correction value from environmental elements according to the current environment at the shooting location;
The age gender detecting means is
The facial expression information generating apparatus according to any one of claims 2 to 8, wherein the age is determined based on the correction value. The person detecting means is
The facial expression information generating apparatus according to any one of claims 2 to 8, wherein a time when the detected person exists in the moving image data is detected for each person. The screen data generating means for generating screen data so that the color-coded frames corresponding to the detection information and the expression level correspond to the movement of the person in the moving image data. The facial expression information generating apparatus according to any one of 1 to 10. Display means for displaying video data;
Photographing means for photographing a person who views video data;
A terminal device having transmission means for transmitting moving image data photographed by the photographing means;
Obtaining means for obtaining the moving image data;
Person detecting means for detecting a person from the moving image data;
Facial expression detection means for detecting facial expressions from the detected human face;
Storage means for storing the number of persons, the facial expression, and the date and time when the facial expression was detected in association with each other, and statistics for statistically calculating the facial expression and the number of persons in association with the video data at specific time intervals An audience rating survey system characterized by comprising: a facial expression information generating device comprising: means. Display means for displaying video data;
Photographing means for photographing a person who views video data;
Person detecting means for detecting a person from moving image data photographed by the photographing means;
Facial expression detection means for detecting facial expressions from the detected human face;
Transmitting means for associating and transmitting the number of persons, the facial expression, and time information of the video data;
A terminal device having
A statistical device having statistical means for statistically calculating the facial expression and the number of persons in association with the video data at specific time intervals;
The audience rating survey system characterized by having. The person detecting means is
14. The audience rating survey system according to claim 12, wherein a time when the detected person exists in the moving image data is detected for each person. An acquisition step of acquiring moving image data;
A person detecting step of detecting a person from the moving image data;
A facial expression detection step for detecting a facial expression from the detected human face;
A storage step of associating and storing the number of persons, the facial expression, and the date and time when the facial expression was detected, and a statistical step of statistically calculating the facial expression and the number of persons at a specific time. A facial expression information generation program. Facial expression detection means for detecting a person from moving image data and detecting an expression from the detected face of the person;
Detection information determining means for determining the age and sex of the person as detection information;
Screen data generating means for generating screen data so that the color-coded frames corresponding to the detection information and the degree of facial expression correspond to the movement of the person in the moving image data. Facial information generator. The screen data generation means includes
The facial expression information generating apparatus according to claim 12, wherein the screen data including only the frame and the degree of facial expression is generated. A facial expression detection step of detecting a person from moving image data and detecting a facial expression from the detected face of the person;
A detection information determination step for determining the age and sex of the person as detection information;
A screen data generation step for generating screen data so that the color-coded frames corresponding to the detection information and the expression level correspond to the movement of the person in the moving image data. Facial information generation program.