JP2021033359A - Emotion estimation device, emotion estimation method, program, information presentation device, information presentation method and emotion estimation system - Google Patents

Abstract

To provide a technique capable of improving accuracy of emotion estimation.SOLUTION: An emotion estimation device is provided that includes: an association processing unit that associates a plurality of person area images in which the same person is captured from a plurality of viewpoints; an evaluation value calculation unit that calculates an evaluation value of a shooting condition for each of the plurality of person area images; and a comprehensive emotion estimation unit that generates comprehensive estimated emotion information of the person based on the estimated emotion information generated from each of the plurality of person area images and the evaluation value.SELECTED DRAWING: Figure 1

Description

The present invention relates to an emotion estimation device, an emotion estimation method, a program, an information presentation device, an information presentation method, and an emotion estimation system.

Patent Document 1 proposes a facial expression recognition device that measures facial expressions with a computer based on time-series images of facial expressions and performs machine recognition of facial expressions.

Japanese Unexamined Patent Publication No. 3-252775

A technique for estimating the facial expression type and emotional expression intensity of a human face image captured by a single camera (imaging device) including the above-mentioned Patent Document 1 (hereinafter referred to as "facial expression estimation" technique) already exists. Is known for. In particular, in recent years, with the advent of deep learning, the accuracy of its estimation (identification) has improved.

On the other hand, in the image recognition technology in the real environment (In the Wild environment), the appearance is based on the disturbance factor in the facial expression estimation processing, for example, the positional relationship between the camera and the subject, as compared with the ideal laboratory control environment. There is a problem that the accuracy of facial expression estimation (including the performance of recognition processing) is lowered due to changes in the image, lighting fluctuation, occlusion, and the like.

Therefore, an object of the present invention is to provide a technique capable of improving the accuracy of emotion estimation.

In order to solve the above problem, according to a certain viewpoint of the present invention, a mapping processing unit for associating a plurality of person area images in which the same person is captured from a plurality of viewpoints and photographing each of the plurality of person area images. Comprehensive emotion that generates comprehensive estimated emotion information of the person based on the evaluation value calculation unit that calculates the evaluation value of the condition, the estimated emotion information generated from each of the plurality of person area images, and the evaluation value. An emotion estimation device including an estimation unit is provided.

The association processing unit may associate the plurality of person area images based on the position information in the space of the person reflected in each of the plurality of person area images.

The association processing unit may associate the plurality of person area images based on the shooting times of the plurality of person area images.

The comprehensive emotion estimation unit may calculate the weight of each of the plurality of estimated emotion information based on the evaluation value, and generate the comprehensive estimated emotion information based on the estimated emotion information and the weight. ..

The comprehensive emotion estimation unit may calculate the weight based on the priority of the imaging conditions and the evaluation value.

The comprehensive emotion estimation unit may calculate the weight based on the normalization between the evaluation values.

The evaluation value calculation unit may set the evaluation value based on the likelihood in the estimation of each of the plurality of estimated emotion information.

The evaluation value calculation unit determines the evaluation value of the shooting condition of the person area image based on the angle or distance between the person and the camera that captures the person area image in response to each of the plurality of person area images. It may be set.

The evaluation value calculation unit corresponds to each of the plurality of person area images, and based on at least one of the degree of light irradiation to the person and the degree of imaging shielding of the person, the shooting conditions of the person area image. The evaluation value of may be set.

The evaluation value calculation unit sets the evaluation value of the shooting condition of the person area image based on at least one of the resolution and the image quality of the person area image corresponding to each of the plurality of person area images. You may.

Further, according to another viewpoint of the present invention, associating a plurality of person area images obtained by capturing the same person from a plurality of viewpoints and calculating an evaluation value of shooting conditions for each of the plurality of person area images. An emotion estimation method including the generation of comprehensive estimated emotion information of the person based on the estimated emotion information generated from each of the plurality of person area images and the evaluation value is provided. ..

Further, according to another viewpoint of the present invention, the computer has a matching processing unit for associating a plurality of person area images in which the same person is captured from a plurality of viewpoints, and shooting conditions for each of the plurality of person area images. An evaluation value calculation unit that calculates an evaluation value, and a comprehensive emotion estimation unit that generates comprehensive estimated emotion information of the person based on the estimated emotion information generated from each of the plurality of person area images and the evaluation value. A program for functioning as an emotion estimation device is provided.

Further, according to another viewpoint of the present invention, the evaluation value of the shooting conditions of each of the plurality of person area images associated with the same person being imaged from the plurality of viewpoints is calculated, and each of the plurality of person area images is calculated. It is provided with a control unit that controls so that when the comprehensive estimated emotion information of the person is generated based on the estimated emotion information generated from the above and the evaluation value, the comprehensive estimated emotion information is presented. , An information presentation device is provided.

The control unit controls so that the person area image in which the person appears is presented, and the comprehensive estimated emotion information is superimposed on the position corresponding to the coordinates in which the person appears in the person area image. You may control it.

Further, according to another viewpoint of the present invention, the evaluation value of the shooting conditions of each of the plurality of person area images associated with the same person being imaged from the plurality of viewpoints is calculated, and each of the plurality of person area images is calculated. When the comprehensive estimated emotional information of the person is generated based on the estimated emotional information generated from the above and the evaluation value, the comprehensive estimated emotional information is controlled to be presented. Information presentation methods are provided.

Further, according to another viewpoint of the present invention, the evaluation value of the shooting conditions of each of the plurality of person area images associated with the same person being imaged from a plurality of viewpoints is calculated by the computer, and the plurality of persons. Control to control so that when the comprehensive estimated emotion information of the person is generated based on the estimated emotion information generated from each of the region images and the evaluation value, the comprehensive estimated emotion information is presented. A program for functioning as an information presenting device is provided.

Further, according to another viewpoint of the present invention, the associating processing unit for associating a plurality of person area images in which the same person is captured from a plurality of viewpoints and the evaluation value of the shooting conditions of the plurality of person area images are determined. An evaluation value calculation unit to be calculated, and a comprehensive emotion estimation unit that generates comprehensive estimated emotion information of the person based on the estimated emotion information generated from each of the plurality of person area images and the evaluation value. Provided is an emotion estimation system including an emotion estimation device and an information presentation device including a control unit for controlling the comprehensive estimated emotion information to be presented.

Further, according to another viewpoint of the present invention, associating a plurality of person area images obtained by capturing the same person from a plurality of viewpoints and calculating an evaluation value of shooting conditions for each of the plurality of person area images. And, based on the estimated emotion information generated from each of the plurality of person area images and the evaluation value, the comprehensive estimated emotion information of the person is generated, and the comprehensive estimated emotion information is presented. Emotion estimation methods are provided, including controlling and including.

As described above, according to the present invention, there is provided a technique capable of improving the accuracy of emotion estimation.

It is explanatory drawing which shows an example of the schematic structure of the information communication system which concerns on embodiment of this invention. It is a block diagram which shows an example of the hardware configuration of the emotion estimation server, the camera, and the information presenting terminal which concerns on this embodiment. It is a block diagram which shows an example of the functional structure of the camera which concerns on the same embodiment. It is a block diagram which shows an example of the functional structure of the emotion estimation server which concerns on the same embodiment. It is explanatory drawing for demonstrating an example of the data table of the emotion person position DB which is associated with the emotion person person collating unit and is stored in the storage part. It is a block diagram which shows an example of the functional structure of the information presenting terminal which concerns on this embodiment. It is explanatory drawing for demonstrating an example of the display screen presented by the presenting part of an information presenting terminal. It is explanatory drawing which shows an example of the operation flow of the information communication system which concerns on this embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

Further, in the present specification and the drawings, a plurality of components having substantially the same or similar functional configurations are distinguished by adding different alphabets after the same reference numerals. However, if it is not necessary to distinguish each of a plurality of components having substantially the same or similar functional configurations, only the same reference numerals are given.

(0. Overview)
Patent Document 1 proposes a facial expression recognition device that measures facial expressions with a computer based on time-series images of facial expressions and performs machine recognition of facial expressions.

From the human face image (or body image) captured by one camera (imaging device) including the above-mentioned Patent Document 1, the type of facial expression (or behavioral behavior) and the emotional expression intensity of the face are estimated. Technology (hereinafter referred to as "facial expression estimation" technology) is already known. In particular, in recent years, with the advent of deep learning, the accuracy of its estimation (identification) has improved.

On the other hand, in image recognition in a real environment (In the Wild environment), compared to an ideal laboratory control environment, the appearance is based on disturbance factors in facial expression estimation processing, for example, the positional relationship between the camera and the subject. There is a problem that the accuracy of facial expression estimation is lowered due to changes, lighting fluctuations, occlusion, and the like.

Furthermore, in recent years, a telework system has been developed that realizes a collaborative environment as if working together even if they are separated by constantly sharing images from multiple cameras installed at remote locations in a remote environment. (Non-Patent Document 1: Masayuki Tokumatsu and Masato Nonaka, Development of Super Realistic Telework System, OKI Technical Review, Vol.84 (1), pp.32-35, 2017). In a remote environment, there is a problem that it is difficult to grasp the situation of the other party, for example, "emotion" compared to the room environment (Non-Patent Document 2: Yasuko Arimoto et al., Emotional understanding under modality control in online communication, Proceedings of the 2014 Autumn Meeting of the Acoustic Society of Japan, pp.385-386, 2014), using the above-mentioned "facial expression estimation technology" to generate facial expression estimation information and present it to a remote partner. Therefore, it is thought that the problem can be alleviated.

However, in an office in a real environment, occlusion due to the above-mentioned lighting fluctuation and reflection in the camera of various indoor equipment occurs with high probability, and the accuracy of the estimated emotion information is lowered. Therefore, we try to solve the problem by using a plurality of cameras included in the telework system.

The present embodiment is a "facial expression estimation system using a plurality of viewpoint images" including a plurality of cameras and a server that performs facial expression estimation processing from a facial image, and the server is an estimation target among a plurality of camera images. "Emotional person matching unit" that associates the captured data or extracted data of the camera, "Imaging condition comparison unit" that compares the shooting conditions of each of multiple cameras and calculates the evaluation value of each of the multiple estimated emotional information, and multiple estimations. Emotion information A "comprehensive emotion estimation unit" that calculates the final estimated emotion information from each evaluation value is provided. By realizing multiple facial expression estimation processes using a camera having a plurality of viewpoints, it is possible to realize highly accurate facial expression estimation as compared with the conventional facial expression estimation technology based on a single camera image.

(1. First Embodiment)
Subsequently, with reference to FIG. 1, a schematic configuration of an information communication system (emotion estimation system) according to the embodiment of the present invention will be described.

FIG. 1 is an explanatory diagram showing an example of a schematic configuration of an information communication system according to the present embodiment. Referring to FIG. 1, the information communication system includes an emotion estimation server (emotion estimation device) 100, a plurality of cameras 200, an information presentation terminal (information presentation device) 300, and a LAN 50, and a part thereof (for example, a plurality of cameras 200). ) May exist in the office 400. Further, in the office 400, in addition to a part of the information communication system, a user 900, an obstacle 500, and a lighting 600 are present as examples. The shooting ranges of the plurality of cameras 200 may overlap each other.

FIG. 2 shows the “device according to the present embodiment” without distinguishing each of the emotion estimation server 100, the camera 200, and the information presentation terminal 300 (hereinafter, the emotion estimation server 100, the camera 200, and the information presentation terminal 300) according to the present embodiment. It is a block diagram which shows an example of the hardware configuration of). It should be noted that it is not necessary that all of the above-mentioned devices are provided with all of the following hardware configurations (for example, the emotion estimation server 100 does not need to be directly provided with a sensor), and each device described later does not need to be provided with a sensor. A hardware module capable of realizing a functional configuration may be provided with an appropriate limitation.

Referring to FIG. 2, the apparatus according to the present embodiment includes a bus 801 and a CPU (Central Processing Unit) 803, a ROM (Read Only Memory) 805, a RAM (Random Access Memory) 807, a storage device 809, a communication interface 811 and a sensor. It includes an 813, an input device 815, a display device 817, and a speaker 819. The CPU 803 executes various processes in the apparatus according to the present embodiment. Further, the ROM 805 stores a program and data for causing the CPU 803 to execute the processing in the apparatus according to the present embodiment. Further, the RAM 807 temporarily stores the program and the data when the processing of the CPU 803 is executed.

The bus 801 connects the CPU 803, the ROM 805, and the RAM 807 to each other. A storage device 809, a communication interface 811, a sensor 813, an input device 815, a display device 817, and a speaker 819 are further connected to the bus 801. Bus 801 includes, for example, a plurality of types of buses. As an example, the bus 801 includes a high-speed bus connecting the CPU 803, ROM 805, and RAM 807, and one or more other buses slower than the high-speed bus.

The storage device 809 stores data to be temporarily or permanently stored in the device according to the present embodiment. The storage device 809 may be, for example, a magnetic storage device such as a hard disk, or an EEPROM (Electrically Erasable and Programmable Read).
It may be a non-volatile memory such as Only Memory, flash memory, MRAM (Magnetoresistive Random Access Memory), FeRAM (Ferroelectric Random Access Memory) and PRAM (Phase change Random Access Memory).

The communication interface 811 is a communication means included in the device according to the present embodiment, and communicates with an external device via a network (or directly). The communication interface 811 may be an interface for wireless communication, and in this case, for example, a communication antenna, an RF circuit, and other circuits for communication processing may be included. Further, the communication interface 811 may be an interface for wired communication, and in this case, for example, a LAN terminal, a transmission circuit, and other circuits for communication processing may be included.

The sensor 813 is, for example, a camera, a microphone, a biosensor, another sensor, or a combination thereof. The camera captures a subject and includes, for example, an optical system, an image sensor, and an image processing circuit. The microphone picks up ambient sound, converts the sound into an electric signal, and converts the electric signal into digital data.

The input device 815 is a touch panel, a mouse, a line-of-sight detection device, and the like. The display device 817 displays an output image (that is, a display screen) from the device according to the present embodiment, and is realized by using, for example, a liquid crystal, an organic EL (Organic Light-Emitting Diode), a CRT (Cathode Ray Tube), or the like. Can be done. The speaker 819 outputs voice, converts digital data into an electric signal, and converts the electric signal into voice.

Next, an example of the functional configuration of the “camera 200” according to the present embodiment will be described with reference to FIG. The camera 200 has a function of generating measurement data in the real world, and is an image captured in the office 400 including the user 900 (may be a moving image, and the image may include an image) or the user 900. Various behaviors / physiological reactions (including facial expressions, gestures, voices, etc.) are externally measured, and the acquired sensor data is transmitted to the emotion estimation server 100 described later.

FIG. 3 is a block diagram showing an example of the functional configuration of the camera 200 according to the present embodiment. Referring to FIG. 3, the camera 200 includes a communication unit 210, a measurement unit 220, and a control unit 230. Although not shown in FIG. 3, the camera 200 may further include a storage unit for storing measurement data, a display unit for showing the internal operation status to the user, and the like.

The communication unit 210 communicates with another device. For example, the communication unit 210 is directly connected to the LAN 50 and communicates with the emotion estimation server 100. Further, it may communicate with another camera 200. The communication unit 210 may be implemented by the communication interface 811.

The measurement unit 220 externally measures the measurement data in the real world (for example, a captured image of a bird's-eye view of the office 400) and the behavior and physiological reaction of the user 900 to acquire the data. The behavior and biological reaction data are, for example, moving behavior in the office 400 measured by a camera, image data including facial expressions and body posture states, and voice data measured by a microphone. Furthermore, pulse data estimated from minute changes in skin color in human body imaging data, line-of-sight motion data and pupil diameter data estimated from eye imaging data, and measurement can be performed if the camera is equipped with an infrared thermography function. It may be higher-order physiological index data that reflects the user's autonomic nervous system activity information such as skin temperature distribution data.

The estimation process may be performed in the camera 200 by the control unit 230 described later, or may be performed in the emotion estimation server 100 by transmitting raw measurement data from the camera 200 to the emotion estimation server 100 described later. Good. The measuring unit 220 may be mounted by the sensor 813.

The control unit 230 provides various functions of the camera 200. The control unit 230 associates the measurement data with the position information data of the user 900 to be measured and the time information data obtained by measuring the measurement data, and transmits the measurement data to the emotion estimation server 100 via the communication unit 210. You may. The camera 200 may perform not only measurement but also analysis processing including preprocessing, feature extraction processing, and estimation, and the control unit 230 may perform various arithmetic processing in that case. The control unit 230 may be mounted by the CPU 803, the ROM 805, and the RAM 807.

The camera 200 may have a function of estimating the position of the user 900 included in the imaging range. For example, the camera 200 may also be equipped with a laser range finder function and may have a three-dimensional measurement function of an imaging range. Further, even if the camera 200 is a general-purpose monocular camera, there are a plurality of existing methods for estimating the existence position of the person to be imaged in the three-dimensional real space, which is a known method (for example, Non-Patent Document 3: Osawa). Tatsuya et al., Person tracking technology for video monitoring, NTT Technology Journal, 19 (8), pp.17-20, 2007).

In the embodiment of the present invention, the camera 200 may be used for fixed installation in an office, for example, in which case information on internal or external parameters of the camera (position in three-dimensional space of the camera, orientation, imaging direction, image). Assuming that the data related to (including information such as the angle and the imaging range) is known, the camera 200 and the emotion estimation server 100 have the data of the camera parameters in the storage unit in advance, and the data is used for estimating the position of the person to be imaged. It may be used (that is, the position of the person to be imaged may be estimated based on the data and the data acquired from the camera 200).

Further, the camera 200 may be, for example, an in-vehicle camera installed in an automobile. In this case as well, the camera 200 may be calculated by imaging data of the surrounding environment of the vehicle, the position of the vehicle (GPS (Global Positioning System), dead reckoning, high-precision map, SLAM (Simultaneous Localization And Mapping), etc.) and posture. Information may be acquired in real time to generate information on external parameters of the camera. Further, the plurality of cameras 200 are cameras installed in a plurality of automobiles, and the plurality of cameras 200 may generate mutual positional relationship information by inter-vehicle communication between the plurality of automobiles.

Next, an example of the functional configuration of the “emotion estimation server 100” according to the present embodiment will be described with reference to FIG. FIG. 4 is a block diagram showing an example of the functional configuration of the emotion estimation server 100 according to the present embodiment. Referring to FIG. 4, the emotion estimation server 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

The communication unit 110 communicates with another device. For example, the communication unit 110 is directly connected to the LAN 50 and communicates with the camera 200 and the information presenting terminal 300. The communication unit 110 may be implemented by the communication interface 811.

The storage unit 120 stores programs and data for the operation of the emotion estimation server 100. The storage unit 120 includes an emotion estimation dictionary DB 121 and an emotion person position DB 122.
In order to estimate (identify) the user's emotions (the emotions may include the type and intensity of facial expressions and gestures) from the sensor data (in the present embodiment, for example, image data including the user 900). Contains data from the trained emotion estimation model (emotion recognition dictionary). The emotion estimation model performs learning processing by associating the sensor data acquired in advance (for example, an image including facial expressions of a large number of persons) with the data of the correct answer information of the emotions of the person to be imaged at the time of acquiring the sensor data. Will be generated. The correct answer information of the emotion may be measured from the person by the questionnaire method or the like in the learning processing phase. In addition, the emotion estimation model is generated by classifying and dividing the data for each individual of the user 900, for each predetermined period, for each behavior type of the user 900, etc., and learning processing each, and a plurality of emotion estimation models according to the conditions are generated. It doesn't matter if it exists.

Since a method for estimating a person's personal emotion from sensor data is known (for example, Japanese Patent Application Laid-Open No. 2012-59107), further description thereof will be omitted in this paper. The emotion estimation model is stored in the emotion estimation dictionary DB 121. In the emotional person position DB 122 described later, the estimated emotional information of the user 900 described later and the position information in the office 400 of the user 900 are stored in association with each other. The storage unit 120 may be mounted by the storage device 809.

The control unit 130 provides various functions of the emotion estimation server 100. The control unit 130 includes a face detection unit 131, an emotion estimation unit 133, an emotion person matching unit 135, a shooting condition comparison unit 137, and a comprehensive emotion estimation unit 139. The control unit 130 may be mounted by the CPU 803, the ROM 805, and the RAM 807.

The face detection unit 131 identifies a region of the face image of the user 900 from the captured image of each camera 200 by the face detection technique, cuts out (extracts) the area, and stores it in the storage unit 120. The face image is a region in which the face of the user 900 of the captured image is captured, and corresponds to an example of an image (person region image) in a region in which the body of a person is captured. At this time, the emotional person collation unit 135, which will be described later, may associate the face image with the position information of the user 900 and store it in the emotional person position DB 122 of the storage unit 120. Since there are a plurality of known methods for face detection technology, the description thereof will be omitted (for example, Non-Patent Document 4: Takayoshi Yamashita et al., Face detection / facial expression recognition technology, Journal of the Institute of Image Information and Television Engineers, 62 (5)). , Pp.708-713, 2008).

The emotion estimation unit 133 is based on behavior image data and biological reaction measurement data (sensor data) acquired from the user 900 via the camera 200 and the communication unit 110, and the estimation model data of individual emotions for each user 900 and the estimated model data thereof. Generates estimated emotional information estimated (identified) by. Further, the emotion estimation unit 133 has a function of storing the generated estimated model data and the estimated emotion information in the storage unit 120. Further, the process of generating the estimated emotion information may be performed by the camera 200, and the emotion estimation server 100 may receive the estimated emotion information from the camera 200 instead of the image data.

Here, the explanation of personal feelings and their estimation methods is supplemented. Personal emotions are, for example, "evaluative reactions to people, things, events, and the environment among various information processes that people perform in their mental processes" (Ortony et al., 1988; Ohira, 2010). Is defined as. Specific types of emotions include happiness, surprise, fear, anger, disgust, sadness, and psychologist James A. on a basic emotion-based discrete model that corresponds to facial expressions by psychologist Paul Ekman. The quadrants of emotions and sorrows in the emotional dimension-based continuous model of pleasure and alertness by Russel are known. Other continuous models include positive or negative emotions by Watson, 3-axis models by Wund (pleasure, excitement, tension), and 4-axis models by Plutchik. Other applied / complex emotions include confusion, interest, mental stress, concentration, fatigue, busyness, creativity, relaxation / tension, motivation, empathy, and reliability. Furthermore, the liveliness that is experienced as a group atmosphere in business activities can be said to be a type of higher-level emotion. The effective range of the definition of emotion in the present invention is wider than the above-mentioned basic emotion, and includes all internal "states" of the user and "situations" that take into account the influence of the user's surrounding environment and context. As an example, the positive emotion and its degree are the degree of rapidity itself, the combination of the degree of pleasure and the degree of arousal, the magnitude of the intensity of happiness in the basic emotion, or the magnitude of the intensity of fear, anger, disgust, sadness, etc. May be expressed as an index.

What kind of emotion and how much a person has can be determined by linguistic reporting of the person's letters, sentences, and symbols, for example, by using the questionnaire method. Well-known examples of the questionnaire include "Affect Grid" and "SAM scale". However, since the measurement method using the questionnaire requires answering work, the measurement itself may interfere with the original purpose work in daily life where some other work such as work is performed. is there.

Therefore, in this information communication system, the emotion estimation unit 133 mechanically obtains emotions (obtained by the questionnaire method or the like) based on the behavioral and biological reaction data measured by the camera 200 and the information presentation terminal 300 described above. Estimate processing. In order to perform the estimation process, the data of the emotion estimation model generated in advance by the learning process is required. The emotion estimation model is generated from, for example, a group of data in which the data of the behavior or biological reaction at a certain time point / situation is associated with the training data consisting of the answer data of the questionnaire. For example, questions about the user's facial expression, voice, heartbeat activity, skin electrical activity, and other behavioral / biological data measured from countless cameras, microphones, and wearable activity meters embedded in the office, and the user's subjective emotions. The correct answer data that was answered on paper is associated with the training data. The behavior / biological data may be feature data for learning processing in which the measured values from the sensors are converted.

The feature amount data may be the position of a representative feature point of the face, the distance of a straight line connecting each of the two points, or the angle formed. Alternatively, the feature data may be the fundamental frequency, power, average speech speed, maximum value and standard deviation of the first-order cepstrum coefficient of speech. Alternatively, the feature amount data may be the average value of the heart rate or the beat interval, the standard deviation, or the heart rate variability. Alternatively, the feature amount data may be an average value of the skin conductance level, a standard deviation, an increase / decrease rate, or the like. These feature data may be used in any way, as an absolute value at a certain time point, or as a relative rate of change between two time points.

For the generation of the emotion estimation model using the training data, for example, a known SVM (Support Vector Machine) or deep learning method may be used as a learning method, or a regression analysis method is simply used. It may be used. Further, the learning model may be generated for each individual user, or a model common to humans may be generated using training data of a plurality of users. The emotion estimation unit 133 can estimate individual emotions from the behavior / biological data of a certain person by using the obtained emotion estimation model data.

The emotion estimation server 100 (for example, the emotion estimation unit 133) may have a function of generating training data for the above-mentioned personal emotion estimation process or the emotion estimation model itself. Further, the above-mentioned feature amount data for the training data is generated not by the emotion estimation server 100 but by the camera 200 or the information presenting terminal 300, and the camera 200 or the information presenting terminal 300 estimates the feature amount data. It may be sent to the server 100. In this embodiment, in particular, as in Patent Document 1 described above, an emotion that inputs a facial image of a person (user 900) and outputs estimated emotion information of the identification result of the facial expression (6 basic facial expressions of Ekman, etc.). The estimation method is mainly assumed.

The emotional person matching unit 135 functions as a matching processing unit that performs processing for associating face images of users 900 acquired from a plurality of cameras 200 from a plurality of viewpoints. At this time, since it is desired to associate with a certain user 900 individual as a reference, for example, when there are two users 900A and 900B in the office 400, the face images (extracted images) of the user 900A and the user 900B are respectively. ) Are not associated with each other (because the personal feelings of the user 900A and the user 900B are different from each other). The emotional person collation unit 135 may store information on the relationship between the associated face images (extracted images) in the storage unit 120. The association between the face images may include not only the case where the face images are directly associated with each other but also the case where the estimated emotion information obtained from each of the plurality of face images is directly associated with each other.

If there are two users, 900A and 900B, in the office 400 and there are two cameras, 200A and 200B, that capture both of them within the imaging range, the face image of the user 900A is two viewpoints taken by the cameras 200A and 200B, respectively. I have a minute. At this time, for example, the face images of the user 900A captured by the camera 200A and the camera 200B are referred to as face images 900A-200A and face images 900A-200B, respectively. Similarly, as the face image of the user 900B, two viewpoints of the face image 900B-200A and the face image 900B-200B can be obtained. At this time, the correct process is to associate the face image 900A-200A with the face image 900A-200B and associate the face image 900B-200A with the face image 900B-200B. Other combinations of different combinations of people to be imaged / estimated, for example, a process of associating a face image 900A-200A with a face image 900B-200B is an error in the present embodiment, and it is necessary to avoid this.

In order to correctly associate the face images, the emotional person matching unit 135 may use the position information of the user 900. That is, the emotional person collation unit 135 may associate a plurality of face images with each other based on the position information in the user's space captured in each of the plurality of face images. For example, the emotional person matching unit 135 may be associated with a plurality of facial images when the positions of the users appearing in the plurality of facial images are within a predetermined range. As described above, the emotional person collation unit 135 is installed in the office 400. Each camera 200 may have a function of estimating the position of the user 900 included in the imaging range. Since a plurality of people cannot exist in a certain three-dimensional position in the physical space in an overlapping manner, the user 900 existing in a certain three-dimensional position is uniquely determined. The emotional person collation unit 135 may store the facial image and the position information for each user 900 in the emotional person position DB 122 in association with each other. The position information may be information on a two-dimensional position on a horizontal plane or the like other than the three-dimensional information.

For example, the above-mentioned face image 900A-200A and face image 900A-200B are associated with position information A (example: X = 0, Y = 0, Z = 0), and the face image 900B-200A and face image 900B are associated with each other. If −200B is associated with position information B (eg, X = 2000, Y = 3000, Z = 0), the emotional person matching unit 135 corresponds to the face images associated with the same position information. You may attach it. That is, the emotional person collation unit 135 associates the face image 900A-200A and the face image 900A-200B associated with the same position information A, and the face image 900B associated with the same position information B. -200A and the face image 900B-200B may be associated with each other.

In the present embodiment, the correspondence between the image of a plurality of viewpoints based on the position information and the estimated emotion information has been mainly assumed and described, but in addition, the emotion person matching unit 135 is a known multi-camera person mapping technique (Person). Re-identification), for example, matching processing using attribute information of each person (person's age, gender, clothes, etc.) and appearance information (information about the appearance of the person such as skin color, clothes color, etc.) You may do. Further, the emotional person collation unit 135 may use a known face recognition technique to perform associative processing using personal identification information.

Here, the above-mentioned emotional person position DB 122 will be described with reference to FIG. FIG. 5 is an explanatory diagram for explaining an example of a data table of the emotional person position DB 122 which is associated with the emotional person matching unit 135 and stored in the storage unit 120, which will be described later. The data table of FIG. 5 includes a data ID, an imaging camera ID, a person position (for user 900), a person ID (for user 900), estimated emotional information (for user 900), and other information (eg, time stamp, evaluation). Data such as value) is stored.

The data ID is identification information for uniquely identifying each data. The imaging camera ID is identification information for uniquely identifying each of the plurality of cameras 200 included in the information communication system according to the present embodiment, and which imaging condition is the sensor data acquired from the camera 200. Can be used to obtain information. The person position includes information on a certain three-dimensional position in the physical space in the office 400 of the user 900 imaged from the camera 200 of the image pickup camera ID. The person ID includes identification information for uniquely identifying the user 900, and is necessary information particularly when a plurality of users 900 exist in the office 400. The estimated emotion information is the estimated emotion information of the user 900 estimated by the emotion estimation unit 133.

Other information includes, for example, an evaluation value of shooting conditions by the shooting condition comparison unit 137, which will be described later, and time stamp data indicating the acquisition time (shooting time) of the sensor data acquired from the camera 200. In the above, an example of associating face images with each other by using the position information of the user 900 has been described, but the emotional person matching unit 135 may add the position information of the user 900 or the position information of the user 900. Time stamp data may be used instead. This increases the possibility that face images showing the same person are correctly associated with each other. That is, the emotional person collation unit 135 may associate a plurality of face images based on the time stamp data of each of the plurality of face images. For example, the emotional person collation unit 135 may associate a plurality of face images when the time stamp data of each of the plurality of face images falls within a predetermined range.

The time stamp data may be given by the camera 200 that captures each face image, but the time stamp data given at the same timing among the plurality of cameras is synchronized between the plurality of cameras so as not to cause a deviation. It is desirable to have a mechanism to take a picture. For example, when there is a time management server capable of communicating with each of a plurality of cameras, synchronization may be achieved by notifying each of the plurality of cameras of the same time by the time management server. The function of the time management server may be possessed by the emotion estimation server 100, or may be possessed by a server other than the emotion estimation server 100.

In FIG. 5, for example, the data having the data ID “0001” and the data having the data ID “0002” are generated from the camera 200 having the imaging camera ID “C01” at the same time T1 (same time stamp data). Data, but may be data for two different users 900.

The shooting condition comparison unit 137 functions as an evaluation value calculation unit that calculates an evaluation value of each shooting condition for a plurality of facial images from a plurality of viewpoints of the user 900 associated with the emotional person matching unit 135. As the evaluation value of the shooting conditions, the influence of disturbance factors in the emotion estimation process, such as changes in appearance due to the imaging direction and posture of a person, lighting fluctuations, occlusion, etc., is small, and the smaller the decrease in estimation accuracy due to them, the higher the evaluation value ( (Preferable) shall be priced.

As an example of the disturbance factor and the evaluation value setting, as for the face direction factor, the more the shooting condition is closer to the front face, the less the influence of the face normalization process and the distortion is required, and the facial expression estimation process can be realized with high accuracy. Therefore, the shooting condition comparison unit 137 sets the evaluation value of the shooting condition of the face image based on the angle between the user 900 captured in the face image and the camera 200 that captures the face image corresponding to each of the plurality of face images. You may. More specifically, the shooting condition comparison unit 137 may give a higher evaluation value to the shooting condition as the angle formed by the shooting optical axis of the camera 200 and the axis directly in front of the face of the target user 900 is smaller.

Further, as a factor of illumination fluctuation, the facial expression estimation process can be realized with higher accuracy as the illumination is uniformly applied to the front surface of the face and the shadow difference due to the illumination in the face region is smaller. Therefore, the shooting condition comparison unit 137 may set the evaluation value of the shooting conditions of the face image based on the degree of light irradiation to the person corresponding to each of the plurality of face images. More specifically, the shooting condition comparison unit 137 calculates the shading difference in the face region from the brightness distribution obtained by analyzing the face image, and the smaller the shading difference, the higher the evaluation value for the shooting condition. Good.

Further, as an occlusion factor, the smaller the shielded area (shielded area) on the face image, the more accurate the facial expression estimation process can be realized. Therefore, the shooting condition comparison unit 137 may set the evaluation value of the shooting condition of the face image based on the degree of shielding of the person corresponding to each of the plurality of face images. More specifically, the photographing condition comparison unit 137 may give a higher evaluation value to the photographing condition as the shielding area obtained by the analysis of the face image is smaller. The shielded area is information about facial feature points extracted from the face image but the likelihood of the extraction process is lower than a predetermined value, or facial feature points not extracted from the face image (for example, features). It may be calculated based on the number of points, the position of the feature points, the distribution of the feature points, etc.).

In addition, as a factor of the amount of image information, since the distance between the camera 200 and the user 900 is closer, the facial expression estimation process with higher accuracy can be realized as the number of pixels of the face image (face area) increases. Therefore, the shooting condition comparison unit 137 sets the evaluation value of the shooting condition of the face image based on the distance between the user 900 captured in the face image and the camera 200 that captures the face image corresponding to each of the plurality of face images. You may. More specifically, the shooting condition comparison unit 137 may give a higher evaluation value to the shooting conditions as the distance between the camera 200 and the target user 900 is smaller.

Further, as a factor of the amount of image information, since the number of images captured by the camera 200 is large, the facial expression estimation process can be realized with higher accuracy as the number of pixels of the face image (face region) is larger. Therefore, the shooting condition comparison unit 137 may set the evaluation value of the shooting condition of the face image based on the resolution of the face image corresponding to each of the plurality of face images. More specifically, the shooting condition comparison unit 137 may give a higher evaluation value to the shooting condition as the resolution of the face image is higher.

As a factor of image quality, the lower the (lossy) compression rate of the image data and the higher the image quality, the less the image noise and the more accurate the facial expression estimation process can be realized. Therefore, the shooting condition comparison unit 137 corresponds to each of the plurality of face images and evaluates the shooting conditions of the face image based on the image quality of the face image (for example, the strength of the image noise generated by the compression process). May be set. More specifically, the shooting condition comparison unit 137 may give a higher evaluation value to the shooting condition as the image quality of the face image is higher.

The above description regarding the evaluation value of the shooting condition is an example, and in addition to this, a higher evaluation value may be given to the shooting condition in which the influence of the disturbance factor can be greatly suppressed so that the accuracy of the facial expression estimation process becomes higher (facial expression estimation). For a description of disturbance factors in processing, for example, Non-Patent Document 5: Wang, M. & Deng, W., Deep face recognition: A survey,
https://arxiv.org/abs/1804.06655).

Further, the shooting condition comparison unit 137 does not mean that the influence of the disturbance factor is small, but more directly, the emotion estimation unit 133 has a high "likelihood" required when performing emotion estimation processing from the face image of the user 900. Depending on the situation, the evaluation value of the shooting conditions may be determined. That is, the photographing condition comparison unit 137 may set the evaluation value based on the likelihood in the estimation of each of the plurality of estimated emotion information. More specifically, the photographing condition comparison unit 137 may give a higher evaluation value to the photographing condition as the likelihood in estimating the estimated emotion information is higher. Generally, the greater the influence of disturbance factors, the smaller the likelihood. The likelihood may be, for example, information indicating the likelihood of the calculated estimated emotion information, or a probability of quantifying the probability of the calculated estimated emotion information between 0 and 1. ..

Explaining with the example of FIG. 1, for the user 900, the face image of the camera 200B taken from the oblique direction of the face is evaluated under the shooting condition as compared with the camera 200A under the shooting condition at a position close to the front of the face. The value will be lower. In addition, the camera 200C shoots from an angle farther from the front of the user 900's face than the camera 200B, and is far from the user 900's position (larger distance), and the obstacle 500 is between the user 900 and the user 900. It exists in and an occlusion is occurring. Therefore, the evaluation value of the shooting conditions of the face image of the camera 200C is lower than that of the camera 200B. As a whole, the height of the evaluation value of the shooting conditions will be in the order of the face image of camera 200A> camera 200B> camera 200C. Similarly, the illumination 600 also affects the evaluation value (the evaluation value is higher when the illumination is uniformly applied so that there is no difference in shading).

The comprehensive emotion estimation unit 139 includes estimated emotion information estimated by the emotion estimation unit 133 from each of a plurality of facial images from a plurality of viewpoints of a certain user 900 associated with the emotion person matching unit 135, and a shooting condition comparison unit 137. Based on the calculated evaluation value, the comprehensive estimated emotional information of a certain user 900 is calculated.

An emotion estimation related to pleasant-unpleasant feelings will be described as an example. When the face image 900A-200A, face image 900A-200B, and face image 900A-200C of the user 900 are extracted by the face detection unit 131 from the captured images of the cameras 200A, 200B, and 200C of the office 400, the emotional person matching unit These facial images are associated with 135. It is assumed that the shooting condition comparison unit 137 gives a (preferably) evaluation value that is 3 times, 2 times, or 1 times higher than the shooting conditions of the face image 900A-200A, the face image 900A-200B, and the face image 900A-200C. .. Further, the emotion estimation unit 133 makes the estimated emotion information corresponding to the face images 900A-200A pleasant, the estimated emotion information corresponding to 900A-200B unpleasant, and the estimated emotion information corresponding to 900A-200C pleasant (simplification of explanation). Therefore, in this example, it is presumed that the emotional intensity is not considered).

In such a case
The reason why the estimated emotional information is "pleasant" is
900A-200A: Evaluation value "3 times", 900A-200C: Evaluation value "1 times", so the total evaluation value of the estimated emotion information "pleasant" is calculated as 3 times + 1 times = 4 times. ..
Estimated emotional information is "unpleasant"
900A-200B: Since the evaluation value is "double", the total evaluation value of the estimated emotion information "discomfort" is double.

Therefore, since the total emotion value of "pleasant": the total evaluation value of "discomfort" = 4: 2, the total evaluation value of "pleasant" is higher. Estimated emotional information is calculated as "pleasant". In this way, the comprehensive emotion estimation unit 139 sums the evaluation values of the shooting conditions of the face image 900A-200A, the face image 900A-200B, and the face image 900A-200C for each value of the estimated emotion information, and makes a total evaluation. The estimated emotional information having the maximum value may be selected as the representative value and used as the comprehensive estimated emotional information of the user 900.

Here, it is mainly assumed that the estimated emotional information having the maximum total evaluation value is used as the comprehensive estimated emotional information of the comprehensive user 900. However, when considering the intensity of emotions, the comprehensive emotion estimation unit 139 obtains a value obtained by multiplying the value of each estimated emotion information by the corresponding evaluation value by summing the values of the plurality of estimated emotion information. The obtained value may be calculated as the comprehensive estimated emotional information of the comprehensive user 900. When such a calculation method is applied to the above example, the comprehensive estimated emotion information of the user 900 is expressed by the following equation (1).

Comprehensive estimated emotion information = 3 × (estimated emotion information of 900A-200A) + 2 × (estimated emotion information of 900A-200B) + 1 × (estimated emotion information of 900A-200C) ・ ・ ・ (1)

Further, the evaluation value calculated by the photographing condition comparison unit 137 does not have to be used as it is by the comprehensive emotion estimation unit 139. For example, the comprehensive emotion estimation unit 139 estimates the estimated emotion information of 900A-200A and the estimation of 900A-200B based on the evaluation values of the shooting conditions of the face image 900A-200A, the face image 900A-200B, and the face image 900A-200C. The weights of the emotional information and the estimated emotional information of 900A-200C may be calculated. Then, the comprehensive emotion estimation unit 139 may calculate the comprehensive estimated emotion information of the user 900 based on these weights and these estimated emotion information.

As an example, the comprehensive emotion estimation unit 139 may calculate the weight based on the normalization between the evaluation values. More specifically, the comprehensive emotion estimation unit 139 sets the total of the evaluation values calculated by the shooting condition comparison unit 137 for the face image 900A-200A, the face image 900A-200B, and the face image 900A-200C to 1. The weight may be calculated by adjusting the above. Then, the comprehensive emotion estimation unit 139 may multiply the calculated weight by the value of the corresponding estimated emotion information. For example, the evaluation values of 3 times, 2 times, and 1 times, which are the evaluation values shown in the formula (1), are 3/6, 2/6, and 1/6, and the formula (1) is as shown in (2) below. Is replaced by.

Comprehensive estimated emotion information = (3/6) × (estimated emotion information of 900A-200A) + (2/6) × (estimated emotion information of 900A-200B) + (1/6) × (900A-200C) Estimated emotion information) ・ ・ ・ (2)

Further, in the above-mentioned example, the case where the comprehensive estimated emotion information of the user 900 is calculated mainly considering one type of shooting condition has been described. However, it is also possible to calculate the comprehensive estimated emotional information of the user 900 in consideration of a plurality of types of shooting conditions. Even in such a case, the comprehensive estimated emotional information for the same type of shooting conditions may be calculated in the same manner as in the above example. The different types of shooting conditions may be treated as equivalent and the weight may be calculated, the shooting conditions may be provided with a priority, or the weight may be calculated based on the priority.

That is, the comprehensive emotion estimation unit 139 calculates the weight based on the priority order of the shooting conditions and the evaluation value. For example, the comprehensive emotion estimation unit 139 may calculate the weight of the evaluation value of the shooting condition having a higher priority even if the evaluation values of the different shooting conditions are the same value. For example, the priority of shooting conditions may be set manually in advance, for example, information such as the standard regression coefficient and contribution rate in multiple regression analysis, and the distribution and features of weights obtained by learning a neural network. It may be set automatically by the system based on information such as quantity selection.

Further, the comprehensive emotion estimation unit 139 obtains the comprehensive estimated emotion information by performing a plurality of pattern recognition corresponding to input data from the plurality of cameras 200, ensemble learning from the neural network, and concat processing of the feature amount. It may be calculated.

Next, an example of the functional configuration of the "information presentation terminal 300" according to the present embodiment will be described. The information presenting terminal 300 processes a communication request to another user 900 in response to an input from user 910 (FIG. 1), or acquires comprehensive estimated emotion information of the other user 900 and sends it to the user 910. Information can be presented.

As an example, the information presentation terminal 300 may be a general-purpose smartphone or tablet terminal. Further, although the information presenting terminal 300 is shown in FIG. 1 so as to correspond to one user 910, it may be a shared terminal shared by a plurality of users 910. As yet another example, the information presentation terminal 300 is transmitted from an automated teller machine (VTM) (VTM) with a video communication function, an automatic teller machine at a station, a housing device with a display unit such as a visual call center system, or the like. It may be a terminal that presents the customer's emotion estimation information based on the measurement data to the support staff.

FIG. 6 is a block diagram showing an example of the functional configuration of the information presentation terminal 300 according to the present embodiment. Referring to FIG. 6, the information presenting terminal 300 includes a communication unit 310, a storage unit 320, a control unit 330, an input unit 340, and a presentation unit 350.

The communication unit 310 communicates with another device. For example, the communication unit 310 is directly connected to the LAN 50 and communicates with the emotion estimation server 100. The communication unit 310 may be implemented by the communication interface 811.

The storage unit 320 stores programs and data for the operation of the information presentation terminal 300. The storage unit 320 may be mounted by the storage device 809.

The control unit 330 provides various functions of the information presentation terminal 300. The control unit 330 may be mounted by the CPU 803, the ROM 805, and the RAM 807.

The input unit 340 accepts input from the user 910. Then, the input unit 340 provides the input result to the control unit 330. The input from the user 910 is, for example, to specify another user 900 as a communication request partner, and is realized by selecting the identification information of the other user 900 or the like. The input unit 340 can be mounted by the input device 815.

The presentation unit 350 presents information that can be perceived by the user under the control of the control unit 330. In the embodiment of the present invention, it is mainly assumed that the presentation unit 350 displays a display screen visually perceived by the user. In such a case, the presentation unit 350 can be realized by the display device 823. However, when the presenting unit 350 presents information perceived by the user's hearing, the presenting unit 350 may be realized by a speaker. Alternatively, when the presenting unit 350 presents information perceived by the user's tactile or olfactory sense, the presenting unit 350 may be realized by a tactile or olfactory presenting device.

For example, the presentation unit 350 presents comprehensive estimated emotional information corresponding to another user 900 designated by the user 910 from the input unit 340. Even if the presentation unit 350 displays the comprehensive estimated emotional information of the other user 900 in the area near the area in which the person image of the other user 900 is displayed in the base bird's-eye view moving image as an example of the communication media. Good. At this time, the presentation unit 350 may present the estimation accuracy of the comprehensive estimated emotion information together with the comprehensive estimated emotion information of the other user 900. Such estimation accuracy may be calculated based on the estimation accuracy of the individual estimated emotion information by the same method as the calculation of the comprehensive estimated emotion information based on the individual estimated emotion information described above.

Further, the comprehensive estimated emotion information and the estimation accuracy of the comprehensive estimated emotion information may be stored and stored in, for example, the storage unit 120 of the emotion estimation server 100 or the storage unit 320 of the information presenting terminal 300. In that case, the control unit 330 processes the historical information of each of the comprehensive estimated emotion information and the estimation accuracy of the comprehensive estimated emotion information into, for example, a time series graph, based on the accumulated data, and presents the unit. The screen may be displayed on the 350.

FIG. 7 is an explanatory diagram for explaining an example of a display screen presented by the presentation unit 350 of the information presentation terminal 300. On the display screen, for example, a moving image captured from a bird's-eye view by the camera 200 as a communication medium is displayed, and a person image of the user 900 is displayed in a screen area near the center of the moving image. Further, the control unit 330 controls so that the comprehensive estimated emotion information is superimposed on the position corresponding to the coordinates in which the user 900 appears in the moving image (person area image). More specifically, the comprehensive estimated emotion information associated with the user 900, the estimation accuracy of the comprehensive estimated emotion information, and their history information are displayed on the screen at a position near the person image of the user 900. ing.

The presentation unit of the information presenting terminal 300 is displayed by displaying the person image of the user 900, the comprehensive estimated emotion information of the user 900, the estimation accuracy of the comprehensive estimated emotion information, and their history information at nearby positions. The user who sees 350 can easily recognize the communication media by associating it with the estimated information. This display function at a nearby position is particularly effective when one communication medium contains information of a plurality of users. This communication system treats the communication media, the comprehensive estimated emotional information of the user 900, and the estimated accuracy of the comprehensive estimated emotional information as data, and the information of the data is related to each other and exerts an effect. It is one of the features of.

Here, the neighborhood position is not particularly limited. For example, the neighborhood position may be a position within a predetermined distance with respect to the position of the person image of the user 900. In the example shown in FIG. 7, the display area including the comprehensive estimated emotion information of the user 900, the estimation accuracy of the comprehensive estimated emotion information, their history information, and the identification information of the user 900 is formed by the balloon shape. It is displayed. This makes it easier to understand the relationship between each piece of information and the user. However, the shape of the display area is not limited to the balloon shape.

Here, it is mainly assumed that the communication medium is data transmitted in real time. However, as a modification, the communication media described above does not necessarily have to be data transmitted in real time, and may be, for example, "recorded" or "recorded" past media data.

As described above, it is assumed that the communication system according to the embodiment of the present invention is a new telephone system, and one function may be to ask the past state of collaborative members in remote areas. At this time, for example, the user 900 presents information on the past recorded person video data, the past comprehensive estimated emotion information linked to the past recorded person video data, and the estimation accuracy of the comprehensive estimated emotion information. It may be acquired from the storage unit 120 of the emotion estimation server 100 via the terminal 300.

For example, the user 910 acquires the recorded person video data of the user 900 at the time two hours before the present, the comprehensive estimated emotion information, and the estimated accuracy of the comprehensive estimated emotion information in association with each other via the information presenting terminal 300. You may be able to do it. In such a case, there is a possibility that the user 900 is already absent in the communication system at the time of real time two hours later. However, when acquiring the past media data, it is not always necessary for a plurality of users to exist in the system at the same time, and it is possible that only one user is using the communication system.

Subsequently, an example of the information processing operation according to the present embodiment will be described with reference to FIG. FIG. 8 is an explanatory diagram showing an example of an operation flow of the information communication system according to the present embodiment. As shown in FIG. 8, in step S1101, the face detection unit 131 of the emotion estimation server 100 identifies an area of the face image of the user 900 from the image captured by the camera 200, cuts out (extracts) the area of the face image, and stores the storage unit. Store in 120. In step S1103, the face detection unit 131 of the emotion estimation server 100 determines whether or not the face has been detected from the images captured by the plurality of cameras 200.

When the face is detected from the images captured by the plurality of cameras 200 in the process of step S1103 (S1103: YES), the emotion person matching unit 135 of the emotion estimation server 100 is acquired from the plurality of cameras 200 in step S1105. Performs a process of associating face images from a plurality of viewpoints of the user 900. In step S1107, the shooting condition comparison unit 137 of the emotion estimation server 100 calculates the evaluation value of each shooting condition for the plurality of face images from the plurality of viewpoints of the user 900 associated with S1105.

In step S1109, the comprehensive emotion estimation unit 139 of the emotion estimation server 100 includes estimated emotion information based on a plurality of facial images of a user 900 associated with the emotion person matching unit 135 from a plurality of viewpoints, and a shooting condition comparison unit 137. Based on the evaluation value calculated by the above, the comprehensive estimated emotional information of a certain user 900 is calculated. At this time, the evaluation values are totaled for each value of the estimated emotion information, and the estimated emotion information having the maximum total evaluation value may be selected as the comprehensive estimated emotion information of the user 900. Alternatively, the weight may be calculated from the evaluation value, and the comprehensive estimated emotion information of the user 900 may be calculated based on the weight and the estimated emotion information.

Further, in the process of step S1103, when the face is not detected from the images captured by the plurality of cameras 200 and the face is detected only from the images captured by a single camera 200 (S1103: NO), the comprehensive emotion estimation unit is performed in step S1111. 139 calculates the estimated emotion information of the user 900 from the face image of the user 900 from a single viewpoint and the emotion information thereof. At this time, the comprehensive emotion estimation unit 139 may use the estimated value estimated by the emotion estimation unit 133 as it is, or the emotion estimation unit 133 based on the information of the evaluation value of the shooting condition obtained by the shooting condition comparison unit 137. The estimated value may be modified (for example, the estimated value of the emotion estimation unit 133 may be changed or increased or decreased depending on the magnitude of the disturbance factor).

As described above, the emotion estimation server 100 has higher estimation accuracy that suppresses the influence of various disturbance factors as compared with the case of a single camera, based on the captured images of the user 900 obtained from a plurality of cameras in the office 400. Estimated emotional information can be obtained.

(2. Summary)
As described above, according to the embodiment of the present invention, by performing the comprehensive emotion estimation process based on the images of the plurality of viewpoints obtained from the plurality of cameras, the emotion estimation process is compared with the conventional emotion estimation process by the single camera. Highly accurate emotion estimation is realized.

Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person having ordinary knowledge in the field of technology to which the present invention belongs can come up with various modifications or modifications within the scope of the technical idea described in the claims. , These are also naturally understood to belong to the technical scope of the present invention.

100 Emotion estimation server 110 Communication unit 120 Storage unit 121 Emotion estimation dictionary DB
122 Emotional person position DB
130 Control unit 131 Face detection unit 133 Emotion estimation unit 135 Emotional person matching unit 137 Shooting condition comparison unit 139 Comprehensive emotion estimation unit 200 Camera 210 Communication unit 220 Measurement unit 230 Control unit 300 Information presentation terminal 310 Communication unit 320 Storage unit 330 Control unit 340 Input section 350 Presentation section 500 Obstacles 600 Lighting

Claims (18)

A mapping processing unit that associates multiple person area images in which the same person is captured from multiple viewpoints,
An evaluation value calculation unit that calculates an evaluation value of shooting conditions for each of the plurality of person area images, and an evaluation value calculation unit.
A comprehensive emotion estimation unit that generates comprehensive estimated emotion information of the person based on the estimated emotion information generated from each of the plurality of person area images and the evaluation value.
Emotion estimation device. The association processing unit associates the plurality of person area images with each other based on the position information in the space of the person reflected in each of the plurality of person area images.
The emotion estimation device according to claim 1. The association processing unit associates the plurality of person area images with each other based on the shooting times of the plurality of person area images.
The emotion estimation device according to claim 1 or 2. The comprehensive emotion estimation unit calculates the weight of each of the plurality of estimated emotion information based on the evaluation value, and generates the comprehensive estimated emotion information based on the estimated emotion information and the weight.
The emotion estimation device according to any one of claims 1 to 3. The comprehensive emotion estimation unit calculates the weight based on the priority of the shooting conditions and the evaluation value.
The emotion estimation device according to claim 4. The comprehensive emotion estimation unit calculates the weight based on the normalization between the evaluation values.
The emotion estimation device according to claim 4. The evaluation value calculation unit sets the evaluation value based on the likelihood in the estimation of each of the plurality of estimated emotion information.
The emotion estimation device according to any one of claims 1 to 6. The evaluation value calculation unit determines the evaluation value of the shooting condition of the person area image based on the angle or distance between the person and the camera that captures the person area image in response to each of the plurality of person area images. Set,
The emotion estimation device according to any one of claims 1 to 7. The evaluation value calculation unit corresponds to each of the plurality of person area images, and based on at least one of the degree of light irradiation to the person and the degree of imaging shielding of the person, the shooting conditions of the person area image. Set the evaluation value of
The emotion estimation device according to any one of claims 1 to 8. The evaluation value calculation unit sets the evaluation value of the shooting condition of the person area image based on at least one of the resolution and the image quality of the person area image corresponding to each of the plurality of person area images. ,
The emotion estimation device according to any one of claims 1 to 9. Corresponding multiple person area images in which the same person is captured from multiple viewpoints
To calculate the evaluation value of the shooting conditions for each of the plurality of person area images,
To generate comprehensive estimated emotional information of the person based on the estimated emotional information generated from each of the plurality of person area images and the evaluation value.
Emotion estimation methods, including. Computer,
A mapping processing unit that associates multiple person area images in which the same person is captured from multiple viewpoints,
An evaluation value calculation unit that calculates an evaluation value of shooting conditions for each of the plurality of person area images, and an evaluation value calculation unit.
A comprehensive emotion estimation unit that generates comprehensive estimated emotion information of the person based on the estimated emotion information generated from each of the plurality of person area images and the evaluation value.
A program for functioning as an emotion estimation device. The evaluation value of the shooting conditions of each of the plurality of person area images associated with the same person being imaged from a plurality of viewpoints is calculated, and the estimated emotion information generated from each of the plurality of person area images and the evaluation value are combined with each other. When the comprehensive estimated emotional information of the person is generated based on the above, a control unit for controlling the comprehensive estimated emotional information to be presented is provided.
Information presentation device. The control unit
It is controlled so that the person area image in which the person appears is presented, and the comprehensive estimated emotion information is superimposed on the position corresponding to the coordinates in which the person appears in the person area image.
The information presenting device according to claim 13. The evaluation value of the shooting conditions of each of the plurality of person area images associated with the same person being imaged from a plurality of viewpoints is calculated, and the estimated emotion information generated from each of the plurality of person area images and the evaluation value are combined with each other. When the comprehensive estimated emotional information of the person is generated based on the above, the comprehensive estimated emotional information is controlled to be presented.
Information presentation method. Computer,
The evaluation value of the shooting conditions of each of the plurality of person area images associated with the same person being imaged from a plurality of viewpoints is calculated, and the estimated emotion information generated from each of the plurality of person area images and the evaluation value are combined with each other. When the comprehensive estimated emotional information of the person is generated based on the above, a control unit for controlling the comprehensive estimated emotional information to be presented is provided.
A program to function as an information presentation device. A mapping processing unit that associates multiple person area images in which the same person is captured from multiple viewpoints,
An evaluation value calculation unit that calculates an evaluation value of shooting conditions for each of the plurality of person area images, and an evaluation value calculation unit.
A comprehensive emotion estimation unit that generates comprehensive estimated emotion information of the person based on the estimated emotion information generated from each of the plurality of person area images and the evaluation value.
Emotional estimation device and
A control unit for controlling the comprehensive estimated emotional information to be presented is provided.
Information presentation device and
Emotion estimation system. Corresponding multiple person area images in which the same person is captured from multiple viewpoints
To calculate the evaluation value of the shooting conditions for each of the plurality of person area images,
To generate comprehensive estimated emotional information of the person based on the estimated emotional information generated from each of the plurality of person area images and the evaluation value.
Controlling the presentation of the comprehensive estimated emotional information
Emotion estimation methods, including.

Images