WO2021234845A1 - 情報処理装置、感情推定方法、及び感情推定プログラム - Google Patents

情報処理装置、感情推定方法、及び感情推定プログラム Download PDF

Info

Publication number
WO2021234845A1
WO2021234845A1 PCT/JP2020/019906 JP2020019906W WO2021234845A1 WO 2021234845 A1 WO2021234845 A1 WO 2021234845A1 JP 2020019906 W JP2020019906 W JP 2020019906W WO 2021234845 A1 WO2021234845 A1 WO 2021234845A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
user
processing apparatus
state
information processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/019906
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
咲子 二本柳
啓吾 川島
修 村山
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Priority to JP2022524747A priority Critical patent/JP7106041B2/ja
Priority to DE112020006934.6T priority patent/DE112020006934T5/de
Priority to PCT/JP2020/019906 priority patent/WO2021234845A1/ja
Priority to CN202080100074.6A priority patent/CN115516499A/zh
Publication of WO2021234845A1 publication Critical patent/WO2021234845A1/ja
Priority to US17/894,289 priority patent/US12380731B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/31—User authentication
    • G06F21/32—User authentication using biometric data, e.g. fingerprints, iris scans or voiceprints
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation

Definitions

  • This disclosure relates to an information processing device, an emotion estimation method, and an emotion estimation program.
  • the user's behavior can be specified based on biometric information, voice information, images, line-of-sight data, etc.
  • the emotions of the user who uses the device can be estimated based on the information indicating the behavior of the user.
  • a technique for estimating emotions has been proposed (see Patent Document 1).
  • the information processing apparatus of Patent Document 1 estimates the emotion of the photographer based on the line-of-sight data of the photographer and the sensor data of the photographer.
  • the device responds or processes according to the estimated emotion. For example, the emotions of the user existing in front of the elevator are identified. If the user is frustrated, the device takes steps to reduce the frustration.
  • a method of estimating the user's emotion based on the state when the user is paying attention can be considered.
  • the emotions when the user is paying attention are not the same.
  • the emotion when the user is paying attention to a pedestrian existing in the background of the subject is different from the emotion when the user is paying attention to the subject to be photographed.
  • it is a mistake to presume that the emotions when the user is paying attention are always the same.
  • the purpose of this disclosure is to improve the estimation accuracy of emotions.
  • the information processing device includes an acquisition unit that acquires input information that is information about a user in a certain situation, an object that the user is paying attention to, and an object that the user is paying attention to based on the input information.
  • the user's emotions are determined based on a specific unit that specifies the state, object information indicating the specified object, state information indicating the specified state, and a preset method for estimating emotions. It has an emotion estimation unit for estimating.
  • FIG. 1 is a diagram showing a functional block included in the information processing apparatus of the first embodiment.
  • the information processing device 100 is a device that executes an emotion estimation method.
  • the information processing device 100 may be called an emotion estimation device.
  • FIG. 2 is a diagram showing a configuration of hardware included in the information processing apparatus of the first embodiment.
  • the information processing device 100 includes a processor 101, a volatile storage device 102, a non-volatile storage device 103, an input interface 104, and an output interface 105.
  • the processor 101 controls the entire information processing device 100.
  • the processor 101 is a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), or the like.
  • the processor 101 may be a multiprocessor.
  • the information processing apparatus 100 may have a processing circuit instead of the processor 101.
  • the processing circuit may be a single circuit or a composite circuit.
  • the volatile storage device 102 is the main storage device of the information processing device 100.
  • the volatile storage device 102 is a RAM (Random Access Memory).
  • the non-volatile storage device 103 is an auxiliary storage device of the information processing device 100.
  • the non-volatile storage device 103 is an HDD (Hard Disk Drive) or an SSD (Solid State Drive).
  • the input interface 104 acquires input information from the outside. The input information will be described later.
  • the output interface 105 outputs information to an external device that can be connected to the information processing device 100.
  • the information processing apparatus 100 includes a storage unit 110, an acquisition unit 120, an object recognition processing execution unit 130, a specific unit 140, a method selection unit 150, and an emotion estimation unit 160.
  • the storage unit 110 may be realized as a storage area secured in the volatile storage device 102 or the non-volatile storage device 103.
  • a part or all of the acquisition unit 120, the object recognition processing execution unit 130, the specific unit 140, the method selection unit 150, and the emotion estimation unit 160 may be realized by a processing circuit.
  • a part or all of the acquisition unit 120, the object recognition processing execution unit 130, the specific unit 140, the method selection unit 150, and the emotion estimation unit 160 may be realized as a module of a program executed by the processor 101.
  • the program executed by the processor 101 is also called an emotion estimation program.
  • an emotion estimation program is recorded on a recording medium.
  • the storage unit 110 may store the method table.
  • a plurality of methods are registered in the method table.
  • Each of the multiple methods is a method of estimating emotions.
  • the method table is also referred to as method information.
  • the acquisition unit 120 acquires the input information X1.
  • the input information X1 is information about the user in a certain situation.
  • the input information X1 may be information about the user in a certain situation at a certain point in time. For example, a situation is a situation in which a user is waiting for an elevator, a situation in which a user is driving a car, and the like. Further, the input information X1 may be expressed as information regarding the behavior of the user.
  • the input information X1 includes an image X2, voice information indicating a user's voice, user's biological information, line-of-sight data, motion information, and the like.
  • the input information X1 may include at least an image X2.
  • the input information X1 may be referred to as sensor information.
  • the biometric information contained in the input information X1 can be acquired by using a non-contact sensor or a contact sensor.
  • the non-contact sensor is a camera, a thermography camera, or an exhalation sensor.
  • biological information such as heartbeat and pulse can be obtained based on the blood flow information on the face surface included in the image obtained from the camera.
  • thermography camera biometric information indicating body temperature can be obtained based on the thermography.
  • biometric information which is information about exhalation, is obtained from the exhalation sensor.
  • the contact sensor is a wearable device such as a smart watch. Biological information such as heart rate, pulse, and sweating can be obtained from wearable devices.
  • the contact sensor is embedded in the handle or the seat belt, the heartbeat and the pulse are obtained from the contact sensor as biometric information.
  • the line-of-sight data included in the input information X1 is data indicating a user's attention position in the image X2.
  • line-of-sight data is obtained from an eye tracking sensor.
  • the motion information included in the input information X1 is information indicating the user's operation. Motion information is obtained by motion capture. In addition, motion information is obtained from Kinect (registered trademark) of Microsoft (registered trademark).
  • the object recognition process execution unit 130 executes a process of recognizing an object included in the image X2. Based on the input information X1, the specifying unit 140 specifies an object that the user is paying attention to and a state when the user is paying attention to the object. Further, the specifying unit 140 may specify the object that the user is paying attention to based on the information obtained by the processing and the input information X1.
  • the method selection unit 150 is a method corresponding to the object information and the state information from a plurality of methods based on the object information indicating the specified object, the state information indicating the specified state, and the method table. Select.
  • the emotion estimation unit 160 estimates the user's emotion based on the selected method, the object information, and the state information.
  • FIG. 3 is a flowchart showing an example of processing executed by the information processing apparatus of the first embodiment.
  • the acquisition unit 120 acquires the input information X1.
  • the object recognition processing execution unit 130 acquires the image X2 included in the input information X1.
  • the image X2 is an image obtained by photographing the periphery of the user. It should be noted that the image X2 may include an object that the user is paying attention to, as will be described later. Therefore, the image X2 does not have to include the user. Therefore, the periphery may be considered as a place away from the user or a range including the user.
  • the object recognition process execution unit 130 executes a process of recognizing an object included in the image X2. For example, in the process, general object recognition or specific object recognition is used. By executing the process, the process result X3 is obtained. That is, the processing result X3 is the information obtained by the processing.
  • the processing result X3 indicates the part of the device operated by the user.
  • the part in the case of an elevator, the part is a call button, an elevator door, a floor display, or a floor.
  • the part In the case of a car, the part is a navigation screen or a window.
  • the processing result X3 indicates a physical part of the user.
  • the physical part is the toes.
  • the processing result X3 is an article worn by the user, an article owned by the user, and the like.
  • the item worn by the user is a watch.
  • an article owned by a user is a smartphone.
  • an ID may be associated with the object (specifically, the name of the object) indicated by the processing result X3.
  • the object indicated by the processing result X3 may be associated with information indicating a position in the image X2.
  • the information indicating the position is represented by coordinates.
  • the processing result X3 may include area information indicating the range of the specified object in the image X2.
  • the object recognition processing execution unit 130 may acquire a plurality of images.
  • the plurality of images are images obtained by photographing a user existing in an elevator hall for 30 seconds in units of 1 second.
  • the object recognition process execution unit 130 executes a process of recognizing an object included in each of a plurality of images. That is, the object recognition processing execution unit 130 executes the processing for recognizing an object for each image.
  • continuous processing results can be obtained. For example, 30 consecutive processing results based on 30 images can be obtained.
  • the identification unit 140 identifies an object that the user is paying attention to.
  • the specific process will be described in detail.
  • the identification unit 140 identifies an object that the user is paying attention to based on the processing result X3 and the image X2 that includes the user.
  • the specifying unit 140 specifies the orientation of the user's face, the direction of the user's body, the peculiar movement when the user is paying attention to the object, the posture of the user, and the like, based on the image X2.
  • the specifying unit 140 can specify the direction of the user's face, the direction of the user's body, the peculiar movement, the posture of the user, and the like by using a known technique.
  • the specifying unit 140 identifies an object that the user is paying attention to based on the specified information and the processing result X3.
  • the specific processing will be specifically described.
  • the identification unit 140 identifies an object existing in the orientation of the user's face as an object that the user is paying attention to, based on the processing result X3.
  • the peculiar operation is an operation of looking at a wristwatch, an operation of looking at a smartphone, and the like.
  • the identification unit 140 identifies the wristwatch that the user is paying attention to based on the processing result X3.
  • the specifying unit 140 identifies the object that the user is paying attention to based on the inclination of the neck and the position of the arm indicated by the posture of the user and the processing result X3. Specifically, when the posture of the user indicates that "the arm is raised and the neck is turned diagonally sideways", the identification unit 140 identifies the wristwatch from the processing result X3.
  • the specifying unit 140 identifies the object that the user is paying attention to based on the line-of-sight data and the processing result X3. Specifically, when the object indicated by the processing result X3 is associated with the information indicating the position in the image X2, the specific unit 140 is the position of the object indicated by the processing result X3 and the user's attention position indicated by the line-of-sight data. Based on, identify the object that the user is paying attention to.
  • the identification unit 140 identifies an object that the user is paying attention to based on the voice information. Specifically, the specifying unit 140 analyzes the user's utterance content indicated by the voice information, and identifies the object that the user is paying attention to based on the analysis result. For example, when the input information X1 includes motion information, the specifying unit 140 identifies an object that the user is paying attention to based on the user's neck direction or body direction indicated by the motion information and the processing result X3. ..
  • the specifying unit 140 specifies the state when the user is paying attention to the object based on the input information X1.
  • the specifying unit 140 specifies the state when the user is paying attention to the object based on the input information X1.
  • the state information indicating the state may be referred to as attention behavior information.
  • the "behavior" of the behavioral information of interest may be considered to be a static or dynamic state or reaction of a human or an animal that can be objectively observed from the outside. The reaction also includes physiological phenomena such as eye movement and heartbeat.
  • the state information includes information indicating the frequency of attention, which is the frequency with which the user pays attention to the object, information indicating the posture when the user is paying attention to the object, and biometric information when the user is paying attention to the object. It is a feature amount based on voice information when the user is paying attention to an object.
  • the frequency of attention will be explained.
  • the object that the user is paying attention to is the floor display
  • the specific unit 140 the user pays attention to the floor display within a preset time based on a plurality of images X2 including the user.
  • the time spent or the number of times of attention is calculated, and the frequency of attention is specified based on the calculation result.
  • the specifying unit 140 specifies the posture when the user is paying attention to the object by using the image X2 including the user. Specifically, the specifying unit 140 identifies the posture when the user is paying attention to the object based on the image X2 including the user and the image for specifying the posture. For example, when the image X2 shows a state in which a user existing in the elevator hall is paying attention to the floor display while folding his arms, the specific unit 140 performs template matching between the image X2 and the image showing the arms folded. By doing so, identify the arms folded. When the posture information obtained from the posture detection sensor is included in the input information X1, the specifying unit 140 identifies the posture based on the posture information indicating the posture of the user.
  • the specific unit 140 specifies the posture based on the skeleton information.
  • the specifying unit 140 specifies the posture based on the motion information.
  • biometric information is biometric information included in the input information X1.
  • biometric information includes heart rate, degree of sweating, and the like.
  • the features based on voice information will be explained.
  • features based on voice information include pitch, power, and spectrum.
  • the feature amount based on the voice information may be the content of the user's utterance.
  • the specific unit 140 specifies the object that the user is paying attention to and the state when the user is paying attention.
  • the object information indicating the specified object and the state information indicating the specified state are referred to as attention-related information X4.
  • attention-related information X4 is illustrated.
  • FIG. 4 is a diagram showing an example of attention-related information according to the first embodiment.
  • the “target of interest” indicated by the attention-related information X4 in FIG. 4 indicates an object that the user is paying attention to.
  • the "attention frequency”, "posture”, and “heart rate” indicated by the attention-related information X4 in FIG. 4 indicate the state when the user is paying attention.
  • the "target of interest” may be associated with a plurality of information indicating the state when the user is paying attention.
  • a frame number may be associated with the "target of interest” and the plurality of pieces of information.
  • the frame number is a number of one frame or a number when a plurality of frames are regarded as one frame.
  • the "attention frequency”, “posture”, and “heart rate” are the “attention frequency”, “posture”, and “heart rate” at a preset time. show.
  • the "attention frequency” and “heart rate” at a preset time may be an average value, a maximum value, a minimum value, or a representative value of the values at a preset time.
  • step S14 may be executed before step S13. Further, step S14 may be executed in parallel with step S13.
  • Step S15 The acquisition unit 120 acquires the method table.
  • the acquisition unit 120 acquires the method table from the storage unit 110.
  • the method table may be stored in an external device that can be connected to the information processing device 100.
  • the acquisition unit 120 acquires the method table from the external device.
  • the method selection unit 150 selects a method to be used when estimating emotions based on the attention-related information X4 and the method table. As described above, a plurality of methods are registered in the method table. A plurality of methods will be specifically described.
  • a method of estimating emotions using an expression is registered.
  • a method of estimating emotions using a feature amount and a threshold value used for determining emotions is registered.
  • a method for presuming irritation may be registered when the frequency of attention is 0.8 or more.
  • a method estimated to be normal may be registered in the method table.
  • a method for estimating emotions by weighting a feature amount used for determining emotions or a numerical value indicating emotions is registered.
  • the numerical value indicating emotion may be expressed as the importance of emotion.
  • a method of estimating emotions by weighting the importance of the information indicated by the state information is registered.
  • a method of estimating emotions by weighting the "attention frequency" indicated by the state information is registered.
  • a method of estimating emotions based on preset rules is registered.
  • a method of estimating emotions using a trained model, or a method of estimating emotions using a classifier such as an SVM (Support Vector Machine) or a neural network is registered in the method table.
  • a specific example of the method table is shown.
  • FIG. 5 is a diagram showing an example of the method table of the first embodiment.
  • the method table 111 is stored in the storage unit 110.
  • the method table 111 has items of a method ID, an object of interest, a condition, and a method.
  • An identifier is registered in the item of the method ID.
  • An object of interest is registered in the item of interest.
  • the condition is registered in the condition item.
  • items of three conditions are exemplified.
  • the number of condition items may be one. That is, the number of condition items is not limited to three.
  • the method is registered in the method item.
  • a method of estimating emotions by weighting a feature amount used for determining emotions or a numerical value indicating emotions is registered.
  • the method selection unit 150 includes "floor display” (attention target), "0.8” (attention frequency), “arms folded” (posture), and "80" (heart rate) included in the attention-related information X4. ) Is searched for in the method table 111.
  • the method selection unit 150 selects a method of estimating emotions by weighting "irritability: 1.5" as a weight.
  • the method selection unit 150 may use a value obtained by multiplying, adding, or averaging the plurality of weights as the weight. Further, the method selection unit 150 may use the maximum value or the minimum value of the plurality of weights as weights.
  • the information indicating the selected method is referred to as method information X5.
  • the method information X5 may be a method ID indicating the selected method.
  • the method information X5 may be information that combines the target of interest and the selected method.
  • the method information X5 may be information such as "floor display: irritation: 1.5".
  • Step S16 The emotion estimation unit 160 estimates the user's emotion based on the attention-related information X4 and the method information X5. Further, when the method information X5 includes the method ID, the emotion estimation unit 160 refers to the method table 111 and specifies the method corresponding to the method ID. The emotion estimation unit 160 estimates the user's emotion based on the specified method and the attention-related information X4.
  • the emotion estimation unit 160 is set to “irritability”. Weight "1.5” to the value indicating "irritability”. Further, for example, when the method information X5 indicates a method of estimating an emotion value using a certain value, the emotion estimation unit 160 estimates the emotion value using the attention-related information X4 and the value.
  • the emotion estimation unit 160 has the attention frequency indicated by the attention-related information X4 and “0.8”. Estimate emotions by comparing with. Also, for example, the emotion estimation unit 160 estimates emotions using the rules, equations, trained models, or classifiers used in the method indicated by method information X5. The emotion estimation unit 160 may change the weight or the threshold value used in the method indicated by the method information X5, and estimate the emotion using the changed value.
  • the estimated result is called result information X6.
  • the result information X6 shows emotions, fatigue, stress, the inside of the user, and the like. Further, the result information X6 may indicate emotions such as emotions and emotions numerically. For example, the numerical value indicating emotion is "joy: 0.5".
  • the emotion estimation unit 160 outputs the result information X6. For example, the emotion estimation unit 160 outputs the result information X6 to the display.
  • the emotion estimation unit 160 estimates emotions for each frame number.
  • the emotion estimation unit 160 outputs a plurality of result information X6.
  • the emotion estimation unit 160 may output the maximum value.
  • the emotion estimation unit 160 may output the numerical value of a specific frame number as a representative value.
  • the information processing apparatus 100 estimates emotions based on the attention-related information X4. Specifically, the information processing apparatus 100 estimates emotions based on an object that the user is paying attention to and a state when the user is paying attention. Therefore, the information processing apparatus 100 can improve the estimation accuracy of emotions.
  • the information processing apparatus 100 selects a method to be used when estimating emotions from the method table 111 based on the attention-related information X4. That is, the information processing apparatus 100 selects the most appropriate method when estimating emotions using the information indicated by the attention-related information X4. Then, the information processing apparatus 100 estimates the emotion by the selected method. Therefore, the information processing apparatus 100 can realize highly accurate estimation.
  • the emotion estimation unit 160 may estimate the user's emotion based on the object information, the state information, and a preset method for estimating the emotion.
  • the preset method is a method of estimating emotions using a threshold value, a feature amount used when determining emotions, or a method of weighting emotions with respect to a numerical value indicating emotions. ..
  • the preset method is a method of estimating emotions using a discriminator such as a rule, an expression, a trained model, or an SVM.
  • FIG. 6 is a diagram showing a functional block included in the information processing apparatus of the second embodiment.
  • the configuration of FIG. 6, which is the same as the configuration shown in FIG. 1, has the same reference numerals as those shown in FIG.
  • the information processing device 100a has an emotion estimation unit 160a.
  • the acquisition unit 120 acquires the input information X1.
  • the input information X1 is information about the user within a preset time in a certain situation.
  • the input information X1 includes a plurality of images X2 obtained by shooting within a preset time, voice information indicating a user's voice within a preset time, and a preset time. It includes biometric information of the user in the room, line-of-sight data within a preset time, motion information within a preset time, and the like.
  • the input information X1 is time-series data indicating feature quantities such as image X2, voice information, biological information, line-of-sight data, and motion information.
  • the function of the emotion estimation unit 160a will be described later.
  • FIG. 7 is a flowchart showing an example of processing executed by the information processing apparatus according to the second embodiment.
  • the process of FIG. 7 is different from the process of FIG. 3 in that step S16a is executed. Therefore, in FIG. 7, step S16a will be described.
  • the same number as the step number in FIG. 3 is assigned, and the description of the process will be omitted.
  • the emotion estimation unit 160a estimates the user's emotion based on the input information X1 and the method information X5. For example, when the method information X5 indicates a method of estimating an emotion using a feature amount and a preset threshold value, the emotion estimation unit 160a calculates an emotion based on the feature amount indicated by the input information X1 and the threshold value. presume. Further, for example, the emotion estimation unit 160a estimates emotions using the rules, equations, trained models, or discriminators used in the method indicated by the method information X5.
  • the estimated result is called result information X6.
  • the result information X6 shows emotions, fatigue, stress, the inside of the user, and the like. Further, the result information X6 may indicate emotions such as emotions and emotions numerically.
  • the emotion estimation unit 160a outputs the result information X6.
  • the emotion was estimated using the attention-related information X4. That is, in the first embodiment, the emotion is estimated using the information at the time when the user is paying attention.
  • the information processing apparatus 100a estimates emotions using the input information X1. Therefore, the information processing apparatus 100a can estimate the user's emotion within a preset time.
  • FIG. 8 is a diagram showing a functional block included in the information processing apparatus of the third embodiment.
  • the configuration of FIG. 8, which is the same as the configuration shown in FIG. 6, has the same reference numerals as those shown in FIG.
  • the information processing apparatus 100b has an acquisition unit 120b, a method selection unit 150b, and a waiting time specifying unit 170.
  • a part or all of the waiting time specifying unit 170 may be realized by a processing circuit.
  • a part or all of the waiting time specifying unit 170 may be realized as a module of a program executed by the processor 101.
  • the acquisition unit 120b acquires the input information X1 and the device information X7.
  • the input information X1 is the input information X1 described in the second embodiment.
  • the device information X7 is information about the device used by the user.
  • the device information X7 includes time information when the elevator call button is pressed, elevator floor display information, information indicating the current position of the elevator cage, and the like.
  • the device information X7 is information indicating that a button of a navigation device provided in a car has been pressed, information on a screen displayed on the navigation device, information on voice guidance output by the navigation device, and the like. be.
  • the functions of the method selection unit 150b and the waiting time specifying unit 170 will be described later.
  • FIG. 9 is a flowchart showing an example of processing executed by the information processing apparatus according to the third embodiment.
  • the process of FIG. 9 differs from the process of FIG. 7 in that steps S11a, 11b, and 15a are executed. Therefore, in FIG. 9, steps S11a, 11b, and 15a will be described. For the other steps in FIG. 9, the same number as the step number in FIG. 7 is assigned, and the description of the process will be omitted.
  • Step S11a The acquisition unit 120b acquires the input information X1 and the device information X7.
  • Step S11b The waiting time specifying unit 170 specifies, as a waiting time, the time from when the user operates the device until a response to the operation is returned, based on the device information X7. The method of specifying the waiting time will be specifically described.
  • the device information X7 includes a pressing time, which is the time when the user presses the elevator call button, and a time when the elevator basket arrives and the door is opened.
  • the waiting time specifying unit 170 specifies the waiting time based on the pressing time and the time when the door is opened.
  • the waiting time specifying unit 170 determines the waiting time based on the pressing time and the time when step S11b is executed (that is, the current time). It may be specified.
  • the device information X7 is assumed to include an operation time which is the time when the navigation device is operated and an execution time which is the time when the response to the operation is executed.
  • the waiting time specifying unit 170 specifies the waiting time based on the operation time and the execution time.
  • the device information X7 includes an input time, which is the time when the voice information of the voice emitted by the user is input to the navigation device, and an execution time, which is the time when the response to the voice is executed. And.
  • the waiting time specifying unit 170 specifies the waiting time based on the input time and the execution time.
  • the information indicating the specified waiting time is referred to as waiting time information X8. Further, the waiting time information X8 may be expressed by the length of time. For example, the length of time is "long" or "short". Note that step S11b may be executed at any timing as long as it is before step S15a is executed.
  • Step S15a The acquisition unit 120b acquires the method table 111.
  • the method selection unit 150b selects a method to be used when estimating emotions based on the attention-related information X4, the waiting time information X8, and the method table 111. Specifically, the method selection unit 150b has a waiting time and an object information from a plurality of methods shown by the method table 111 based on the waiting time, the object information, the state information, and the method table 111 shown by the waiting time information X8. And select the method corresponding to the situation information.
  • the method table 111 is illustrated.
  • FIG. 10 is a diagram showing an example of the method table of the third embodiment.
  • FIG. 10 shows that the method table 111 has an item of the condition “waiting time”.
  • the method selection unit 150b includes "floor display” (attention target), "0.8” (attention frequency), “arms folded” (posture), and "80" (heart rate) included in the attention-related information X4. ) And the record that matches the conditions of "150 seconds" (waiting time) included in the waiting time information X8 is searched from the method table 111.
  • the method selection unit 150b selects a method of estimating emotions by weighting "irritability: 1.5" as a weight.
  • the information indicating the selected method is referred to as method information X5.
  • the method information X5 may be a method ID indicating the selected method.
  • the method information X5 may be information that combines the target of interest and the selected method.
  • the method information X5 may be information such as "floor display: irritation: 1.5".
  • the information processing apparatus 100b selects a method to be used for estimating emotions from the method table 111 based on the attention-related information X4 and the waiting time information X8. That is, the information processing apparatus 100b selects a method in which the waiting time is taken into consideration. Then, the information processing apparatus 100b estimates the emotion by the selected method. Therefore, the information processing apparatus 100b can realize highly accurate estimation.
  • the emotion estimation unit 160a estimates the emotion by the method selected by the method selection unit 150b.
  • the emotion estimation unit 160 estimates the emotion by the method selected by the method selection unit 150b. That is, in the modified example of the third embodiment, step S16 is executed after step S15a.
  • the information processing apparatus 100b estimates the emotion based on the attention-related information X4. Specifically, the information processing apparatus 100b estimates emotions based on an object that the user is paying attention to and a state when the user is paying attention. Therefore, the information processing apparatus 100b can improve the estimation accuracy of emotions.
  • the information processing apparatus 100b selects a method to be used when estimating emotions from the method table 111 based on the attention-related information X4 and the waiting time information X8. That is, the information processing apparatus 100b selects a method in which the waiting time is taken into consideration. Then, the information processing apparatus 100b estimates the emotion by the selected method. Therefore, the information processing apparatus 100b can realize highly accurate estimation.
  • Embodiment 4 Next, the fourth embodiment will be described.
  • the matters different from the second embodiment will be mainly described.
  • the description of the matters common to the second embodiment will be omitted.
  • FIGS. 6 and 7 are referred to.
  • FIG. 11 is a diagram showing a functional block included in the information processing apparatus of the fourth embodiment.
  • the configuration of FIG. 11, which is the same as the configuration shown in FIG. 6, has the same reference numerals as those shown in FIG.
  • the information processing apparatus 100c has a method selection unit 150c and an identification unit 180.
  • a part or all of the identification unit 180 may be realized by a processing circuit.
  • a part or all of the identification unit 180 may be realized as a module of a program executed by the processor 101.
  • the storage unit 110 may store a plurality of method tables. The multiple method tables will be described later. The function of the method selection unit 150c will be described later.
  • the identification unit 180 identifies at least one of a user and a user type based on the input information X1.
  • FIG. 12 is a flowchart showing an example of processing executed by the information processing apparatus according to the fourth embodiment.
  • the process of FIG. 12 differs from the process of FIG. 7 in that steps S11c and 15b are executed. Therefore, in FIG. 12, steps S11c and 15b will be described. For the other steps in FIG. 12, the same number as the step number in FIG. 7 is assigned, and the description of the process will be omitted.
  • Step S11c The identification unit 180 identifies at least one of the user and the user type based on the input information X1.
  • the identification unit 180 identifies the user by using the image X2. Specifically, the identification unit 180 identifies the user based on the image X2 and known technology. For example, the known technique is general object recognition or specific object recognition. Further, for example, when the input information X1 includes motion information, the identification unit 180 identifies the user based on the habit of the user specified from the motion information. Specifically, the identification unit 180 identifies the user based on the characteristic motion indicated by the motion information and the information indicating the habit of the motion that can identify an individual. Information indicating the habit may be stored in the storage unit 110.
  • the identification unit 180 identifies the user by using the feature amount based on the voice information. Further, for example, when biometric information such as a heartbeat, an iris, and a fingerprint is included in the input information X1, the identification unit 180 identifies the user by using the biometric information. Specifically, the identification unit 180 identifies the user based on biometric information and known technology.
  • the identification unit 180 uses the identification unit 180 based on the change in the facial expression of the user indicated by the image X2 or the facial expression of the user specified from the plurality of images X2. Identify the type of. For example, the types identified are angry types, mild types, and so on. Further, for example, when the motion information is included in the input information X1, the identification unit 180 identifies the type of the user based on the characteristic operation indicated by the motion information. Further, for example, when the voice information is included in the input information X1, the identification unit 180 identifies the type of the user based on the way of speaking specified from the voice information.
  • the identification unit 180 identifies the type of the user by using the biometric information. Specifically, the identification unit 180 identifies the type of user based on the feature amount indicated by the biometric information. For example, the type to be identified is a type that easily sweats.
  • the identification unit 180 uses the identified user and the user type. You may identify the type of user based on the information.
  • the identification unit 180 selects at least one of the user and the type of the user based on the information. Identify.
  • the identified information is referred to as identification information X9.
  • the identification information X9 is information indicating at least one of the user's name and type. Further, the identification information X9 may be information indicating at least one of the identified user ID and the identified type ID. Note that step S11c may be executed at any timing as long as it is before step S15b is executed.
  • Step S15b The acquisition unit 120 acquires a method table corresponding to the user or type indicated by the identification information X9.
  • the acquired method table is stored in the storage unit 110 or an external device.
  • the storage unit 110 stores the method tables 111a1 to 111a3 corresponding to the individual and the method tables 111b1 to 111b3 corresponding to the type.
  • the acquisition unit 120 acquires the method table 111a1 corresponding to the user U1 indicated by the identification information X9 from the storage unit 110.
  • a method suitable for estimating the emotion of the user U1 is registered in the method item of the method table 111a1.
  • a method suitable for estimating the user's emotion indicated by the identification information X9 is registered in each of the method tables 111a1 to 111a3.
  • the acquisition unit 120 acquires the method table 111b1 corresponding to the type TY1 indicated by the identification information X9 from the storage unit 110.
  • a method suitable for estimating the emotion of the user of type TY1 is registered in the method item of the method table 111b1.
  • a method suitable for estimating the emotion of the type of user indicated by the identification information X9 is registered in each of the method tables 111b1 to 111b3.
  • the method selection unit 150c selects a method to be used when estimating emotions based on the attention-related information X4 and the acquired method table.
  • the selection process is the same as step S15 of the first embodiment. Therefore, the description of the selection process will be omitted.
  • the acquisition unit 120 acquires the method table 111.
  • the method selection unit 150c selects a method to be used when estimating emotions based on the attention-related information X4, the identification information X9, and the method table 111.
  • the method table 111 is illustrated.
  • FIG. 13 is a diagram showing an example of the method table of the fourth embodiment.
  • FIG. 13 shows that the method table 111 has a condition "user" item and a condition "type” item.
  • the method selection unit 150c includes "floor display” (attention target), “0.8” (attention frequency), “arms folded” (posture), and “80” (heart rate) included in the attention-related information X4. ), And the records that match the conditions of "Usr 1 " (user) and “angry” (type) included in the identification information X9 are searched from the method table 111.
  • the method selection unit 150c selects a method of estimating emotions by weighting "irritability: 1.5" as a weight.
  • the information indicating the selected method is referred to as method information X5.
  • the method information X5 may be a method ID indicating the selected method.
  • the method information X5 may be information that combines the target of interest and the selected method.
  • the method information X5 may be information such as "floor display: irritation: 1.5".
  • the information processing apparatus 100c selects a method suitable for estimating the emotion of the user by acquiring the method table corresponding to the user indicated by the identification information X9. Then, the information processing apparatus 100c estimates the emotion by the selected method. Therefore, the information processing apparatus 100c can realize highly accurate estimation. Further, the information processing apparatus 100c selects a method suitable for estimating the emotion of the user of the type by acquiring the method table corresponding to the type indicated by the identification information X9. Then, the information processing apparatus 100c estimates the emotion by the selected method. Therefore, the information processing apparatus 100c can realize highly accurate estimation.
  • the emotion estimation unit 160a estimates the emotion by the method selected by the method selection unit 150c.
  • the emotion estimation unit 160 estimates the emotion by the method selected by the method selection unit 150c. That is, in the modified example of the fourth embodiment, step S16 is executed after step S15b.
  • the information processing apparatus 100c estimates the emotion based on the attention-related information X4. Specifically, the information processing apparatus 100c estimates emotions based on an object that the user is paying attention to and a state when the user is paying attention. Therefore, the information processing apparatus 100c can improve the estimation accuracy of emotions.
  • the information processing apparatus 100c selects a method suitable for estimating the emotion of the user by acquiring the method table corresponding to the user indicated by the identification information X9. Then, the information processing apparatus 100c estimates the emotion by the selected method. Therefore, the information processing apparatus 100c can realize highly accurate estimation. Further, the information processing apparatus 100c selects a method suitable for estimating the emotion of the user of the type by acquiring the method table corresponding to the type indicated by the identification information X9. Then, the information processing apparatus 100c estimates the emotion by the selected method. Therefore, the information processing apparatus 100c can realize highly accurate estimation.
  • 100, 100a, 100b, 100c information processing device 101 processor, 102 volatile storage device, 103 non-volatile storage device, 104 input interface, 105 output interface, 110 storage unit, 111,111a1 to 111a3, 111b1 to 111b3 method table, 120, 120b acquisition unit, 130 object recognition processing execution unit, 140 specific unit, 150, 150b, 150c method selection unit, 160, 160a emotion estimation unit, 170 waiting time identification unit, 180 identification unit.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Security & Cryptography (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Psychiatry (AREA)
  • Medical Informatics (AREA)
  • Hospice & Palliative Care (AREA)
  • Developmental Disabilities (AREA)
  • Psychology (AREA)
  • Social Psychology (AREA)
  • Child & Adolescent Psychology (AREA)
  • Biophysics (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Educational Technology (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Ophthalmology & Optometry (AREA)
  • User Interface Of Digital Computer (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • Image Analysis (AREA)
PCT/JP2020/019906 2020-05-20 2020-05-20 情報処理装置、感情推定方法、及び感情推定プログラム Ceased WO2021234845A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
JP2022524747A JP7106041B2 (ja) 2020-05-20 2020-05-20 情報処理装置、感情推定方法、及び感情推定プログラム
DE112020006934.6T DE112020006934T5 (de) 2020-05-20 2020-05-20 Informationsverarbeitungsvorrichtung, verfahren zur emotionseinschätzung und programm zur gefühlseinschätzung
PCT/JP2020/019906 WO2021234845A1 (ja) 2020-05-20 2020-05-20 情報処理装置、感情推定方法、及び感情推定プログラム
CN202080100074.6A CN115516499A (zh) 2020-05-20 2020-05-20 信息处理装置、情绪估计方法和情绪估计程序
US17/894,289 US12380731B2 (en) 2020-05-20 2022-08-24 Information processing device, and emotion estimation method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/019906 WO2021234845A1 (ja) 2020-05-20 2020-05-20 情報処理装置、感情推定方法、及び感情推定プログラム

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/894,289 Continuation US12380731B2 (en) 2020-05-20 2022-08-24 Information processing device, and emotion estimation method

Publications (1)

Publication Number Publication Date
WO2021234845A1 true WO2021234845A1 (ja) 2021-11-25

Family

ID=78708311

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/019906 Ceased WO2021234845A1 (ja) 2020-05-20 2020-05-20 情報処理装置、感情推定方法、及び感情推定プログラム

Country Status (5)

Country Link
US (1) US12380731B2 (https=)
JP (1) JP7106041B2 (https=)
CN (1) CN115516499A (https=)
DE (1) DE112020006934T5 (https=)
WO (1) WO2021234845A1 (https=)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022124014A1 (ja) * 2020-12-07 2022-06-16 ソニーグループ株式会社 情報処理装置、データの生成方法、グループ化モデルの生成方法、グループ化モデルの学習方法、情動推定モデルの生成方法及びグループ化用ユーザ情報の生成方法
WO2023188698A1 (ja) * 2022-03-31 2023-10-05 株式会社島津製作所 評価方法、評価用装置、およびプログラム

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06251273A (ja) * 1993-02-23 1994-09-09 Mitsubishi Electric Corp 運転者状態判定装置
JP2005348872A (ja) * 2004-06-09 2005-12-22 Nippon Hoso Kyokai <Nhk> 感情推定装置及び感情推定プログラム
JP2017201499A (ja) * 2015-10-08 2017-11-09 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ 情報提示装置の制御方法、及び、情報提示装置

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4627152B2 (ja) * 2004-06-01 2011-02-09 三星電子株式会社 危機監視システム
US20100086204A1 (en) * 2008-10-03 2010-04-08 Sony Ericsson Mobile Communications Ab System and method for capturing an emotional characteristic of a user
JP2012059107A (ja) * 2010-09-10 2012-03-22 Nec Corp 感情推定装置、感情推定方法およびプログラム
TW201220216A (en) * 2010-11-15 2012-05-16 Hon Hai Prec Ind Co Ltd System and method for detecting human emotion and appeasing human emotion
US20130243270A1 (en) * 2012-03-16 2013-09-19 Gila Kamhi System and method for dynamic adaption of media based on implicit user input and behavior
CN107000982B (zh) * 2014-12-03 2020-06-09 因温特奥股份公司 与电梯交替交互的系统和方法
JP2019047234A (ja) * 2017-08-31 2019-03-22 ソニーセミコンダクタソリューションズ株式会社 情報処理装置、情報処理方法、およびプログラム
JP2019101775A (ja) 2017-12-04 2019-06-24 京セラドキュメントソリューションズ株式会社 ユーザー操作支援装置及びユーザー操作支援プログラム
CN110147729A (zh) * 2019-04-16 2019-08-20 深圳壹账通智能科技有限公司 用户情绪识别方法、装置、计算机设备及存储介质
EP3800605B1 (en) * 2019-10-03 2025-11-19 Tata Consultancy Services Limited Methods and systems for predicting wait time of queues at service area

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06251273A (ja) * 1993-02-23 1994-09-09 Mitsubishi Electric Corp 運転者状態判定装置
JP2005348872A (ja) * 2004-06-09 2005-12-22 Nippon Hoso Kyokai <Nhk> 感情推定装置及び感情推定プログラム
JP2017201499A (ja) * 2015-10-08 2017-11-09 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ 情報提示装置の制御方法、及び、情報提示装置

Also Published As

Publication number Publication date
US20220415086A1 (en) 2022-12-29
JP7106041B2 (ja) 2022-07-25
US12380731B2 (en) 2025-08-05
DE112020006934T5 (de) 2023-01-19
CN115516499A (zh) 2022-12-23
JPWO2021234845A1 (https=) 2021-11-25

Similar Documents

Publication Publication Date Title
JP7083809B2 (ja) プライバシーの保護を伴う人物の識別しおよび／または痛み、疲労、気分、および意図の識別および定量化のためのシステムおよび方法
US10380603B2 (en) Assessing personality and mood characteristics of a customer to enhance customer satisfaction and improve chances of a sale
US10262196B2 (en) System and method for predicting neurological disorders
JP7229174B2 (ja) 人識別システム及び方法
US20190323895A1 (en) System and method for human temperature regression using multiple structures
US20210319585A1 (en) Method and system for gaze estimation
JP2020533701A (ja) 対象者の識別のためのカメラ及び画像校正
US9408562B2 (en) Pet medical checkup device, pet medical checkup method, and non-transitory computer readable recording medium storing program
JP2019522300A (ja) 精神障害の療法のためのモバイルおよびウェアラブルビデオ捕捉およびフィードバックプラットフォーム
JP6946551B2 (ja) 映像アライメント方法及びその装置
JP7278972B2 (ja) 表情解析技術を用いた商品に対するモニタの反応を評価するための情報処理装置、情報処理システム、情報処理方法、及び、プログラム
US12380731B2 (en) Information processing device, and emotion estimation method
CN116434296A (zh) 一种实时人脸识别监控行为方法、装置、设备和介质
US20060093189A1 (en) Movement identification apparatus
KR102239015B1 (ko) 영상 얼라인먼트 방법 및 그 장치
CN114677733A (zh) 信息预警方法、系统、装置、终端设备、介质及程序产品
Hsu et al. Extraction of visual facial features for health management
JP7816493B2 (ja) 商品検索装置、商品検索方法、及び、プログラム
Khargonkar et al. Dominant Hand detection using Base Smartphone sensors
Patil et al. Analysis of Eye-Gaze Estimation Systems for different approaches of Depression Detection
JPWO2021024372A5 (https=)

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20937003

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022524747

Country of ref document: JP

Kind code of ref document: A

122 Ep: pct application non-entry in european phase

Ref document number: 20937003

Country of ref document: EP

Kind code of ref document: A1