WO2022024194A1 - 感情解析システム - Google Patents

感情解析システム Download PDF

Info

Publication number
WO2022024194A1
WO2022024194A1 PCT/JP2020/028748 JP2020028748W WO2022024194A1 WO 2022024194 A1 WO2022024194 A1 WO 2022024194A1 JP 2020028748 W JP2020028748 W JP 2020028748W WO 2022024194 A1 WO2022024194 A1 WO 2022024194A1
Authority
WO
WIPO (PCT)
Prior art keywords
biological reaction
unit
analysis
change
analyzed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/028748
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
渉三 神谷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Imbesideyou Inc
Original Assignee
Imbesideyou Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Imbesideyou Inc filed Critical Imbesideyou Inc
Priority to PCT/JP2020/028748 priority Critical patent/WO2022024194A1/ja
Priority to JP2022540269A priority patent/JP7165463B2/ja
Priority to PCT/JP2021/027452 priority patent/WO2022024956A1/ja
Publication of WO2022024194A1 publication Critical patent/WO2022024194A1/ja
Priority to JP2022166394A priority patent/JP2023015068A/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion

Definitions

  • the present invention relates to an emotion analysis system.
  • Patent Document 1 a technique for analyzing emotions received by others in response to a speaker's remark is known (see, for example, Patent Document 1).
  • the emotion analysis system of the present invention analyzes specific emotions different from others for the analysis target person among the plurality of people in an environment where an online session is performed by a plurality of people. ..
  • the emotion analysis system of the present embodiment is an environment in which an online session is held by a plurality of people, and the specific emotions (feelings that occur for oneself or another person's words and actions) are different from those of the other person for the analysis target person among the plurality of people. It is a system that analyzes (such as comfort / discomfort or its degree).
  • An online session is, for example, an online conference, an online class, an online chat, etc., in which terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between the terminals through the server. It is designed to be able to communicate.
  • the moving image handled in the online session includes the face image and voice of the user who uses the terminal.
  • the moving image also includes an image such as a material shared and viewed by a plurality of users. It is possible to switch between the face image and the material image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the material image at the same time. Further, it is possible to display the image of one of a plurality of people on the full screen, or to display the image of a part or all of the users on a small screen.
  • the analysis target person can be specified, for example, by the analysis target person himself (designating himself as the analysis target person).
  • the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the organizer) to specify any user as the analysis target.
  • Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes.
  • the organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.
  • FIG. 1 is a block diagram showing a functional configuration example of the emotion analysis system according to the present embodiment.
  • the emotion analysis system of the present embodiment has a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiarity determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification as functional configurations.
  • the part 16 is provided.
  • Each of the above functional blocks 11 to 16 can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in the server device, for example.
  • DSP Digital Signal Processor
  • each of the above functional blocks 11 to 16 is actually configured to include a computer CPU, RAM, ROM, etc., and is a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. Is realized by the operation of.
  • the moving image acquisition unit 11 acquires a moving image obtained by shooting a plurality of people (multiple users) with a camera provided in each terminal during an online session. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires the moving image from each terminal, including the moving image being displayed on each terminal and the moving image being hidden.
  • the biological reaction analysis unit 12 analyzes changes in the biological reaction of each of a plurality of persons based on the moving image (whether or not it is displayed on the screen) acquired by the moving image acquisition unit 11.
  • the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and a voice, and analyzes changes in the biological reaction from each.
  • the biological reaction analysis unit 12 analyzes the user's face image using the frame image separated from the moving image acquired by the moving image acquisition unit 11, and thereby at least one of the facial expression, the line of sight, the pulse, and the movement of the face. Analyze changes in biological reactions related to one. In addition, the biological reaction analysis unit 12 analyzes changes in the biological reaction regarding at least one of the user's speech content and voice quality by analyzing the voice separated from the moving image acquired by the moving image acquisition unit 11.
  • the biological reaction analysis unit 12 calculates a biological reaction index value that reflects the content of the change in the biological reaction by quantifying the change in the biological reaction according to a predetermined standard.
  • Analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is specified from the frame image, and the specified facial expressions are classified into a plurality of types according to an image analysis model trained in advance by machine learning. Then, based on the classification result, it is analyzed whether a positive facial expression change occurs between consecutive frame images, a negative facial expression change occurs, and how large the facial expression change occurs. The facial expression change index value according to the analysis result is output.
  • Analysis of changes in the line of sight is performed, for example, as follows. That is, for each frame image, the area of the eyes is specified from the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the line of sight is large or small, and whether the movement is frequent or infrequent. The change in the line of sight is also related to the degree of concentration of the user.
  • the biological reaction analysis unit 12 outputs the line-of-sight change index value according to the analysis result of the line-of-sight change.
  • Analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified from the frame image. Then, using a trained image analysis model that captures the numerical value of the face color information (G in RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform showing the change in color information is formed, and the pulse is specified from this waveform. When a person is nervous, the pulse becomes faster, and when he / she feels calm, the pulse becomes slower.
  • the biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.
  • Analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the area of the face is specified from the frame image, and the orientation of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the face is large or small, and whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be combined and analyzed. For example, it may be possible to analyze whether the speaker's face being displayed is viewed straight, whether the speaker is viewed with an upper eye or a lower eye, or whether the speaker is viewed from an angle.
  • the biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.
  • the content of the statement is analyzed as follows, for example. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing a known voice recognition process on the voice for a specified time (for example, a time of about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words unnecessary for expressing conversation such as particles and acronyms are removed. Then, the remaining words are vectorized, and whether a positive emotional change is occurring, a negative emotional change is occurring, and how large the emotional change is occurring is analyzed, and the analysis result is used. Outputs the statement content index value.
  • Voice quality analysis is performed as follows, for example. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change is occurring, a negative voice quality change is occurring, and how loud the voice quality change is occurring, and according to the analysis result. Outputs the voice quality change index value.
  • the biological reaction analysis unit 12 uses at least one of the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value calculated as described above.
  • the biological reaction index value is calculated.
  • the biological reaction index value is calculated by weighting the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value.
  • the peculiarity determination unit 13 determines whether or not the change in the biological reaction analyzed for the analysis target person is specific to the change in the biological reaction analyzed for the other person other than the analysis target person. In the present embodiment, the peculiarity determination unit 13 compares the changes in the biological reaction analyzed for the analysis target person with those of others based on the biological reaction index values calculated for each of the plurality of users by the biological reaction analysis unit 12. To determine whether it is specific or not.
  • the peculiarity determination unit 13 calculates the variance of the biological reaction index value calculated for each of a plurality of persons by the biological reaction analysis unit 12, and compares the variance with the biological reaction index value calculated for the analysis target person. It is determined whether or not the change in the biological reaction analyzed for the person to be analyzed is specific compared to the other person.
  • the following three patterns can be considered as cases where the changes in the biological reaction analyzed for the person to be analyzed are more specific than those of others.
  • the first is the case where a particularly large change in the biological reaction has not occurred for the other person, but a relatively large change in the biological reaction has occurred for the person to be analyzed.
  • the second is the case where a particularly large change in the biological reaction has not occurred in the subject to be analyzed, but a relatively large change in the biological reaction has occurred in the other person.
  • the third is the case where a relatively large change in the biological reaction occurs in both the analysis target person and the other person, but the content of the change is different between the analysis target person and the other person.
  • the related event identification unit 14 identifies an event occurring with respect to at least one of the analysis subject, another person, and the environment when a change in the biological reaction determined to be specific by the peculiarity determination unit 13 occurs. .. For example, the related event identification unit 14 identifies the behavior of the analysis target person himself / herself from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the behavior of another person from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the environment when a specific change in the biological reaction occurs for the analysis target person from the moving image. The environment is, for example, a shared material displayed on the screen, an environment reflected in the background of the person to be analyzed, and the like.
  • the clustering unit 15 includes changes in biological reactions determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of eyes, pulse, facial movement, speech content, and voice quality) and the peculiarity.
  • the degree of correlation with the event (event specified by the related event identification unit 14) that occurs when a change in the biological reaction occurs is analyzed, and it is determined that the correlation is above a certain level.
  • the clustering unit 15 clusters the analysis target person or the event in any of a plurality of pre-segmented classifications according to the content of the event, the degree of negativeness, the magnitude of the correlation, and the like.
  • the clustering unit 15 clusters the analysis target person or the event in any of a plurality of pre-segmented classifications according to the content of the event, the degree of positiveness, the magnitude of the correlation, and the like.
  • the analysis result notification unit 16 determines at least one of the changes in the biological reaction determined to be specific by the peculiarity determination unit 13, the event specified by the related event identification unit 14, and the classification clustered by the clustering unit 15. , Notify the designated person of the analysis target (analysis target person or the organizer of the online session).
  • the analysis result notification unit 16 analyzes the analysis target as an event that occurs when a specific change in biological reaction occurs in the analysis target person (any of the above-mentioned three patterns; the same applies hereinafter). Notify the person to be analyzed of the person's own words and actions. As a result, the person to be analyzed can grasp that he / she has different emotions from others when he / she makes a certain word or action. At this time, the change of the specific biological reaction specified for the analysis target person may also be notified to the analysis target person. Further, the change in the biological reaction of the other person to be compared may be further notified to the analysis target person.
  • the analysis result notification unit 16 is the organizer of the online session, in which the event occurring when a specific change in the biological reaction of the analysis target person is different from that of others occurs together with the specific change in the biological reaction. Notify to. This allows the organizer of the online session to know what kind of phenomenon influences what kind of emotional change as a phenomenon peculiar to the designated analysis target person. Then, it becomes possible to take appropriate measures for the analysis target person according to the grasped contents.
  • the analysis result notification unit 16 notifies the organizer of the online session of the event occurring when the analysis target person has a specific change in biological reaction different from that of others or the clustering result of the analysis target person. do.
  • the organizer of the online session can grasp the tendency of the behavior peculiar to the analysis target person and predict the behavior or state that may occur in the future, depending on which classification the specified analysis target person is clustered into. be able to. Then, it becomes possible to take appropriate measures for the analysis target person.
  • the biological reaction index value is calculated by quantifying the change in the biological reaction according to a predetermined standard, and the analysis target person is analyzed based on the biological reaction index value calculated for each of the plurality of persons.
  • An example of determining whether or not a change in a biological reaction has been made is specific compared to another person has been described, but the present invention is not limited to this example. For example, it may be as follows.
  • the biological reaction analysis unit 12 analyzes the movement of the line of sight for each of the plurality of people and generates a heat map showing the direction of the line of sight.
  • the change in the biological reaction analyzed for the analysis target person is measured by the comparison between the heat map generated for the analysis target person by the biological reaction analysis unit 12 and the heat map generated for the other person. It is determined whether or not it is specific by comparing with the change in the biological reaction analyzed for.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • Image Analysis (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • User Interface Of Digital Computer (AREA)
PCT/JP2020/028748 2020-07-27 2020-07-27 感情解析システム Ceased WO2022024194A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
PCT/JP2020/028748 WO2022024194A1 (ja) 2020-07-27 2020-07-27 感情解析システム
JP2022540269A JP7165463B2 (ja) 2020-07-27 2021-07-26 感情解析システムおよび感情解析装置
PCT/JP2021/027452 WO2022024956A1 (ja) 2020-07-27 2021-07-26 感情解析システムおよび感情解析装置
JP2022166394A JP2023015068A (ja) 2020-07-27 2022-10-17 感情解析システムおよび感情解析装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/028748 WO2022024194A1 (ja) 2020-07-27 2020-07-27 感情解析システム

Publications (1)

Publication Number Publication Date
WO2022024194A1 true WO2022024194A1 (ja) 2022-02-03

Family

ID=80035649

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/JP2020/028748 Ceased WO2022024194A1 (ja) 2020-07-27 2020-07-27 感情解析システム
PCT/JP2021/027452 Ceased WO2022024956A1 (ja) 2020-07-27 2021-07-26 感情解析システムおよび感情解析装置

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/027452 Ceased WO2022024956A1 (ja) 2020-07-27 2021-07-26 感情解析システムおよび感情解析装置

Country Status (2)

Country Link
JP (2) JP7165463B2 (https=)
WO (2) WO2022024194A1 (https=)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022024194A1 (ja) * 2020-07-27 2022-02-03 株式会社I’mbesideyou 感情解析システム
CN115982473B (zh) * 2023-03-21 2023-06-23 环球数科集团有限公司 一种基于aigc的舆情分析编排系统
WO2026013768A1 (ja) * 2024-07-09 2026-01-15 Ｎｔｔ株式会社 仮想現実体験制御装置、方法およびプログラム

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015075908A (ja) * 2013-10-09 2015-04-20 日本電信電話株式会社 感情情報表示制御装置、その方法及びプログラム
JP2018068618A (ja) * 2016-10-28 2018-05-10 株式会社東芝 感情推定装置、感情推定方法、感情推定プログラム、および感情カウントシステム
JP2018121752A (ja) * 2017-01-30 2018-08-09 国立大学法人 東京大学 画像解析装置、画像解析方法、および画像解析プログラム
JP2019058625A (ja) * 2017-09-26 2019-04-18 株式会社エモスタ 感情読み取り装置及び感情解析方法
JP2020109578A (ja) * 2019-01-07 2020-07-16 本田技研工業株式会社 情報処理装置及びプログラム

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022024194A1 (ja) * 2020-07-27 2022-02-03 株式会社I’mbesideyou 感情解析システム

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2015075908A (ja) * 2013-10-09 2015-04-20 日本電信電話株式会社 感情情報表示制御装置、その方法及びプログラム
JP2018068618A (ja) * 2016-10-28 2018-05-10 株式会社東芝 感情推定装置、感情推定方法、感情推定プログラム、および感情カウントシステム
JP2018121752A (ja) * 2017-01-30 2018-08-09 国立大学法人 東京大学 画像解析装置、画像解析方法、および画像解析プログラム
JP2019058625A (ja) * 2017-09-26 2019-04-18 株式会社エモスタ 感情読み取り装置及び感情解析方法
JP2020109578A (ja) * 2019-01-07 2020-07-16 本田技研工業株式会社 情報処理装置及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
SONEDA, YUSUKE: "Designing and Building an Online Meeting Quality Evaluation System", PROCEEDINGS (3) OF THE 81ST NATIONAL CONVENTION OF THE IPSJ (2019, 28 February 2019 (2019-02-28), pages 3 - 365 -3-366 *

Also Published As

Publication number Publication date
JPWO2022024956A1 (https=) 2022-02-03
JP7165463B2 (ja) 2022-11-04
WO2022024956A1 (ja) 2022-02-03
JP2023015068A (ja) 2023-01-31

Similar Documents

Publication Publication Date Title
JP2023015068A (ja) 感情解析システムおよび感情解析装置
WO2022024354A1 (ja) 反応解析システム
WO2022024355A1 (ja) 感情解析システム
JP2025089509A (ja) ビデオミーティング評価システム及びビデオミーティング評価サーバ
JP7553128B2 (ja) 感情解析システム
WO2022024353A1 (ja) 反応解析システム
JP7734987B2 (ja) 動画像分析システム
JP7730575B2 (ja) 動画像分析システム
JP7629226B2 (ja) ビデオミーティング評価端末、ビデオミーティング評価システム及びビデオミーティング評価プログラム
JP7705176B2 (ja) 動画像分析システム
JP7668017B2 (ja) ビデオミーティング評価端末
JP7691150B2 (ja) 動画像分析システム
JP7691149B2 (ja) 動画像分析システム
JP7734984B2 (ja) 動画像分析システム
JP7477909B2 (ja) ビデオミーティング評価端末、ビデオミーティング評価システム及びビデオミーティング評価プログラム
JP7121436B1 (ja) 動画像分析プログラム
JP7629227B2 (ja) ビデオミーティング評価システム及びビデオミーティング評価サーバ
JP7734989B2 (ja) 動画像分析システム
JP7734990B2 (ja) 動画像分析システム
JP7688923B2 (ja) ビデオミーティング評価システム及びビデオミーティング評価サーバ
JP7152819B1 (ja) 動画像分析プログラム
JP7388768B2 (ja) 動画像分析プログラム
JP7121433B1 (ja) 動画像分析プログラム
JP7734983B2 (ja) 動画像分析システム
JP7121439B1 (ja) 動画像分析システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20946761

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20946761

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP