WO2022024354A1 - 反応解析システム - Google Patents

反応解析システム Download PDF

Info

Publication number
WO2022024354A1
WO2022024354A1 PCT/JP2020/029467 JP2020029467W WO2022024354A1 WO 2022024354 A1 WO2022024354 A1 WO 2022024354A1 JP 2020029467 W JP2020029467 W JP 2020029467W WO 2022024354 A1 WO2022024354 A1 WO 2022024354A1
Authority
WO
WIPO (PCT)
Prior art keywords
biological reaction
unit
moving image
change
analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/029467
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
渉三 神谷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Imbesideyou Inc
Original Assignee
Imbesideyou Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Imbesideyou Inc filed Critical Imbesideyou Inc
Priority to PCT/JP2020/029467 priority Critical patent/WO2022024354A1/ja
Priority to PCT/JP2021/028142 priority patent/WO2022025200A1/ja
Priority to JP2022539571A priority patent/JP7242114B2/ja
Publication of WO2022024354A1 publication Critical patent/WO2022024354A1/ja
Anticipated expiration legal-status Critical
Priority to JP2023031017A priority patent/JP2023075197A/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management

Definitions

  • the present invention relates to a reaction analysis system.
  • Patent Document 1 a technique for analyzing emotions received by others in response to a speaker's remark is known (see, for example, Patent Document 1). Further, there is also known a technique of analyzing changes in a subject's facial expression over a long period of time in a time series and estimating the emotions held during that period (see, for example, Patent Document 2). Further, a technique for identifying an element that has the greatest influence on emotional changes is also known (see, for example, Patent Documents 3 to 5). Furthermore, there is also known a technique of comparing a subject's normal facial expression with the current facial expression and issuing an alert when the facial expression is dark (see, for example, Patent Document 6).
  • An object of the present invention is to analyze a specific reaction different from the previous one for a participant in an environment where an online session is held by a plurality of people, and to make it possible to utilize the analysis result.
  • the reaction analysis system of the present invention is based on a moving image obtained by photographing a participant for each of a plurality of online sessions in an environment where an online session is held by a plurality of people. , Analyze different specific reactions for participants.
  • the reaction analysis system of the present embodiment is different from the previous one for the participants based on the moving images obtained by photographing the participants for each of a plurality of online sessions in an environment where an online session is held by a plurality of people. It is a system that analyzes specific reactions.
  • the biological reaction of the participants is analyzed.
  • the emotions of the participants may be analyzed through the analysis of biological reactions.
  • An online session is, for example, an online conference, an online class, an online chat, etc., in which terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between the terminals through the server. It is designed to be able to communicate. It is preferable that the target online session in the present embodiment is performed multiple times with the same participant or almost the same participant.
  • the moving image handled in the online session includes the face image and voice of the user who uses the terminal.
  • the moving image also includes an image such as a material shared and viewed by a plurality of users. It is possible to switch between the face image and the material image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the material image at the same time. Further, it is possible to display the image of one of a plurality of people on the full screen, or to display the image of a part or all of the users on a small screen.
  • the leader, facilitator, or administrator of the online session designates any user as the analysis target.
  • Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes.
  • the organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.
  • all the participants may be the analysis target without designating the analysis target person.
  • FIG. 1 is a block diagram showing a functional configuration example of the reaction analysis system according to the present embodiment.
  • the reaction analysis system of the present embodiment has a moving image acquisition unit 11, a biological reaction analysis unit 12, a peculiarity determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification as functional configurations.
  • the part 16 is provided.
  • Each of the above functional blocks 11 to 16 can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in the server device, for example.
  • DSP Digital Signal Processor
  • each of the above functional blocks 11 to 16 is actually configured to include a computer CPU, RAM, ROM, etc., and is a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. Is realized by the operation of.
  • the moving image acquisition unit 11 acquires a moving image obtained by shooting a plurality of people (multiple users) with a camera provided in each terminal during the online session for each of a plurality of online sessions. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires the moving image from each terminal, including the moving image being displayed on each terminal and the moving image being hidden.
  • the biological reaction analysis unit 12 analyzes changes in the biological reaction of the designated analysis target person based on the moving image acquired by the moving image acquisition unit 11. It should be noted that the change in the biological reaction may be analyzed for each of a plurality of persons including others other than the analysis target person. In the present embodiment, the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and a voice, and analyzes changes in the biological reaction from each.
  • the biological reaction analysis unit 12 analyzes the user's face image using the frame image separated from the moving image acquired by the moving image acquisition unit 11, and thereby at least one of the facial expression, the line of sight, the pulse, and the movement of the face. Analyze changes in biological reactions related to one. Further, the biological reaction analysis unit 12 analyzes changes in the biological reaction regarding at least one of the user's speech content and voice quality by analyzing the voice separated from the moving image acquired by the moving image acquisition unit 11.
  • the biological reaction analysis unit 12 calculates a biological reaction index value that reflects the content of the change in the biological reaction by quantifying the change in the biological reaction according to a predetermined standard.
  • Analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is specified from the frame image, and the specified facial expressions are classified into a plurality of types according to an image analysis model trained in advance by machine learning. Then, based on the classification result, it is analyzed whether a positive facial expression change occurs between consecutive frame images, a negative facial expression change occurs, and how large the facial expression change occurs. The facial expression change index value according to the analysis result is output.
  • Analysis of changes in the line of sight is performed, for example, as follows. That is, for each frame image, the area of the eyes is specified from the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the line of sight is large or small, and whether the movement is frequent or infrequent. The change in the line of sight is also related to the degree of concentration of the user.
  • the biological reaction analysis unit 12 outputs the line-of-sight change index value according to the analysis result of the line-of-sight change.
  • Analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified from the frame image. Then, using a trained image analysis model that captures the numerical value of the face color information (G in RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform showing the change in color information is formed, and the pulse is specified from this waveform. When a person is nervous, the pulse becomes faster, and when he / she feels calm, the pulse becomes slower.
  • the biological reaction analysis unit 12 outputs a pulse change index value according to the analysis result of the pulse change.
  • Analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the area of the face is specified from the frame image, and the orientation of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In addition, it may be possible to analyze whether the movement of the face is large or small, and whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be combined and analyzed. For example, it may be possible to analyze whether the speaker's face being displayed is viewed straight, whether the speaker is viewed with an upper eye or a lower eye, or whether the speaker is viewed from an angle.
  • the biological reaction analysis unit 12 outputs a face orientation change index value according to the analysis result of the face orientation change.
  • the content of the statement is analyzed as follows, for example. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing a known voice recognition process on the voice for a specified time (for example, a time of about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words unnecessary for expressing conversation such as particles and acronyms are removed. Then, the remaining words are vectorized, and whether a positive emotional change is occurring, a negative emotional change is occurring, and how large the emotional change is occurring is analyzed, and the analysis result is used. Outputs the statement content index value.
  • Voice quality analysis is performed as follows, for example. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change is occurring, a negative voice quality change is occurring, and how loud the voice quality change is occurring, and according to the analysis result. Outputs the voice quality change index value.
  • the biological reaction analysis unit 12 uses at least one of the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value calculated as described above.
  • the biological reaction index value is calculated.
  • the biological reaction index value is calculated by weighting the facial expression change index value, the line-of-sight change index value, the pulse change index value, the face orientation change index value, the speech content index value, and the voice quality change index value.
  • the change in the biological reaction analyzed for the analysis target person in one online session is the change in the biological reaction analyzed for the analysis target person in the online session prior to the one online session. Judge whether it is specific or not by comparison.
  • the peculiarity determination unit 13 is based on the biological reaction index value calculated for the analysis target person by the biological reaction analysis unit 12, and the change in the biological reaction analyzed for the analysis target person is more specific than before. Judge whether or not.
  • the peculiarity determination unit 13 calculates the variance of the biological reaction index value calculated for each of a plurality of online sessions for the analysis target person by the biological reaction analysis unit 12, and the biological reaction index value calculated for this online session. By contrasting with the variance, it is determined whether or not the change in the biological reaction analyzed for the subject to be analyzed this time is more specific than before.
  • the related event identification unit 14 identifies an event occurring with respect to at least one of the analysis subject, another person, and the environment when a change in the biological reaction determined to be specific by the peculiarity determination unit 13 occurs. .. For example, the related event identification unit 14 identifies the behavior of the analysis target person himself / herself from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the behavior of another person from the moving image when a specific change in the biological reaction occurs for the analysis target person. In addition, the related event identification unit 14 identifies the environment when a specific change in the biological reaction occurs for the analysis target person from the moving image.
  • the environment is, for example, a shared material displayed on the screen, an environment reflected in the background of the person to be analyzed, and the like. By identifying such an event, it is possible to capture an event that may have influenced the change in the reaction of the analysis subject.
  • the clustering unit 15 is the content of the change in the biological reaction determined to be specific by the peculiarity determination unit 13 (for example, one or a combination of the line of sight, the pulse, the movement of the face, the content of speech, and the voice quality) and the past.
  • the pattern of change in biological response is clustered based on the magnitude of change from. This makes it possible to analyze and present patterns of behaviors that may occur in the future for the analysis target person. For example, it is possible to understand that the subject to be analyzed may withdraw from the online session when the degree of smile gradually decreases and negative biological reactions continue for a predetermined number of times. You may want to give an alert to let you know.
  • the analysis result notification unit 16 is at least one of a change in a biological reaction determined to be specific by the peculiarity determination unit 13, an event specified by the related event identification unit 14, and a change pattern clustered by the clustering unit 15. Is notified to the designated person (the organizer of the online session) of the analysis target.
  • the organizer of the online session may be notified of the future behavior pattern of the analysis target person predicted from the clustered change pattern, a predetermined alert based on the future behavior pattern, and the like.
  • the organizer of the online session grasps the behavior tendency of the specified analysis target person and predicts the behavior and state that may occur in the future, depending on what kind of change pattern the specified analysis target person is clustered into. be able to. Then, it becomes possible to take appropriate measures for the analysis target person.

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Strategic Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Human Resources & Organizations (AREA)
  • Operations Research (AREA)
  • General Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • Physics & Mathematics (AREA)
  • Economics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Medical Treatment And Welfare Office Work (AREA)
PCT/JP2020/029467 2020-07-31 2020-07-31 反応解析システム Ceased WO2022024354A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
PCT/JP2020/029467 WO2022024354A1 (ja) 2020-07-31 2020-07-31 反応解析システム
PCT/JP2021/028142 WO2022025200A1 (ja) 2020-07-31 2021-07-29 反応解析システムおよび反応解析装置
JP2022539571A JP7242114B2 (ja) 2020-07-31 2021-07-29 反応解析システムおよび反応解析装置
JP2023031017A JP2023075197A (ja) 2020-07-31 2023-03-01 反応解析システムおよび反応解析装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/029467 WO2022024354A1 (ja) 2020-07-31 2020-07-31 反応解析システム

Publications (1)

Publication Number Publication Date
WO2022024354A1 true WO2022024354A1 (ja) 2022-02-03

Family

ID=80035314

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/JP2020/029467 Ceased WO2022024354A1 (ja) 2020-07-31 2020-07-31 反応解析システム
PCT/JP2021/028142 Ceased WO2022025200A1 (ja) 2020-07-31 2021-07-29 反応解析システムおよび反応解析装置

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/028142 Ceased WO2022025200A1 (ja) 2020-07-31 2021-07-29 反応解析システムおよび反応解析装置

Country Status (2)

Country Link
JP (2) JP7242114B2 (https=)
WO (2) WO2022024354A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115381411A (zh) * 2022-09-19 2022-11-25 慧医谷中医药科技(天津)股份有限公司 一种脉象分析信息处理方法及系统

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023183280A (ja) * 2022-06-15 2023-12-27 京セラ株式会社 電子機器、電子機器の制御方法及び制御プログラム
JP2023183278A (ja) * 2022-06-15 2023-12-27 京セラ株式会社 電子機器、電子機器の制御方法及び制御プログラム

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010191689A (ja) * 2009-02-18 2010-09-02 Nec Corp 現象対策支援装置、端末、現象対策支援システム、現象対策支援方法及び現象対策支援プログラム

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9113035B2 (en) 2013-03-05 2015-08-18 International Business Machines Corporation Guiding a desired outcome for an electronically hosted conference
WO2014192457A1 (ja) * 2013-05-30 2014-12-04 ソニー株式会社 クライアント装置、制御方法、システム、およびプログラム
JP2016046705A (ja) 2014-08-25 2016-04-04 コニカミノルタ株式会社 会議録編集装置、その方法とプログラム、会議録再生装置、および会議システム
JP6801459B2 (ja) * 2017-01-10 2020-12-16 日本電気株式会社 情報処理装置、いじめ発見方法、情報処理システム、および、コンピュータプログラム
JP7225631B2 (ja) * 2018-09-21 2023-02-21 ヤマハ株式会社 画像処理装置、カメラ装置、および画像処理方法
JP2020048610A (ja) 2018-09-21 2020-04-02 富士ゼロックス株式会社 状態評価システム

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010191689A (ja) * 2009-02-18 2010-09-02 Nec Corp 現象対策支援装置、端末、現象対策支援システム、現象対策支援方法及び現象対策支援プログラム

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ANDO, JUN: "Visualizing Employee Health", NIHON KEIZAI SHIMBUN, vol. 13, 20 April 2020 (2020-04-20) *
ANONYMOUS: "AI Sakura discovers mental disorders early! -It is now possible to support the mental health of employees who are newly teleworking- ", TIFANA, 3 June 2020 (2020-06-03), XP055903252, Retrieved from the Internet <URL:https://www.tifana.ai/news/20200603> [retrieved on 20220321] *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115381411A (zh) * 2022-09-19 2022-11-25 慧医谷中医药科技(天津)股份有限公司 一种脉象分析信息处理方法及系统

Also Published As

Publication number Publication date
WO2022025200A1 (ja) 2022-02-03
JPWO2022025200A1 (https=) 2022-02-03
JP2023075197A (ja) 2023-05-30
JP7242114B2 (ja) 2023-03-20

Similar Documents

Publication Publication Date Title
WO2022024354A1 (ja) 反応解析システム
JP2023015068A (ja) 感情解析システムおよび感情解析装置
WO2022024355A1 (ja) 感情解析システム
WO2022024353A1 (ja) 反応解析システム
WO2022064621A1 (ja) ビデオミーティング評価システム及びビデオミーティング評価サーバ
JP7553128B2 (ja) 感情解析システム
JP7465040B1 (ja) コミュニケーション可視化システム
JP7691149B2 (ja) 動画像分析システム
JP7691150B2 (ja) 動画像分析システム
JP7668017B2 (ja) ビデオミーティング評価端末
JP7121436B1 (ja) 動画像分析プログラム
JP7388768B2 (ja) 動画像分析プログラム
JP7734987B2 (ja) 動画像分析システム
JP7734983B2 (ja) 動画像分析システム
JP7121433B1 (ja) 動画像分析プログラム
JP7100938B1 (ja) 動画像分析プログラム
JP7705176B2 (ja) 動画像分析システム
JP7734989B2 (ja) 動画像分析システム
JP7705177B2 (ja) 動画像分析システム
JP7629226B2 (ja) ビデオミーティング評価端末、ビデオミーティング評価システム及びビデオミーティング評価プログラム
JP7734990B2 (ja) 動画像分析システム
US11935329B2 (en) Video analysis program
JP7477909B2 (ja) ビデオミーティング評価端末、ビデオミーティング評価システム及びビデオミーティング評価プログラム
JP7730575B2 (ja) 動画像分析システム
JP7471683B2 (ja) 反応通知システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20947476

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20947476

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP