WO2022024353A1 - 反応解析システム - Google Patents

反応解析システム Download PDF

Info

Publication number
WO2022024353A1
WO2022024353A1 PCT/JP2020/029466 JP2020029466W WO2022024353A1 WO 2022024353 A1 WO2022024353 A1 WO 2022024353A1 JP 2020029466 W JP2020029466 W JP 2020029466W WO 2022024353 A1 WO2022024353 A1 WO 2022024353A1
Authority
WO
WIPO (PCT)
Prior art keywords
moving image
participants
reaction
displayed
reaction analysis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/029466
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
渉三 神谷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Imbesideyou Inc
Original Assignee
Imbesideyou Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Imbesideyou Inc filed Critical Imbesideyou Inc
Priority to PCT/JP2020/029466 priority Critical patent/WO2022024353A1/ja
Priority to JP2022539466A priority patent/JP7197957B2/ja
Priority to PCT/JP2021/027637 priority patent/WO2022025024A1/ja
Publication of WO2022024353A1 publication Critical patent/WO2022024353A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10—Services

Definitions

  • the present invention relates to a reaction analysis system.
  • Patent Document 1 a technique for analyzing emotions received by others in response to a speaker's remark is known (see, for example, Patent Document 1). Further, there is also known a technique in which a participant's face image is displayed on a small screen while the shared material is displayed so that the participant's state can be grasped (see, for example, Patent Document 2).
  • an object of the present invention is to make it possible to grasp the state of participants who are not displayed on the screen in an environment where an online session is held by a plurality of people.
  • reaction analysis system of the present invention in an environment where an online session is held by a plurality of participants, regardless of whether or not the participants are displayed on the screen during the online session.
  • the reaction of the participant is analyzed based on the moving image obtained by photographing the participant, and the analysis result is presented.
  • the reaction analysis system of the present embodiment is obtained by photographing a participant in an environment where an online session is performed by a plurality of participants regardless of whether or not the participant is displayed on the screen during the online session. It is a system that analyzes the reaction of the participants based on the moving image and presents the analysis result. In this embodiment, the biological reaction of the participants is analyzed. In addition, the emotions of the participants (feelings that occur in the words and actions of oneself or others, such as comfort / discomfort or the degree thereof) may be analyzed through the analysis of biological reactions.
  • An online session is, for example, an online conference, an online class, an online chat, etc., in which terminals installed in multiple locations are connected to a server via a communication network such as the Internet, and moving images are transmitted between the terminals through the server. It is designed to be able to communicate.
  • the moving image handled in the online session includes the face image and voice of the user who uses the terminal.
  • the moving image also includes an image such as a material shared and viewed by a plurality of users. It is possible to switch between the face image and the material image on the screen of each terminal to display only one of them, or to divide the display area and display the face image and the material image at the same time. Further, it is possible to display the image of one of a plurality of people on the full screen, or to display the image of a part or all of the users on a small screen.
  • FIG. 1 is a block diagram showing a functional configuration example of the reaction analysis system according to the present embodiment.
  • the reaction analysis system of the present embodiment includes a moving image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13 as functional configurations.
  • Each of the above functional blocks 11 to 13 can be configured by any of hardware, DSP (Digital Signal Processor), and software provided in the server device, for example.
  • DSP Digital Signal Processor
  • each of the above functional blocks 11 to 13 is actually configured to include a computer CPU, RAM, ROM, etc., and is a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. Is realized by the operation of.
  • the moving image acquisition unit 11 acquires a moving image obtained by shooting a plurality of people (multiple users) with a camera provided in each terminal during an online session. It does not matter whether the moving image acquired from each terminal is set to be displayed on the screen of each terminal. That is, the moving image acquisition unit 11 acquires the moving image from each terminal, including the moving image being displayed on each terminal and the moving image being hidden.
  • the biological reaction analysis unit 12 analyzes changes in the biological reaction of each of a plurality of persons based on the moving image (whether or not it is displayed on the screen) acquired by the moving image acquisition unit 11.
  • the biological reaction analysis unit 12 separates the moving image acquired by the moving image acquisition unit 11 into a set of images (a collection of frame images) and a voice, and analyzes changes in the biological reaction from each.
  • the biological reaction analysis unit 12 analyzes the user's face image using the frame image separated from the moving image acquired by the moving image acquisition unit 11, and thereby at least one of the facial expression, the line of sight, the pulse, and the movement of the face. Analyze changes in biological reactions related to one. In addition, the biological reaction analysis unit 12 analyzes changes in the biological reaction regarding at least one of the user's speech content and voice quality by analyzing the voice separated from the moving image acquired by the moving image acquisition unit 11.
  • a person's emotions change, it appears as changes in biological reactions such as facial expressions, eyes, pulse, facial movements, speech content, and voice quality.
  • changes in biological reactions caused by changes in the user's emotions are analyzed.
  • the change in the user's emotion may be analyzed by analyzing the change in the biological reaction.
  • the emotion analyzed in this case is, for example, the degree of comfort / discomfort.
  • Analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is specified from the frame image, and the specified facial expressions are classified into a plurality of types according to an image analysis model trained in advance by machine learning. Then, based on the classification result, it is analyzed whether a positive facial expression change occurs between consecutive frame images, a negative facial expression change occurs, and how large the facial expression change occurs.
  • Analysis of changes in the line of sight is performed, for example, as follows. That is, for each frame image, the area of the eyes is specified from the frame image, and the orientation of both eyes is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In particular, in the present embodiment, it is analyzed where the participants who are not displayed on the screen are looking at the shared materials displayed on the screen. In addition, it may be possible to analyze whether the movement of the line of sight is large or small, and whether the movement is frequent or infrequent. The change in the line of sight is also related to the degree of concentration of the user.
  • Analysis of pulse changes is performed, for example, as follows. That is, for each frame image, the face area is specified from the frame image. Then, using a trained image analysis model that captures the numerical value of the face color information (G in RGB), the change in the G color on the face surface is analyzed. By arranging the results along the time axis, a waveform showing the change in color information is formed, and the pulse is specified from this waveform. When a person is nervous, the pulse becomes faster, and when he / she feels calm, the pulse becomes slower.
  • Analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the area of the face is specified from the frame image, and the orientation of the face is analyzed to analyze where the user is looking. For example, it analyzes whether the speaker's face being displayed, the shared material being displayed, or the outside of the screen is being viewed. In particular, in the present embodiment, it is analyzed where the participants who are not displayed on the screen are looking at the shared materials displayed on the screen. In addition, it may be possible to analyze whether the movement of the face is large or small, and whether the movement is frequent or infrequent. The movement of the face and the movement of the line of sight may be combined and analyzed. For example, it may be possible to analyze whether the speaker's face being displayed is viewed straight, whether the speaker is viewed with an upper eye or a lower eye, or whether the speaker is viewed from an angle.
  • the content of the statement is analyzed as follows, for example. That is, the biological reaction analysis unit 12 converts the voice into a character string by performing a known voice recognition process on the voice for a specified time (for example, a time of about 30 to 150 seconds), and morphologically analyzes the character string. By doing so, words unnecessary for expressing conversation such as particles and acronyms are removed. Then, the remaining words are vectorized, and whether a positive emotional change is occurring, a negative emotional change is occurring, and the magnitude of the emotional change is analyzed.
  • a specified time for example, a time of about 30 to 150 seconds
  • Voice quality analysis is performed as follows, for example. That is, the biological reaction analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified time (for example, a time of about 30 to 150 seconds). Then, based on the acoustic characteristics, it is analyzed whether a positive voice quality change is occurring, a negative voice quality change is occurring, and how loud the voice quality change is occurring. Further, in the present embodiment, it is analyzed at what timing during the online session the participants who are not displayed on the screen make a voice.
  • a specified time for example, a time of about 30 to 150 seconds.
  • the reaction information presentation unit 13 presents information indicating changes in the biological reaction analyzed by the biological reaction analysis unit 12, including participants who are not displayed on the screen.
  • the reaction information presentation unit 13 presents information indicating changes in the biological reaction to the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer).
  • Organizers of online sessions include, for example, instructors of online classes, chairs and facilitators of online conferences, and coaches of sessions for coaching purposes.
  • the organizer of an online session is usually one of a plurality of users who participate in the online session, but may be another person who does not participate in the online session.
  • the organizer of the online session can also grasp the state of the participants who are not displayed on the screen in the environment where the online session is held by multiple people.
  • the reaction information presentation unit 13 relates to an analysis result regarding a change in the line of sight as to where the user was looking at the shared material being displayed, and a voice indicating which part of the shared material was displayed.
  • the heat map may be projected and displayed on the shared material based on at least one of the analysis results.

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Tourism & Hospitality (AREA)
  • Strategic Management (AREA)
  • General Business, Economics & Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Marketing (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Economics (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Primary Health Care (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • User Interface Of Digital Computer (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
PCT/JP2020/029466 2020-07-31 2020-07-31 反応解析システム Ceased WO2022024353A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/JP2020/029466 WO2022024353A1 (ja) 2020-07-31 2020-07-31 反応解析システム
JP2022539466A JP7197957B2 (ja) 2020-07-31 2021-07-27 反応解析システムおよび反応解析装置
PCT/JP2021/027637 WO2022025024A1 (ja) 2020-07-31 2021-07-27 反応解析システムおよび反応解析装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/029466 WO2022024353A1 (ja) 2020-07-31 2020-07-31 反応解析システム

Publications (1)

Publication Number Publication Date
WO2022024353A1 true WO2022024353A1 (ja) 2022-02-03

Family

ID=80035326

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/JP2020/029466 Ceased WO2022024353A1 (ja) 2020-07-31 2020-07-31 反応解析システム
PCT/JP2021/027637 Ceased WO2022025024A1 (ja) 2020-07-31 2021-07-27 反応解析システムおよび反応解析装置

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/027637 Ceased WO2022025024A1 (ja) 2020-07-31 2021-07-27 反応解析システムおよび反応解析装置

Country Status (2)

Country Link
JP (1) JP7197957B2 (https=)
WO (2) WO2022024353A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7333570B1 (ja) 2022-12-21 2023-08-25 株式会社Shift プログラム、方法、情報処理装置、システム

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005269207A (ja) * 2004-03-18 2005-09-29 Nippon Telegr & Teleph Corp <Ntt> 情報伝達方法及びこの方法を実現するための通信装置とそのプログラム
JP2007213364A (ja) * 2006-02-10 2007-08-23 Nec Corp 画像変換装置、画像変換方法及び画像変換プログラム
US20190019067A1 (en) * 2017-07-11 2019-01-17 International Business Machines Corporation Multimedia conferencing system for determining participant engagement
US20200169693A1 (en) * 2016-02-03 2020-05-28 Hewlett-Packard Development Company, L.P. Eye gaze angle feedback in a remote meeting

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008294724A (ja) * 2007-05-24 2008-12-04 Panasonic Corp 画像受信装置
JP2011065467A (ja) 2009-09-17 2011-03-31 Sharp Corp 会議中継装置及びコンピュータプログラム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005269207A (ja) * 2004-03-18 2005-09-29 Nippon Telegr & Teleph Corp <Ntt> 情報伝達方法及びこの方法を実現するための通信装置とそのプログラム
JP2007213364A (ja) * 2006-02-10 2007-08-23 Nec Corp 画像変換装置、画像変換方法及び画像変換プログラム
US20200169693A1 (en) * 2016-02-03 2020-05-28 Hewlett-Packard Development Company, L.P. Eye gaze angle feedback in a remote meeting
US20190019067A1 (en) * 2017-07-11 2019-01-17 International Business Machines Corporation Multimedia conferencing system for determining participant engagement

Also Published As

Publication number Publication date
JPWO2022025024A1 (https=) 2022-02-03
WO2022025024A1 (ja) 2022-02-03
JP7197957B2 (ja) 2022-12-28

Similar Documents

Publication Publication Date Title
JP2023015068A (ja) 感情解析システムおよび感情解析装置
JP2023075197A (ja) 反応解析システムおよび反応解析装置
JP2025186362A (ja) 感情解析システムおよび感情解析装置
WO2022024353A1 (ja) 反応解析システム
JP7553128B2 (ja) 感情解析システム
JP7465040B1 (ja) コミュニケーション可視化システム
WO2022024356A1 (ja) 組織属性解析システム
WO2022064621A1 (ja) ビデオミーティング評価システム及びビデオミーティング評価サーバ
JP7668017B2 (ja) ビデオミーティング評価端末
JP7121436B1 (ja) 動画像分析プログラム
JP7121433B1 (ja) 動画像分析プログラム
JP7691150B2 (ja) 動画像分析システム
JP7734983B2 (ja) 動画像分析システム
JP7691149B2 (ja) 動画像分析システム
JP7734990B2 (ja) 動画像分析システム
JP7734988B2 (ja) 動画像分析システム
JP7734989B2 (ja) 動画像分析システム
JP7745906B2 (ja) 動画像分析システム
WO2022254497A1 (ja) 動画像分析システム
WO2022230136A1 (ja) 動画像分析システム
WO2022201272A1 (ja) 動画像分析プログラム
JP2022146949A (ja) 動画像分析プログラム
WO2022064620A1 (ja) ビデオミーティング評価システム及びビデオミーティング評価サーバ
WO2022201267A1 (ja) 動画像分析プログラム
WO2022230066A1 (ja) 動画像分析システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20947133

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20947133

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP