JP2024079546A - Information analysis system, information analysis apparatus, information analysis method, and program - Google Patents

Abstract

To solve the problem in which a conventional system can identify the states of participants in an online event, only based on their speeches, but cannot identify the states of the participants based on their expressions to utilize information obtained from the speeches and the expressions.SOLUTION: A communication terminal 3 in an information analysis system 1 is configured to: acquire image data related to images including faces of participants captured by a web camera, or the like, and voice data related to speeches of the participants obtained by a voice microphone, or the like (Step S103); and transmit the acquired image data and the voice data of the participants to an information analysis apparatus 5 (Step S106). The information analysis apparatus 5 analyzes the states of the participants on the basis of the image data and the voice data transmitted by the communication terminal 3, and generates report data related to an analysis result to be displayed on the communication terminal 3 (Step S108).SELECTED DRAWING: Figure 12

Description

The present invention relates to an information analysis system, an information analysis device, an information analysis method, and a program.

Mechanisms that acquire and output the activity level of individuals based on their speech in a group are already known.

One such technique involves acquiring the speech activity level of each of multiple individuals in a group, setting a reference value for the speech of each individual in the group based on the acquired speech activity level of each individual, and outputting the result of comparing the acquired speech activity level of each individual with the set reference value (see, for example, Patent Document 1).

On the other hand, as working styles become more diverse, the traditional method of people holding meetings in the same physical space is being replaced by online methods using remote environments such as web conferencing.

However, with conventional technology, while it is possible to grasp the state of participants based on their speech at online events, there is an issue in that it is difficult to grasp their state based on facial expressions as well as to utilize the information obtained from speech and facial expressions.

In order to solve the above-mentioned problems, the invention according to claim 1 provides an information analysis system having a communication terminal capable of acquiring information related to multiple participants who have participated in a specific event held online, and an information analysis device that analyzes the information transmitted by the communication terminal, wherein the communication terminal has an acquisition means for acquiring image information and audio information of the multiple participants, and a first transmission means for transmitting the acquired image information and audio information to the information analysis device, and the information analysis device has an analysis means for analyzing the status of the multiple participants based on the image information and the audio information transmitted by the communication terminal, and a generation means for generating a status notification that can be displayed on the communication terminal based on the analyzed status of the multiple participants.

As described above, the present invention has the effect of making it possible to grasp the state of participants based on not only their speech but also their facial expressions, even in online events, and to utilize the information obtained from their speech and facial expressions.

FIG. 1 is a diagram illustrating an example of an overall configuration of an information analysis system. 1A and 1B are conceptual diagrams showing an example of a convolutional neural network, in which (a) shows an example of nodding detection in late fusion, and (b) shows an example of nodding detection in early fusion. FIG. 13 is a conceptual diagram illustrating an example of smile detection. FIG. 2 is a diagram illustrating an example of a hardware configuration of a communication terminal. FIG. 2 illustrates an example of a hardware configuration of an information analysis device. FIG. 2 is a diagram illustrating an example of a functional configuration of the information analysis system. FIG. 13 is a conceptual diagram illustrating an example of a user information management table. FIG. 13 is a conceptual diagram illustrating an example of an analysis history management table. FIG. 13 is a conceptual diagram showing an example of an individual vitality level management table. FIG. 13 is a conceptual diagram illustrating an example of a group activity level management table. 11 is a sequence diagram showing an example of each process of participant information notification, member registration, and authentication. FIG. FIG. 11 is a sequence diagram showing an example of information analysis processing for an online event. FIG. 13 is a diagram showing an example of a display screen of a communication terminal at an online event. 13 is a flowchart showing details of information analysis processing at an online event. 13 is a flowchart showing details of information analysis processing at an online event. FIG. 13 is a diagram showing an example of a report display screen (morning assembly scores) on the communication terminal. FIG. 13 is a diagram showing an example of a report display screen (energy score) on the communication terminal. FIG. 13 is a diagram showing an example of a report display screen (past online morning assembly scores) on a communication terminal. FIG. 13 is a diagram showing an example of a report display screen (when a nod is detected) on the communication terminal. FIG. 13 is a diagram showing an example of a report display screen (when a smile is detected) on the communication terminal. FIG. 13 is a diagram showing an example of a report display screen (utterance ratio) in the communication terminal. FIG. 13 is a diagram showing an example of a report display screen (speaker) in the communication terminal. FIG. 13 is a diagram showing an example of a report display screen (when words of gratitude are detected) on the communication terminal. FIG. 11 is a sequence diagram showing an example of information analysis processing at an online event according to the second embodiment. FIG. 11 is a diagram showing an example of a display screen of a communication terminal at an online event according to the second embodiment. FIG. 11 is a sequence diagram illustrating an example of an information analysis process for an online event according to a modified example of the embodiment. FIG. 13 is a diagram illustrating an example of an overall configuration of an information analysis system according to a third embodiment. FIG. 13 is a conceptual diagram showing an example of a personal data usage management table according to the third embodiment. FIG. 13 is a sequence diagram showing an example of a personal data usage setting process according to the third embodiment. FIG. 13 is a diagram illustrating an example of a sign-in screen in a communication terminal according to a third embodiment. FIG. 13 is a diagram showing an example of a video playback screen on a communication terminal according to a third embodiment. FIG. 13 is a diagram showing an example of a registration screen related to confirmation and consent on a communication terminal according to the third embodiment. FIG. 13 is a diagram showing an example of a user setting screen in a communication terminal according to the third embodiment. FIG. 13 is a sequence diagram showing an example of information analysis processing at an online event according to the third embodiment. 13 is a flowchart showing details of an information analysis process for an online event according to the third embodiment. 13 is a flowchart showing details of an information analysis process for an online event according to the third embodiment. 13 is a flowchart showing details of a report data generation process according to the third embodiment. FIG. 13 is a diagram showing an example of a report display screen (vitality score) on the communication terminal according to the third embodiment.

Below, we will explain the form for implementing the invention using the drawings. Note that in the explanation of the drawings, the same elements are given the same reference numerals, and if there are overlapping parts, their explanation will be omitted.

First Embodiment
[Overall configuration of the information analysis system]
FIG. 1 is a diagram showing an example of the overall configuration of an information analysis system. As shown in FIG. 1, the information analysis system 1 has various devices and terminals including a communication terminal 3 and an information analysis device 5. The communication terminal 3 and the information analysis device 5 hold a Web conference (online morning assembly, online meeting) in an online event holding space surrounded by a dotted line. The online event holding space is shown for convenience as a virtual space in which an online Web conference is held. Furthermore, in the information analysis system 1, the communication terminal 3 and the information analysis device 5 are connected to each other via a communication network 100. The communication terminal 3 may be connected to the information analysis device 5 via a firewall provided inside the communication network 100.

The communication network 100 is a communication network through which an unspecified number of communications are conducted, and is constructed using the Internet, an intranet, a LAN (Local Area Network), etc. Note that the communication network 100 may include not only wired communication, but also communication networks using wireless communication such as 4G (4th Generation), 5G (5th Generation), WiMAX (Worldwide Interoperability for Microwave Access), and LTE (Long Term Evolution).

<Communication terminal>
The communication terminal 3 is a communication terminal that is used by a user (participant, leader, etc.) who uses the information analysis system 1 and can acquire information related to multiple participants who have participated in a specific event held online. The communication terminal 3 is realized by an information processing device (computer system) for communication that is equipped with a general OS or the like, and is one of the components that configure the information analysis system 1. The communication terminal 3 also installs and stores in a storage unit a communication app for communicating with the information analysis device 5, an online conference app for online meetings, and the like.

The communication terminal 3 may be a communication terminal having an image acquisition unit that captures and acquires images, and an audio acquisition unit that collects and acquires audio, in addition to the communication functions of commonly used PCs (Personal Computers), portable notebook PCs, mobile phones, smartphones, tablet terminals, and wearable terminals (sunglasses type, wristwatch type, etc.). The communication terminal 3 may further be a communication device or communication terminal capable of running browser software and various application software.

<Information Analysis Device>
The information analyzing device 5 is a device that analyzes information transmitted by the communication terminal 3. The information analyzing device 5 is realized by one or more information processing devices (computer systems) equipped with a general server OS, etc. The information analyzing device 5 also installs and stores in a storage unit a communication application for communicating with other devices and communication terminals.

The information analysis device 5 may be constructed by a single computer such as a commonly used PC (Personal Computer) or a portable notebook PC, or may be constructed by multiple computers in which each section (function or means) such as storage is divided and arbitrarily assigned. In addition, all or part of the functions of the information analysis device 5 may be a server computer existing in a cloud environment, or a server computer existing in an on-premise environment. The information analysis device 5 may further use a communication device or communication terminal capable of operating software such as browser software.

In addition, the information analysis device 5 may have an environment in which a system available for a commonly known cloud-based web service (e.g., AWS: Amazon Web Services) can be used.

When sending a report or the like to the communication terminal 3, the information analysis device 5 may notify (send) the data (information) by push notification (transmission). In that case, the information analysis device 5 performs the push notification via, for example, FCM (Firebase Cloud Messaging), which is an example of a push notification server.

●About terminology●
In this embodiment, an event is a general term for an activity in which multiple participants share a topic or topic at a specific date and time. Events include, for example, meetings, morning assemblies, afternoon assemblies, evening assemblies, and consultations. In this embodiment, it is assumed that these events are held online using Web services. Based on this assumption, a "specific event held online" in this embodiment is treated as an example of "online communication." The communication includes image information (image data) and audio information (audio data). Note that the image information represents at least one of information related to nods and information related to smiles contained in the images of each of the multiple participants in the specific event. Furthermore, the audio information represents information related to the speech of each of the multiple participants in the specific event.

In addition, in this embodiment, a user refers to each participant who participates in an online event using a communication terminal 3, or an administrator (leader, etc.) who manages each participant. Hereinafter, in this embodiment, these are sometimes collectively referred to as "users" for convenience.

<Nodding detection>
First, in this embodiment, a method for detecting "nodding" of participants who have participated in a specific online event will be described. FIG. 2 is a conceptual diagram showing an example of a convolutional neural network, in which (a) shows an example of nodding detection in late fusion, and (b) shows an example of nodding detection in early fusion. Note that the conceptual diagram of the machine learning model for nodding detection shown in FIG. 2 is a partially simplified version of that shown in Reference 1 below. In addition, in FIG. 2, it is also possible to use a machine learning model that excludes speech sections.

Reference 1: Analyzing Multifunctionality of Head Movements in Face-to-Face Conversations Using Deep Convolutional Neural Networks
KAZUHIRO OTSUKA 1, (Member, IEEE), AND MASAHIRO TSUMORI2
As the structure of a convolutional neural network (CNN) model, FIG. 2(a) shows a late fusion model, and FIG. 2(b) shows an early fusion model. Both models are composed of a convolutional layer (conv1) and a pooling layer (pool1), a second convolutional layer (conv2) and a pooling layer (pool2), a third convolutional layer (conv3), a fully connected layer (fc4), and a final fully connected layer (fc5). For details, please refer to Reference 1 mentioned above, and the explanation will be omitted.

In this embodiment, the following procedure, for example, is presented as a flow of nodding detection.
Step 1: Data Collection Data collection involves collecting pre-recorded video data showing nodding throughout the video, and video data recorded at a given online event.
Step 2: Annotation In annotation, a nod is annotated for step 1, and training data is created (generated).
Step 3: Detecting features of the dataset In detecting features of the dataset, face keypoints are acquired using an existing library (e.g., the MediaPipe library). Then, head movement is calculated from the acquired keypoints.

Step 4: Model creation (generation)
In model creation (generation), a model is constructed from existing paper information (Reference 1 above).

For nodding detection, you may use the information in Reference 2 below.

Reference 2: Nodding detection system based on head motion and voice rhythm
Shunsuke OTA＊, Seiko TAKI＊＊, Mitsuru JINDAI＊＊＊ and Toshiyuki YASUDA＊＊＊
Step 5: Model evaluation In model evaluation, the constructed model is evaluated.

Using steps 1-5 above as an example, it is possible to detect participants' "nodding" from image data captured at an online event.

<Smile detection>
Next, a method for detecting the "smiles" of participants who have participated in a specific online event will be described. FIG. 3 is a conceptual diagram showing an example of smile detection. In this embodiment, smile detection using FIG. 3 is premised on performing different processes depending on whether or not the participant is wearing a mask. Based on this premise, RAN (Region Attention Networks) enclosed in a square in FIG. 3 will be described. Note that a description of the other processes shown in FIG. 3 will be omitted.

In RAN, several types of cropped images are generated by cutting out parts of the input image, and each of the cropped images is input into a neural network, which then takes into account the "importance" of each and outputs a final evaluation (whether the face is smiling or not). The "importance" of each cropped image is also calculated at the same time by the neural network. This configuration allows inference to be made by focusing on only the parts of the facial image that are important for facial expression recognition, making inference robust even when parts of the face are hidden.

<<Mask wearing judgment>>
Here, the mask wearing judgment will be described. The mask wearing judgment is performed by a mask wearing judgment model that inputs a person's face image and outputs binary information indicating whether or not a mask is worn on the person's face captured as a face image. The mask wearing judgment model is generated by supervised machine learning using a known machine learning algorithm such as a convolutional neural network (CNN) using training data consisting of pairs of a person's face image and binary information indicating whether or not a mask is worn on the person's face captured in the face image.

<<Smile detection when wearing a mask/not wearing a mask>>
Next, the detection of smiles when wearing/not wearing a mask will be described. Smile detection is performed by a smile detection model that inputs a person's face image and outputs binary information indicating whether the face of the person photographed in the face image is smiling. However, if the input image is a face image of a person wearing a mask, smile detection is performed by the smile detection model when wearing a mask, and if the input image is a face image of a person not wearing a mask, smile detection is performed by the smile detection model when not wearing a mask. Both the smile detection model when wearing a mask and the smile detection model when not wearing a mask are generated by supervised machine learning using RAN and training data consisting of pairs of a person's face image and binary information indicating whether the face of the person photographed in the face image is smiling. However, the training data used to generate the smile detection model when wearing a mask is composed only of face images of people wearing masks, and the training data used to generate the smile detection model when not wearing a mask is composed only of face images of people not wearing masks.

[Concept of evaluation index in embodiment]
Here, the concept of the evaluation index in this embodiment will be explained. Here, as the evaluation index, "individual vitality level" indicating the vitality level of an individual participant participating in an online event and "group activity level" indicating the activity level of a group (team) consisting of multiple participants are defined.

<Calculation of individual vitality level>
First, an overview of the calculation of the "individual vitality level" executed in this embodiment will be described.

The vitality level of each participant taking part in an online event (hereafter referred to as "individual vitality level") is calculated using only image data. Specifically, the time of "nodding" and "smiling" is detected. In order to calculate individual vitality level, it is necessary to use indicators that are likely to appear as behavior during online events such as morning meetings (hereafter referred to simply as "online events"). In particular, "nodding" and "smiling" are likely to appear as changes in participants' behavior during online events. Therefore, "nodding" and "smiling" are used to calculate individual vitality level.

If individual vitality levels were evaluated as absolute values, there would be large differences depending on the participants' personalities, making it difficult to calculate each individual's vitality level. Therefore, this problem can be overcome by using data from at least the past few sessions (for example, the past three sessions).

<Calculation of group activity level in a group (team)>
Next, an overview of the calculation of the "group activity level" executed in this embodiment will be described.

When an online event is taking place, the activity level of a group (team) (hereafter referred to as "group activity level") is calculated using image data and voice data of each participant. Specifically, the time each participant "nodded" and "smiled" is detected using image data. In addition, the speech ratio of each participant (who spoke and at what rate), the speaker ratio (who spoke), and the number of words of thanks are detected using voice data.

In this embodiment, "nodding," "smiles," "speech ratio," "speaker ratio," and "thanks" are each scored, and the sum of these scores is displayed as the group (team) collective activity level (score). As described below, "nodding," "smiles," "speech ratio," "speaker ratio," and "thanks" are each scored on a maximum of 20 points, for example, but the ratio of these scores may be changed. Furthermore, the group (team) collective activity level may be displayed using symbols or marks instead of scores.

In addition, to improve the collective activity level of a group (team), it is necessary to ensure psychological safety. In order to improve psychological safety, "nodding" and "smiles" are important indicators. In a place (space) where a specific event, including online events, increasing "nodding" can create empathy among participants and give the other person a sense of security. In addition, since increasing the number of smiles from one person can spread to those around them, "smiles" are important indicators for creating a good atmosphere in a group (team). In addition, "nodding" and "smiles" are adopted in this embodiment because they are characteristics that are easy to detect from moving images.

Furthermore, gratitude is also an important indicator for ensuring psychological safety. In an organization without gratitude, there is a danger that "natural rights" and "taking things for granted" will become prevalent. Therefore, verbalizing gratitude creates psychological safety and a sense of solidarity within a group (team).

Furthermore, even in a short online morning meeting, if you want to improve communication, it is not desirable for only certain people to be speaking. Therefore, by measuring and visualizing the ratio of how much participants other than a certain person (a facilitator such as a leader) are speaking, it is expected that the leader will be able to pass the talk to other participants more often. However, since it is also not desirable for only certain members to be speaking, in this embodiment, by visualizing who is speaking, it is possible to pass the talk to other participants equally.

The information analysis system in this embodiment defines the above evaluation indexes, calculates the "individual vitality" and "group activity" of each participant who takes part in a specific event held online, and provides the results as a report. This makes it possible to understand the state of participants based on not only their speech but also their facial expressions, even in online events, i.e. environments where it is not possible to directly grasp the facial expressions of each participant, and to make use of the information obtained from their speech and facial expressions.

[Hardware configuration]
Next, the hardware configuration of the communication terminal or device constituting the information analysis system according to the embodiment will be described with reference to Figures 4 and 5. Note that components may be added or deleted from the hardware configuration of the communication terminal or device shown in Figures 4 and 5 as necessary.

<Hardware configuration of communication terminal>
Fig. 4 is a diagram showing an example of the hardware configuration of a communication terminal. As shown in Fig. 4, the communication terminal 3 is constructed by, for example, a computer, and includes a CPU 301, a ROM 302, a RAM 303, an EEPROM 304, a display 307, a short-range communication I/F 308, a CMOS sensor 309, and an image sensor I/F 310. The communication terminal 3 further includes a network I/F 311, a touch panel 312, a pointing device 313, a media I/F 315, an external device connection I/F 316, a sound input/output I/F 317, a microphone 318, a speaker 319, and a bus line 320. However, the CMOS sensor 309 and the image sensor I/F 310 that constitute the function as an image acquisition unit such as a Web camera, and the sound input/output I/F 317 and the microphone 318 that constitute the function as a sound acquisition unit such as a sound microphone may each be configured to be detachably attached to the communication terminal 3.

Of these, the CPU 301 controls the operation of the entire communication terminal 3. The ROM 302 stores programs and the like used to drive the CPU 301. The RAM 303 is used as a work area for the CPU 301. The EEPROM 304 reads or writes various data such as apps under the control of the CPU 301. The display 307 displays various information such as a cursor, menu, window, text, or image. In this embodiment, the display 307 functions as an example of a display means. The short-range communication I/F 308 is a communication circuit for performing data communication with a communication device or communication terminal having a wireless communication interface such as NFC (Near Field Communication), Bluetooth (registered trademark, omitted below), or Wi-Fi (registered trademark, omitted below). The CMOS sensor 309 is a type of built-in imaging means that captures an image of a subject under the control of the CPU 301 to obtain image data or video data. The imaging means may be an imaging means configured with a CCD (Charge Coupled Device) sensor or the like instead of a CMOS sensor. The image sensor I/F 310 is a circuit that controls the driving of the CMOS sensor 309.

The network I/F 311 is an interface for data communication using the communication network 100. The touch panel 412 is a type of input means for operating the communication terminal 3 by the user pressing, clicking, tapping, etc. on a predetermined button, icon, etc. arranged on the display 307. The pointing device 313 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. Note that the communication terminal 3 may have an input means of at least one of a touch panel and a keyboard instead of or in addition to the pointing device 313. The media I/F 315 controls the reading or writing (storing) of data from the recording medium 314 such as a flash memory. The external device connection I/F 416 is an interface for connecting various external devices. The external device in this case is, for example, a USB (Universal Serial Bus) memory, etc. The sound input/output I/F 317 is a circuit that processes the input and output of sound signals between the microphone 318 and the speaker 319 according to the control of the CPU 301. The microphone 318 is a built-in circuit that converts sound into an electrical signal, and obtains information using the electrical signal by acquiring voice or sound waves emitted from an external speaker or the like. The speaker 319 is a built-in circuit that converts electrical signals into physical vibrations to produce sounds such as music or voice. The bus line 320 is an address bus, data bus, or the like for electrically connecting each component such as the CPU 301.

<Hardware configuration of information analysis device>
Fig. 5 is a diagram showing an example of the hardware configuration of the information analysis device. As shown in Fig. 5, the information analysis device 5 is constructed by, for example, a computer, and includes a CPU 501, a ROM 502, a RAM 503, an EEPROM 504, a HD (Hard Disk) 505, a HDD (Hard Disk Drive) controller 506, a network I/F 511, a media I/F 515, an external device connection I/F 516, and a bus line 520. These hardware resources are similar to the hardware resources of the CPU 301, ROM 302, RAM 303, EEPROM 304, network I/F 311, media I/F 315, external device connection I/F 316, and bus line 320 of the communication terminal 3, and therefore description thereof will be omitted.

Here, the HD 505 stores various data such as programs. The HDD controller 506 controls the reading and writing of various data from and to the HD 505 under the control of the CPU 501. Here, the information analysis device 5 may have a hardware configuration equipped with an SSD (Solid State Drive) instead of the HD 505 and the HDD controller 506.

The above program may be recorded in a computer-readable recording medium as an installable or executable file, or may be distributed by downloading via a network. Examples of recording media include CD-Rs (Compact Disc Recordable), DVDs (Digital Versatile Disks), Blu-ray Discs (Blu-ray is a registered trademark; omitted below), SD cards, and USB memories. The recording media may be provided domestically or internationally as a program product. For example, the information analysis device 5 realizes the information analysis method according to the present invention by executing the program according to the present invention.

[Functional configuration of the information analysis system]
Next, the functional configuration of the present embodiment will be described with reference to Fig. 6 to Fig. 10. Fig. 6 is a diagram showing an example of the functional configuration of the information analysis system. Fig. 6 shows the communication terminal 3 or the information analysis device 5 shown in Fig. 1 that is related to the processing or operation described later.

<Functional configuration of communication terminal>
Next, the functional configuration of the communication terminal 3 will be described. As shown in Fig. 6, the communication terminal 3 has a transmission/reception unit 31, an operation reception unit 32, an acquisition unit 33, a display control unit 34, a startup unit 36, a registration unit 38, and a memory/readout unit 39. Each of these functional units is a function or means realized by operating any of the hardware resources shown in Fig. 4 by an instruction from the CPU 301 according to a program for the communication terminal 3 expanded from at least one of the ROM 302, the EEPROM 304, and the recording medium 314 to the RAM 303. The communication terminal 3 also has a storage unit 3000 constructed by at least one of the ROM 302, the EEPROM 304, and the recording medium 314 shown in Fig. 4. Furthermore, the storage unit 3000 has installed therein a communication program (communication application) for communicating with each device and each terminal via the communication network 100, a browser application, a video conference application for holding an online conference, an online morning meeting, and the like, and the like.

<<Functional configuration of communication terminal>>
Next, each functional configuration of the communication terminal 3 will be described in detail. The transmission/reception unit 31 of the communication terminal 3 shown in Fig. 6 is mainly realized by the processing of the CPU 301 on the network I/F 311 and the short-range communication I/F 308, and transmits and receives various data (or information) with the information analyzing device 5 via the communication network 100. The transmission/reception unit 31 also transmits the acquired image information and audio information to the information analyzing device. The transmission/reception unit 31 further transmits various information including face image data of each participant and email address of each participant to the information analyzing device 5 as participant information notification. In this embodiment, the transmission/reception unit 31 functions as an example of at least one of the first transmission means and the first reception means.

The operation reception unit 32 is realized mainly by the CPU 301 processing signals generated by various operations received by at least one of the display 307, the touch panel 312, and the pointing device 313. Note that the operation reception unit 32 may use signals generated by various operations received by an input means such as a keyboard instead of the display 307, the touch panel 312, and the pointing device 313. Note that the display 307 and the touch panel 312 may share their hardware resources and functions. Furthermore, the operation reception unit 32 receives operations to start and end an online morning meeting by each participant or leader in an event such as an online morning meeting. In this embodiment, the operation reception unit 32 functions as an example of a reception means.

The acquisition unit 33 is mainly realized by the processing of the CPU 301, and acquires various information received by the transmission/reception unit 31. The acquisition unit 33 also acquires facial image data and speech voice data of each participant in an online morning meeting, etc., using the respective hardware resources. Here, facial image data is an example of image information (visual information), and speech voice data is an example of audio information. The acquisition unit 33 further acquires report data (an example of a status notification) including predetermined content transmitted by the information analysis device 5 via the transmission/reception unit 31. In this embodiment, the acquisition unit 33 functions as an example of an acquisition means.

The display control unit 34 is mainly realized by the processing of the CPU 301 on the display 307, and controls the display of various screens and information (data) in the communication terminal 3. The display control unit 34 also displays report data related to the status of the multiple participants transmitted by the information analysis device 5 on the display 307. In this embodiment, the "status of the participants" refers to the facial expression detection state of each of the multiple participants based on at least one of the number of nods and the number of smiles of the participants, and the activity state of the multiple participants as a whole based on the facial expression detection state of each of the multiple participants. Here, the facial expression detection state includes the smile detection state and the nod detection state described later. Furthermore, the activity state of the entire participants includes the group activity level described later. In this embodiment, instead of or in addition to the display control unit 34, the display control unit 54 of the information analysis device 5 may have a function of displaying a screen corresponding to the selection of the first option and the second option on the analysis result screen shown in the report data on the display 307 (an example of a display means). The display control unit 34 also displays a display screen generated by HTML or the like on the display 307 using a browser, for example. In this embodiment, the display control unit 34 functions as an example of a display control means.

The launch unit 36 is mainly realized by the processing of the CPU 301, and launches various applications including a video conferencing application in the communication terminal 3. In this embodiment, the launch unit 36 functions as an example of a launching means.

The registration unit 38 is mainly realized by the processing of the CPU 301, and registers various information in the communication terminal 3. In this embodiment, the registration unit 38 functions as an example of a registration means.

The memory readout unit 39 is realized mainly by the processing of the CPU 301 on at least one of the ROM 302, the EEPROM 304, and the recording medium 314, and stores various data (or information) in the memory unit 3000 and reads various data (or information) from the memory unit 3000. In this embodiment, the memory readout unit 39 functions as an example of a memory readout means.

<Functional configuration of information analysis device>
Next, the functional configuration of the information analyzing device 5 will be described. As shown in Fig. 6, the information analyzing device 5 has a transmission/reception unit 51, an acquisition unit 52, an authentication unit 53, a display control unit 54, a judgment unit 55, an analysis processing unit 56, a generation unit 57, a registration unit 58, and a storage/reading unit 59. Each of these functional units is a function or means realized by operating any of the hardware resources shown in Fig. 5 by an instruction from the CPU 501 according to a program for the information analyzing device 5 expanded from at least one of the ROM 502, the EEPROM 504, the HD 505, and the recording medium 514 to the RAM 503. The information analyzing device 5 also has a storage unit 5000 constructed by at least one of the ROM 502, the EEPROM 504, the HD 505, and the recording medium 514 shown in Fig. 5. Furthermore, a communication program (communication application), a browser application, etc. for communicating with the communication terminal 3 via the communication network 100 are installed and stored in the storage unit 5000.

<<Functional configuration of information analysis device>>
Next, each functional configuration of the information analyzing device 5 will be described in detail. The transmitting/receiving unit 51 of the information analyzing device 5 shown in Fig. 6 is mainly realized by the processing of the CPU 501 on the network I/F 511 and the short-range communication I/F 508, and transmits and receives various data (or information) with the communication terminal 3 via the communication network 100. In addition, the transmitting/receiving unit 51 transmits report data (an example of a status notification) related to the status of multiple participants generated by the generating unit 57 described later to the communication terminal 3. In this embodiment, the transmitting/receiving unit 51 functions as an example of at least one of the second transmitting means and the second receiving means.

The acquisition unit 52 is mainly realized by the processing of the CPU 501, and acquires, for example, a posted document (file) transmitted (posted) by the communication terminal 3. The acquisition unit 52 also acquires various information including facial image data (image information) of each participant, voice data (voice information) related to speech, and email addresses of each participant as participant information notifications transmitted by the communication terminal 3 via the transmission/reception unit 51. The acquisition unit 52 also acquires either image information and voice information transmitted by a communication terminal 3 used by a specific participant among the multiple participants, or image information and voice information transmitted by a communication terminal 3 used by each of the multiple participants. The image information includes "nodding" and "smile" as facial expression images of the participants. The voice information includes "statement content" and "words of gratitude" as the participant's voice. In this embodiment, the acquisition unit 52 functions as an example of an acquisition means.

The authentication unit 53 is mainly realized by the processing of the CPU 501, and performs authentication processing based on the facial image data and email address of each participant (member) transmitted by the communication terminal 3. The authentication unit 53 also performs password management and user management for each participant when an online event is held. In this embodiment, the authentication unit 53 functions as an example of an authentication means and an example of a processing means.

The display control unit 54 is mainly realized by the processing of the CPU 501 on the display 307 of the communication terminal 3, and controls the display of various screens and information (data) on the communication terminal 3. The display control unit 54 also uses, for example, a browser to display a display screen generated by HTML or the like on the display 307 of the communication terminal 3. In this embodiment, the display control unit 54 functions as an example of a display control means.

The judgment unit 55 is mainly realized by the processing of the CPU 501, and performs various judgment processes in the entire information analysis device 5. In this embodiment, the judgment unit 55 functions as an example of a judgment means.

The analysis processing unit 56 is mainly realized by the processing of the CPU 501, and performs a series of information analyses using a machine learning model in the information analysis device 5. The analysis processing unit 56 also analyzes the states of the multiple participants based on the image information and voice transmitted by the communication terminal 3. The analysis processing unit 56 also analyzes the states of the multiple participants based on at least one of the information related to nods and the information related to smiles, and the information related to speech, for each of the multiple participants. The analysis processing unit 56 further analyzes the states of the multiple participants by performing an analysis using a machine learning model on at least one of the information related to nods and the information related to smiles. The analysis processing unit 56 further analyzes the states of the multiple participants based on either the image information and voice information transmitted by the communication terminal 3 used by a specific participant among the multiple participants, or the image information and voice information transmitted by the communication terminal 3 used by each of the multiple participants. In this embodiment, the analysis processing unit 56 functions as an example of an analysis processing means. Note that the analysis processing unit 56 may use the above-mentioned AWS.

The generating unit 57 is mainly realized by the processing of the CPU 501, and generates various information (data) in the information analyzing device 5. The generating unit 57 also generates report data (an example of a status notification) as a status notification that can be displayed on the communication terminal 3, based on the analyzed statuses of the multiple participants. The generating unit 57 also generates a status notification based on the facial expression detection status and activity status of each of the multiple participants. The generating unit 57 also generates, as the status notification, a chart (in graph form, table form, etc.) as a result of visually expressing the facial expression detection status and activity status of each of the multiple participants, and a comment on the results for a specific participant among the multiple participants. In this embodiment, the generating unit 57 functions as an example of a generating means.

The registration unit 58 is mainly realized by the processing of the CPU 501, and registers information in various data tables managed by the information analysis device 5. In addition, the registration unit 58 registers image information of multiple participants participating in a specified event and the email addresses of each participant in a user information management DB (an example of a user information management means) described below that manages user information as participant notification information for the specified event before the specified event is executed. In addition, the registration unit 58 registers user IDs (email addresses) of multiple participants who are registered and authenticated in advance and participate in the specified event, and image information of each participant before the specified event is executed. In this embodiment, the registration unit 58 functions as an example of a registration means.

The memory/read unit 59 is mainly realized by the processing of the CPU 501 on at least one of the ROM 502, the EEPROM 504, the HD 505, and the recording medium 514, and stores various data (or information) in the memory unit 5000 and reads various data (or information) from the memory unit 5000. In this embodiment, the memory/read unit 59 functions as an example of a memory/read means.

●User information management table●
FIG. 7 is a conceptual diagram showing an example of a user information management table. The structure of the data table described below is an example, and is not limited to this. In the storage unit 5000, a user information management DB 5001 configured by a user information management table as shown in FIG. 7 is constructed. In the user information management table, a group (team) name, a leader flag, and a user name are associated with each user ID (email address), stored, and managed. Among these, the user ID is identification information of a participant (including a leader, which is an example of a specific participant) who participates in a predetermined event such as an online morning meeting, and is, for example, an email address. The group (team) name is the name of the group or team to which the participant belongs. The leader flag is a flag indicating whether each user is a leader of a certain group or team, and if it is "True", it indicates that the user is a leader. The user name is the name of the user associated with the user ID. In this embodiment, when "ID" is described hereinafter, it is assumed that it includes the meaning of "identification information".

In this embodiment, the user information management table (user information management DB5001) functions as an example of a user information management means.

●Analysis history management table●
FIG. 8 is a conceptual diagram showing an example of an analysis history management table. Note that the data table described below is an example, and is not limited to this. In the storage unit 5000, an analysis history management DB 5002 configured by an analysis history management table as shown in FIG. 8 is constructed. In the analysis history management table, for each user ID, a group (team) name, date and time, event content, participation flag, report generation flag, undetected time in an image, nodding time, smiling time, speech time, number of times of utterance of words of thanks, and total time are stored and managed in association with each other. Among these, the date and time indicates the date and time when an event including an online event was held. The content of the event is a title such as an online morning meeting. The participation flag is a flag indicating whether or not a participant indicated by a user ID participated in the corresponding event, and if it is "True", it indicates that the participant has participated. The report generation flag is a flag indicating whether or not a report for the corresponding event has been generated, and if it is "True", it indicates that the report has been generated. The undetected time in an image is the time in [seconds] during which a participant associated with a user ID was undetected in an image (video) in which the event was recorded. In addition, if the undetected time in the image is excessively longer than the time of other participants, it becomes an indicator for creating a report indicating an alert for that participant. The nodding time indicates the time in [seconds] that the participant associated with the user ID was detected as nodding during the event. The smiling time indicates the time in [seconds] that the participant associated with the user ID was detected as smiling during the event. The remark time indicates the time in [seconds] that the participant associated with the user ID spoke (uttered) during the event. The number of times words of thanks were spoken indicates the number of times the participant associated with the user ID spoke (uttered) words of thanks during the event. The total time is the time from the start to the end of an online morning meeting, etc. In the analysis history management table, each of the above-mentioned management items is accumulated and managed as history for each user ID.

In this embodiment, the analysis history management table (analysis history management DB5002) functions as an example of an analysis history management means.

●Individual vitality management table●
Fig. 9 is a conceptual diagram showing an example of an individual vitality management table. Note that the data table described below is an example and is not limited to this. A individual vitality management DB 5003 configured by an individual vitality management table as shown in Fig. 9 is constructed in the memory unit 5000. In the individual vitality management table, the date and time, the content of the event, the nodding time [seconds], the smiling time [seconds], and the individual vitality are stored and managed in association with each other for each tab, which is represented by a user ID.

Among these, the date and time is the date and time when the event was held. The content of the event is the name and agenda of an online morning meeting, online conference, etc. The event may be held periodically, such as daily, weekly, or monthly. The nodding time is the time in seconds that a nod of each participant in a specified event is detected. The smile time is the time in seconds that a smile of each participant in a specified event is detected. The individual vitality level is a numerical representation of the degree of the facial expression of an individual participant in a specified event, calculated using a machine learning model based on the nodding time and smile time. This individual vitality level may be expressed, for example, as a score from 0 to 20 points.

In this embodiment, the personal vitality management table (personal vitality management DB5003) functions as an example of a personal vitality management means.

●Group activity management table●
Fig. 10 is a conceptual diagram showing an example of a group activity level management table. Note that the data table described below is an example and is not limited to this. A group activity level management DB 5004 configured by a group activity level management table as shown in Fig. 10 is constructed in the memory unit 5000. In the group activity level management table, the constituent members, date and time, and event scores are associated with each tab, which is represented by a group ID, and are stored and managed.

Of these, constituent members indicate the members that make up a group (team). Note that since constituent members represent all members of the group (team), if there are people absent from a specific event held online, they do not necessarily correspond to participants. The event score is the score (morning assembly score) of morning assemblies and the like held online. The event score is a score calculated based on the individual vitality of each constituent member (each participant indicated by each user ID) described above. This event score represents the group activity level in the report sent to the communication terminal 3 described later.

In this embodiment, the group activity management table (group activity management DB5004) functions as an example of a group activity management means.

[Processing or Operation of the Embodiment]
Next, each process or operation in the information analysis system according to the first embodiment will be described with reference to FIGS.

<Participant information notification, member registration and authentication processing>
First, the participant information notification, member registration, and authentication process will be described. Fig. 11 is a sequence diagram showing an example of each process of participant information notification, member registration, and authentication. As shown in Fig. 11, a user who uses the information analysis system 1 starts a web browser by accessing an online conference app or a predetermined URL using a communication terminal 3 (step S11).

Next, the transmitting/receiving unit 31 of the communication terminal 3 transmits a participant information notification to the information analyzing device 5 (step S12). As a result, the transmitting/receiving unit 51 of the information analyzing device 5 receives the participant information notification transmitted by the communication terminal 3. At this time, the participant information notification includes the name of each participant, facial image data of each participant, and email address of each participant. Note that the facial image data of each participant may be facial image data captured in advance for each participant (e.g., photo data of an employee ID card).

Next, the registration unit 58 of the information analysis device 5 executes a member registration process (step S13). Specifically, the registration unit 58 registers the email address of each participant included in the participant information notification received in step S12 in the user ID of the user information management DB 5001 (see FIG. 7).

In addition, the member registration process in step S13 described above may register each participant's information in a specific database on the AWS that is accessible to the information analysis device 5, instead of transmitting the participant information from the communication terminal 3. Furthermore, the member registration process may involve each participant registering themselves, as in commonly known web services, or the leader of each participant may register the members of the group (team) collectively. Furthermore, a service provider that involves information analysis may receive participant information related to each participant and perform user registration.

Then, the authentication unit 53 executes an authentication process (step S14). Specifically, the authentication unit 53 compares the members registered in step S13 with the information of each participant who will participate in the online event to be held, and if there is no registered information, the authentication unit 53 does not provide services related to the online event.

The processing of steps S11-S14 described above may be limited to the initial processing when the online conference app or the web browser for online conferences is launched. Therefore, the communication terminal 3 may assume that the various processing steps described above have been completed and that the two devices are capable of communicating with each other in communication with the information analysis device 5 described below.

<Information analysis processing for online events>
Next, the information analysis process in the online event will be described. Fig. 12 is a sequence diagram showing an example of the information analysis process in the online event. As shown in Fig. 12, the display control unit 34 of the communication terminal 3 displays an online morning assembly screen on the display 307 in response to the start of the online conference app in step S11 (step S101).

●Screen display example●
Here, an example of a display screen in the communication terminal 3 will be described. FIG. 13 is a diagram showing an example of a display screen in the communication terminal when a specific event is held. As shown in FIG. 13, an online morning assembly screen 3111 is displayed on the display 307 of the communication terminal 3 by the display control unit 34. Here, the online conference app started by the communication terminal 3 may be a conference tool used online as is generally known. In this case, a morning assembly (event) to be held based on a user's operation is specified, and an online morning assembly screen corresponding to an operation of an entry (participation) button, an icon, etc. is displayed on the display 307 of the communication terminal 3. On the online morning assembly screen 3111, the face image data (still images or videos) of each participant (participant A, B, C, D, E, F (leader)) participating in the online morning assembly are arranged in Z-shaped order from the top left. In this way, by associating the position of the image of the participant displayed on the communication terminal 3 with the participant information managed by the information analysis device 5, the information analysis device 5 can easily identify the participant included in each image data transmitted from the communication terminal 3. In addition, the task bar displays a video recording button 3551, a sound recording button 3552, a start button 3553, and an end button 3554. Participants can operate these buttons with a cursor or the like to perform each operation of recording, sound recording, starting, and ending the online morning assembly. Each operation can be performed by the leader of the group (team) participating in the online morning assembly on behalf of the leader using the communication terminal 3 used by the leader. On the other hand, each member of the group (team) participating in the online morning assembly can also perform each operation using their own communication terminal 3. In addition, in the online morning assembly screen 3111, the display control unit 34 displays a speech (comment) display mark 3112 in the frame of the speaker who is speaking. This allows visual confirmation of who is speaking (commenting) among the members participating in the online morning assembly. In addition, in the online morning assembly screen 3111, the display control unit 34 may display a facial expression detection monitor (square dotted frame) that follows the facial movements of each participant, and monitor the facial expressions of each participant in real time. In this embodiment, hereinafter, when the term “button” is used, it is intended to include the “icon” displayed on the display 307 .

Returning to FIG. 12, the operation reception unit 32 of the communication terminal 3 receives an operation to start the online morning meeting (step S102). Specifically, the operation reception unit 32 receives an operation by each participant or the leader on the start button 3553 shown in FIG. 13.

Next, the acquisition unit 33 and the memory readout unit 39 acquire and store the image data and voice data of the participants (step S103). Specifically, the acquisition unit 33 of each communication terminal 3 used by each participant who participated in the online morning meeting acquires image data related to an image including the face of each participant using the communication terminal 3 by an image acquisition unit composed of a web camera, etc., and voice data related to the speech of each participant by a voice acquisition unit composed of an audio microphone, etc. Thereafter, the memory readout unit 39 stores and stores the acquired image data and voice data in a specified area of the memory unit 3000. At this time, the memory readout unit 39 may store and store the image data and voice data in the communication terminal 3 used by each participant, respectively, or may store and store them collectively in the communication terminal 3 used by the leader. As an alternative method, the memory readout unit 39 may store and store the image data and voice data using a dedicated database instead of a specified area of the memory unit 3000. In addition, while the online morning meeting is being held, the memory readout unit 39 of the communication terminal 3 may repeatedly execute the process of step S103 at a predetermined time interval (e.g., every 5 seconds) using the image acquisition unit and audio acquisition unit described above until the next online morning meeting operation end operation is accepted (executing a loop process).

Next, the operation reception unit 32 receives an operation to end the online morning meeting (step S104). Specifically, the operation reception unit 32 receives an operation on the end button 3554 shown in FIG. 13.

Next, the memory readout unit 59 reads out the image data and audio data of the participants (step S105). Specifically, the memory readout unit 59 reads out the image data and audio data saved in step S103 from a predetermined area of the memory unit 3000 of each communication terminal 3. At this time, the memory readout unit 59 converts the read out image data into a predetermined image (video) data format (e.g., mp4) and converts the audio data into a predetermined audio data format (e.g., wav). Thereafter, the memory readout unit 59 cooperates with the generation unit 57 to generate a folder for each of the format-converted image data and audio data for each of the user IDs of the participants.

Next, the transmission/reception unit 31 transmits (uploads) the image data and audio data of the multiple participants acquired in step S105 to the information analysis device 5 (step S106). As a result, the transmission/reception unit 51 of the information analysis device 5 receives the data transmitted by the communication terminal 3. At this time, the received data includes image data and audio data of all the people to be uploaded, and a folder created by the email address of each participant. Note that the data to be uploaded may be data recorded during the morning meeting using an online conference app, or may be data recorded using an online conference tool.

Then, the storage/reading unit 59 of the information analysis device 5 performs a data saving process (step S107). Specifically, the storage/reading unit 59 stores and saves the image data and audio data of all the people to be uploaded that were received in step S106 in a specified area of the storage unit 5000. More specifically, the storage/reading unit 59 adds the received values to a specified job queue, and when the specified time arrives, checks the group (team) list and the result index, and adds the job queue. At this time, the queue attributes include the user ID, group (team) name, date and time of addition, and attendance information of participants.

Next, the analysis processing unit 56 executes information analysis processing (step S108). Specifically, the analysis processing unit 56 analyzes the status of multiple participants who have participated in a specific event held online, based on the image information (image data) and audio information (audio data) transmitted by the communication terminal 3. Thereafter, the generation unit 57 generates report data (an example of a status notification) as a status notification that can be displayed on the communication terminal 3, based on the analyzed status of the multiple participants. The information analysis processing of step S108 will be described in detail below.

<<Details of information analysis processing>>
Next, the details of the information analysis process will be described. FIG. 14A and FIG. 14B are flowcharts showing details of the information analysis process in an online event. Note that each process shown below is an example of the information analysis process, and is not limited thereto. In FIG. 14A, first, the acquisition unit 52 reads the acquired image data (recorded data) and audio data (audio recording data) via the storage and reading unit 59, and acquires a copy image and a copy audio (step S108-1). Note that the copy image data representing the acquired copy image is used to detect at least one of the number of nods, the number of smiles, and the time, which will be described later. In addition, the copy audio data representing the acquired copy audio is used for the speech ratio, processing of whether or not the speaker has spoken, and detection of words of thanks, which will be described later.

Next, the analysis processing unit 56 detects the facial key points of each participant using a machine learning model for the copy image data representing the acquired copy image (step S108-2). Specifically, the analysis processing unit 56 detects the facial key points of each participant using a video analysis package (e.g., FaceMesh) that is commonly known for extracting faces for the copy image data representing the copy image.

Then, the analysis processing unit 56 detects smiles using a machine learning model (step S108-3). Specifically, the analysis processing unit 56 detects smiles from the facial images of each participant using the facial key points detected in step S108-2 and the trained neural network.

After executing the process of step S108-2, the analysis processing unit 56 calculates the head movement of each participant as a separate process (step S108-4). Specifically, the analysis processing unit 56 calculates the head movement of each participant based on the facial key points detected in step S108-2.

Next, the analysis processing unit 56 detects nodding using a machine learning model (step S108-5). Specifically, the analysis processing unit 56 detects nodding from the facial images of each participant using the calculated value of the head movement detected in step S108-4 and the trained neural network.

After executing the process of step S108-1, the analysis processing unit 56 executes a noise reduction process as a further process (step S108-6). Specifically, the analysis processing unit 56 executes a noise reduction process on the copy audio data representing the acquired copy audio.

Then, the analysis processing unit 56 converts the voice data into text (step S108-7). Specifically, after performing the noise reduction process of step S108-6, the analysis processing unit 56 performs, for example, a process of transcribing commonly known voice, sign language, etc. into text.

Then, the analysis processing unit 56 cooperates with the generation unit 57 to generate speech section data (step S108-8), and then returns to the processing of step S108-2.

After executing the process of step S108-7, the analysis processing unit 56 further executes separate processes of detecting the number of thank yous (step S108-9) and detecting the speech ratio (step S108-10).

The number of smiles, nods, and thank yous, and the speech ratio detected in each of the processes in steps S108-3, S108-5, S108-9, and S108-10 described above may be temporarily stored in a specified area of the memory unit 5000.

After executing each process of steps S108-3, S108-5, S108-9, and S108-10, the analysis processing unit 56 analyzes and calculates the individual vitality level (step S108-11). Specifically, the analysis processing unit 56 first executes the above-mentioned smile detection process, nod detection process, thank number detection process, and speech ratio detection process, and then searches the analysis history management DB 5002 (see FIG. 8) using the user ID (email address) of each participant as a search key to read out the corresponding smile time and nod time. After that, the analysis processing unit 56 analyzes and calculates the individual vitality level between each of the read values as reference values in the past history of each participant and each of the values detected in the above-mentioned steps S108-3 and S108-5. More specifically, when analyzing and calculating the individual vitality level based on the smile of a certain participant, for example, the analysis processing unit 56 refers to the smile time and nod time of the participant managed in the analysis history management DB 5002. At this time, it is preferable for the analysis processing unit 56 to refer to the duration of smiling and nodding in the same or similar event (past "This week's work progress") as the event currently being analyzed (e.g., "This week's work progress"). Using this method as an example, the analysis processing unit 56 analyzes and calculates each of the smiling and nodding of each participant using the above-mentioned machine learning model (trained neural network). The memory reading unit 59 then stores and saves the calculated individual vitality level in the corresponding individual vitality level item in the individual vitality level management DB 5003 (see FIG. 9). The information analysis device 5 then transitions to the process from (circle A) onwards.

Following (circle A) in FIG. 14B, the judgment unit 55 judges whether all the team members are present (step S108-12). Specifically, the judgment unit 55 searches the user information management DB 5001 (see FIG. 7) via the memory/read unit 59 using the email addresses of each participant received in step S106 as search keys. The judgment unit 55 then compares all the participants of the group (team) participating in the event to be analyzed with the user IDs registered in the user information management DB 5001 and belonging to that group (team), and judges whether all the participants are present, excluding those known to be absent.

If it is determined that all team members are present (step S108-12: YES), the analysis processing unit 56 analyzes and calculates the group activity level (step S108-13). Specifically, the analysis processing unit 56 analyzes and calculates the group activity level between the number of words of thanks detected in step S108-9 and the number of utterances, the duration of utterances, and the speech ratio detected in step S108-10. More specifically, the analysis processing unit 56 analyzes and calculates the group activity level based on the individual vitality level calculated in step S108-11, and the number of utterances, the duration of utterances, and the calculated speech ratio managed in the analysis history management DB 5002. Note that when calculating the group activity level, it is preferable to refer to the number of words of thanks, the number of utterances, the duration of utterances, and the speech ratio in the same or similar events (past "this week's work progress") as the event currently being analyzed (for example, "this week's work progress"). Using this method as an example, the analysis processing unit 56 uses the above-mentioned machine learning model (trained neural network) to analyze and calculate the group activity level based on the analysis results for all team members. The memory reading unit 59 then stores and saves the calculated group activity level in the corresponding event score item of the group activity level management DB 5004 (see FIG. 10).

Next, the generating unit 57 generates UI screen data (report data) according to the calculated group activity level (step S108-14). Specifically, the generating unit 57 refers to the individual activity level management DB 5003 (see FIG. 9) and the group activity level management DB 5004 (see FIG. 10) to generate report data corresponding to the individual activity level calculated in step S108-11 and the event score (group activity level) calculated in step S108-13. This report data includes feedback comments to each participant who participated in the online morning meeting or the leader of the group (team). The feedback comments included in the report data may be comments appropriately generated by the information analysis device 5 according to the calculated scores, or may be selected from a plurality of comments prepared in advance. Thereafter, the memory reading unit 59 stores the generated report data in a predetermined area of the memory unit 5000 and exits this flow. At this time, the analysis processing unit 56 may delete the completed target job from the job queue and then exit this flow. The information analysis process, including the judgment process in step S108-12, is then performed each time each individual's data is registered in the batch process.

If it is determined that not all team members have been gathered (step S108-12: NO), the analysis processing unit 56 waits for a predetermined time (step S108-15), and then the judgment unit 55 judges again whether all team members have been gathered (step S108-16).

If all team members are present after waiting for the specified time (step S108-16: YES), the judgment unit 55 executes the processes of steps S108-13 and S108-14 described above and exits this flow.

On the other hand, if the entire team is not present even after waiting for a predetermined time (step S108-16: NO), the analysis processing unit 56 analyzes and calculates the group activity level excluding the absent members (step S108-17). Specifically, the analysis processing unit 56 analyzes and calculates the group activity level for the currently present participants based on the number of words of thanks detected in step S108-9 and the number of utterances, the duration of utterances, and the speech ratio detected in step S108-10. More specifically, the analysis processing unit 56 analyzes and calculates the group activity level for the currently present participants based on the individual vitality level calculated in step S108-11, and the number of utterances, the duration of utterances, and the calculated speech ratio managed in the analysis history management DB 5002. The method of processing at this time is the same as the method in step S108-13 described above, so a description thereof will be omitted.

Next, the generation unit 57 generates UI screen data (report data, e.g., PDF) according to the group activity level excluding absent members (step S108-18). The generation of the report data in step S108-18 may be the same as the method described in step S108-14. Thereafter, the storage/readout unit 59 stores the generated report data in a specified area of the storage unit 5000 and exits this flow.

In addition, in the above-mentioned step S108-15, the information analysis device 5 may set a predetermined deadline (predetermined time) in advance, and after waiting for the deadline, a participant who has no data may be regarded as an absentee, and the processing from step S108-17 onwards may be executed. Furthermore, instead of the judgment processing of step S108-16 performed after waiting for the predetermined time in step S108-15, the information analysis device 5 may repeat the judgment processing of step S108-16 for a predetermined time (perform a loop processing).

Returning to FIG. 12, the transmission/reception unit 51 of the information analyzing device 5 transmits the report data for each case generated in step S108 to the communication terminal 3 (step S109). As a result, the transmission/reception unit 31 of the communication terminal 3 receives each report data transmitted by the information analyzing device 5.

The process of sending the report from the information analysis device 5 in step S109 may be performed while the event is being held (before the event ends).

Furthermore, instead of transmitting the report to the communication terminal 3 in step S109, the information analyzing device 5 may generate a report in the following manner in step S108 described above. For example, the information analyzing device 5 may generate a report that can be displayed on a web browser operated on the communication terminal 3 in response to access to a specific web browser by a participant (user) using the communication terminal 3. More specifically, after generating the report, instead of automatically sending the report to a participant (e.g., a leader) by email or the like, the information analyzing device 5 may save the generated report on the web browser and display the generated report on the web browser when the leader accesses the web browser. In that case, for example,
After analyzing the online morning meeting held online, the information analysis device 5 or a dedicated server generates and stores the generated report.
- Users, including participants who attended the online morning meeting, access a web browser.
A report is sent from a database (DB) managed by the information analysis device 5 or a dedicated server, and the report is displayed on the Web browser.
A report generated according to the above procedure may be provided and displayed.

Next, the display control unit 34 of the communication terminal 3 causes the report data received in step S109 to be displayed on the display 307 (step S110). At this time, the display control unit 34 may display the email sent by the information analysis device 5 as a report on the display 307 of the communication terminal 3, or may display the contents of the report on a web browser.

●Screen display example●
An example of a report display screen in the communication terminal 3 according to this embodiment will be described below. For the screen display in the communication terminal 3, a method of displaying a predetermined image on the display 307 of the communication terminal 3 using an existing screen display method may be adopted. The report included in the UI screen example shown in FIG. 15 and subsequent figures presents an overall overview, such as the morning assembly score and vitality score (individual vitality level) in the online morning assembly, and then presents details such as the nods and smiles of each participant. Note that the contents and display forms of the reports shown in FIG. 15 to FIG. 22 are merely examples and are not limited thereto.

Morning assembly score (group activity level) FIG. 15 is a diagram showing an example of a report display screen (morning assembly score) in a communication terminal. As shown in FIG. 15, a morning assembly score screen 3121 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. The morning assembly score screen 3121 displays the group activity level (morning assembly score) based on the individual energy level of each participant calculated in step S108-11. The breakdown is a nod score, a smile score, a speaker score, a speech ratio score, and a gratitude score, each of which is expressed, for example, as a score out of a maximum of 20 points. Then, as an overall evaluation score, a total score out of the maximum score (maximum of 100 points) is displayed. The morning assembly score screen 3121 further includes a comment on this score as an overall evaluation. As a result, each participant or a leader of a group (team) who receives this report can use the notified score and overall evaluation as a reference for the next online morning assembly of the group (team). In Fig. 15, the group activity level is expressed as a score, but it may be expressed as a symbol, mark, etc. instead of a score. For example, marks or symbols such as sunny, cloudy, rainy, etc. may be used. Such a display form is applicable to all things displayed as a score hereafter.

○ Energy score (individual vitality level)○ FIG. 16 is a diagram showing an example of a report display screen (vitality score) in a communication terminal. As shown in FIG. 16, a vitality score screen 3131 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. The vitality score screen 3131 displays a vitality score based on the individual vitality level calculated in step S108-11. The breakdown is the vitality score of each participant in an event (for example, a morning assembly) held on a certain date and time, and is expressed, for example, as a score out of 10 points. The vitality score screen 3131 further includes a comment on the score of each participant as an overall evaluation. This allows each participant or the leader of a group (team) who receives this report to grasp the individual vitality level of each participant at once. In particular, if there is a participant with a low score, the leader can issue an alert to that participant to encourage them to speak to him or her.

In addition, in Figures 15 and 16, from the perspective of protecting personal information, instead of returning the personal information of each member directly to all participants or to the leader, it is also possible to return only the score of the entire team and the results of each participant.

Past online morning assembly scores FIG. 17 is a diagram showing an example of a report display screen (past online morning assembly scores) in a communication terminal. As shown in FIG. 17, a past morning assembly score screen 3141 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. The past morning assembly score screen 3141 displays the past morning assembly scores of a certain participant based on the individual vitality calculated in step S108-11. The breakdown is displayed on a scale of 20 points for the nod score, smile score, speaker score, speech ratio score, and gratitude score of the group (team) in the past several morning assemblies including the latest morning assembly day. The results on the past morning assembly score screen 3141 may be displayed in a chart (graph format, table format, etc.) that corresponds the date and time of the past online morning assembly and each score. This allows each participant or group (team) leader who receives this report to grasp the scores of each participant in the past online morning assembly at once, and also makes it possible to estimate changes in each participant due to factors such as changes in date and time and the content of the morning assembly.

Nodding score FIG. 18 is a diagram showing an example of a report display screen (when nodding is detected) in a communication terminal. As shown in FIG. 18, a nodding score screen 3151 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. The nodding score screen 3151 displays the number of noddings and the nodding scores of participants who nod four or more times per minute based on the individual vitality calculated in step S108-11 in a graph and a score. The upper graph on the nodding score screen 3151 shows the results of the conference on the analyzed day, and the lower graph shows the results of the comparison with the past. The nodding score screen 3151 further provides feedback on comments according to the nodding scores and comments based on the comparison with the past. This allows each participant or group (team) leader who receives this report to easily understand and compare the results of the event on the analyzed day with those of past events. In this embodiment, the number of noddings is counted, but the analysis may be performed using nodding time instead of the number of noddings.

Smile Score FIG. 19 is a diagram showing an example of a report display screen (when a smile is detected) in a communication terminal. As shown in FIG. 19, a smile score screen 3161 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. The smile score screen 3161 displays the number of smiles and smile scores of participants who have smiled twice or more based on the individual vitality calculated in step S108-11 in a graph and a score. The upper graph on the smile score screen 3161 shows the results of the meeting on the analyzed day, and the lower graph shows the results of a comparison with the past. The smile score screen 3161 also provides feedback on comments according to the smile score and comments based on a comparison with the past. This allows each participant or group (team) leader who receives this report to easily understand and compare the results of the event on the analyzed day with those of past events. In this embodiment, the number of smiles is counted, but the analysis may be performed using the time spent smiling instead of the number of smiles.

○ Speech ratio ○ FIG. 20 is a diagram showing an example of a report display screen (speech ratio) in a communication terminal. As shown in FIG. 20, a speech ratio score screen 3171 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. The speech ratio score screen 3171 displays the speech time of participants other than the facilitator (e.g., leader) as a graph and a score as a ratio of the facilitator's speech ratio based on the individual vitality calculated in step S108-11. The upper graph in the speech ratio score screen 3171 shows the results of the conference on the analyzed day, and the lower graph shows the comparison result with the past. The speech ratio score screen 3171 further provides feedback on comments according to the score of the speech ratio and comments based on the comparison with the past. This allows each participant or the leader of the group (team) who receives this report to easily grasp and compare the relationship of the speech time of participants other than the facilitator.

Speaker Score FIG. 21 is a diagram showing an example of a report display screen (speaker) in a communication terminal. As shown in FIG. 21, a speaker score screen 3181 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. The speaker score screen 3181 displays the relationship between the number of speakers based on the individual vitality calculated in step S108-11 with ○ and × notation. The table in the upper part of the speaker score screen 3181 shows the results of the conference on the analyzed day, and the graph in the lower part shows the results of comparison with the past. The speaker score screen 3181 further provides feedback on comments according to the speech (comment) results of each participant and comments based on the comparison with the past. This allows each participant or group (team) leader who receives this report to easily grasp the speech situation of each participant and easily compare the speech situations between the participants. Also, if a greeting at the beginning of a meeting were counted as an utterance, almost all participants would be counted as speakers, so a threshold is set for the duration of each utterance (for example, 6 seconds). This prevents simple greetings or short responses to questions from being recognized as utterances. In this embodiment, the speaker scores are displayed as O or X marks, but they may also be displayed as a bar graph or the like.

〇Thanks Score〇 FIG. 22 is a diagram showing an example of a report display screen (when words of thanks are detected) in a communication terminal. As shown in FIG. 22, a thanks score screen 3191 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. The thanks score screen 3191 displays the number of thanks expressed based on the individual vitality calculated in step S108-11 in a graph and a score. In this case, the thanks may be, for example, words such as "Thank you,""It helped me," and "Thanks to you," and may include words of thanks in not only Japanese but also in foreign languages. The table in the upper part of the thanks score screen 3191 shows the results of the conference on the analyzed day, and the graph in the lower part shows the results of a comparison with the past. The thanks score screen 3191 further provides feedback on comments according to the number of thanks expressed between each participant and comments based on a comparison with the past. This allows each participant or group (team) leader who receives this report to easily grasp the degree of gratitude of each participant.

In the information analysis system according to this embodiment, for example, when the processes of steps S106 and S109 described above are executed, other devices, etc. may exist between the communication terminal 3 and the information analysis device 5. In other words, each piece of information (data) transmitted and received between the communication terminal 3 and the information analysis device 5 may be configured to be transmitted and received once via another device, etc. The above-described configuration can be applied even when other processing steps exist between the communication terminal 3 and the information analysis device 5.

[Major Effects of the First Embodiment]
As described above, according to this embodiment, the communication terminal 3 in the information analysis system 1 acquires image data relating to images including the faces of each participant using a webcam or the like, and audio data relating to the speech of each participant using an audio microphone or the like (step S103), and transmits the acquired image data and audio data of the multiple participants to the information analysis device 5 (step S106).
The information analysis device 5 analyzes the states of the multiple participants based on the image data and voice data transmitted by the communication terminal 3, and generates report data related to the analysis results that can be displayed on the communication terminal 3 (step S108). This has the effect of making it possible to grasp the states of the participants based on not only their speech but also their facial expressions, and to utilize information obtained from their speech and facial expressions, even in online events.

Second Embodiment
Next, a second embodiment will be described. In the second embodiment, each communication terminal 3 used by each participant who participates in an online morning meeting, which is an example of an online event, records and audios the video and audio data of each participant, and transmits the video and audio data individually to the information analysis device 5. Therefore, the hardware resources constituting the communication terminal 3 and the information analysis device 5 constituting the information analysis system 1 are the same as those in the first embodiment, and therefore the description will be omitted. Furthermore, the functional configurations of the communication terminal 3 and the information analysis device 5 are also the same as those in the embodiment, and therefore the description will be omitted.

<Information analysis processing for online events>
Next, the information analysis process in the online event will be described. FIG. 23 is a sequence diagram showing an example of the information analysis process in the online event according to the second embodiment. Note that the processes of steps S201 (display of the online morning assembly screen) and S202 (acceptance of the online morning assembly start operation) shown in FIG. 23 are the same as the processes of steps S101 and S102 described in the first embodiment. Note that the online event held in the second embodiment is an online morning assembly as in the first embodiment.

●Screen display example●
Here, an example of a display screen in the communication terminal 3 will be described. FIG. 24 is a diagram showing an example of a display screen in the communication terminal in an online event according to the second embodiment. As shown in FIG. 24, an online morning assembly screen 3211 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. Here, the online conference application started on the communication terminal 3 may be a conference tool used online as is generally known. In this case, the morning assembly (event) to be held based on the operation of each user participating in the online morning assembly is specified, and an online morning assembly screen corresponding to the operation of an entry (participation) button, an icon, etc. is displayed on the display 307 of each communication terminal 3. The online morning assembly screen 3211 displays face image data (still image or video) of participant A participating in the online morning assembly. At this time, the images of the other participants B, C, D, E, and F are displayed next to the image of participant A in a size smaller than the face image of participant A. Therefore, in this display form, the facial expression detection monitor is started only for the facial expression of participant A. Also, the task bar displays a video recording button 3651, a sound recording button 3652, a start button 3653, and an end button 3654. Each participant can operate these buttons with a cursor or the like to record, record, start, and end the online morning assembly at their own timing. Also, on the online morning assembly screen 3211, the display control unit 34 may display the frame of the speaker who is speaking in a thicker frame or with a blinking frame. This allows the user to visually confirm who is speaking (uttering) among the members participating in the online morning assembly.

Returning to FIG. 23, the acquisition unit 33 and the memory readout unit 39 acquire and store image data and voice data of the participants (step S203). Specifically, the acquisition unit 33 of each communication terminal 3 used by each participant who participated in the online morning assembly acquires image data related to an image including the face of the participant using the communication terminal 3 by an image acquisition unit composed of a web camera or the like, and voice data related to the speech of each participant by a voice acquisition unit composed of a voice microphone or the like. Thereafter, the memory readout unit 39 stores and saves the acquired image data and voice data in a predetermined area of the memory unit 3000. Note that the memory readout unit 39 of the communication terminal 3 may repeatedly execute the process of step S203 at a predetermined time interval (e.g., 5-second intervals) while the online morning assembly is being held, using the above-mentioned image acquisition unit and voice acquisition unit, until the next online morning assembly operation end operation is accepted (execution of loop processing).

The subsequent processing in step S204 is similar to the processing in step S104 described in the first embodiment.

Next, the memory readout unit 59 reads out the image data and audio data of the participants (step S105). Specifically, the memory readout unit 59 reads out the image data and audio data saved in step S203 from a predetermined area of the memory unit 3000 of each communication terminal 3. At this time, the memory readout unit 59 converts the read out image data into a predetermined image (video) data format (e.g., mp4) and converts the audio data into a predetermined audio data format (e.g., wav). Thereafter, the memory readout unit 59 cooperates with the generation unit 57 to generate a folder for each of the format-converted image data and audio data for each of the user IDs of the participants.

Next, the transmitting/receiving unit 31 of each communication terminal 3 transmits (uploads) the data acquired in step S205 to the information analyzing device 5 at a timing of each communication terminal 3's choice (step S206). As a result, the transmitting/receiving unit 51 of the information analyzing device 5 receives each data transmitted by each communication terminal 3. At this time, each received data includes image data and audio data of each participant to be uploaded, and a folder created by the participant's email address. Note that the data to be uploaded may be data recorded during the morning meeting using an online conference app launched by each communication terminal 3, or may be data recorded by an online conference tool.

The processing from steps S207 to S210 is similar to the processing from steps S107 to S110 described in the first embodiment, so a description thereof will be omitted.

The above-described embodiment may be any combination of the image data and audio data acquisition and storage method and data upload method in the first embodiment and the image data and audio data acquisition and storage method and data upload method in the second embodiment.

[Major Effects of the Second Embodiment]
As described above, according to this embodiment, each communication terminal 3 in the information analysis system 1 acquires image data and voice data of the participant himself/herself at a timing of the participant's choice, and uploads the acquired image data and voice data at a timing of the participant's choice to the information analysis device 5. This has the effect of reducing the workload of the leader of the group (team) to which the participants belong, since the leader is not required to acquire image data and voice data or upload data, in addition to the effect of the first embodiment.

[Modifications of the embodiment]
Next, a modified example of the embodiment will be described. The modified example of the embodiment also has the same system configuration, hardware configuration, and functional configuration as those shown in the first and second embodiments, so the description thereof will be omitted. In this modified example, image data and audio data of each participant acquired by the communication terminal 3 is transmitted (uploaded) to the information analysis device 5 during the online event without waiting for the end of the online event.

<Information analysis processing for online events>
Next, the information analysis process in the online event will be described. FIG. 25 is a sequence diagram showing an example of the information analysis process in the online event according to the modified example of the embodiment. Note that the processes of steps S251 (display of the online morning assembly screen) and S252 (acceptance of the online morning assembly start operation) shown in FIG. 25 are the same as the processes of steps S101 and S102 described in the first embodiment. Note that the online event held in the modified example is an online morning assembly as in the first embodiment.

Next, the acquisition unit 33 acquires image data and voice data of the participants (step S253). Specifically, the acquisition unit 33 of each communication terminal 3 used by each participant who participated in the online morning meeting acquires image data related to an image including the face of each participant using the communication terminal 3 by an image acquisition unit composed of a web camera or the like, and acquires voice data related to the speech of each participant by an audio acquisition unit composed of an audio microphone or the like.

Then, the transmitting/receiving unit 31 of the communication terminal 3 transmits (uploads) the data acquired in step S253 to the information analyzing device 5 (step S254). As a result, the transmitting/receiving unit 51 of the information analyzing device 5 receives the data transmitted by each communication terminal 3. At this time, the received data includes image data and audio data of each participant to be uploaded, and a folder created by the participant's email address. Note that the data to be uploaded may be data recorded during the morning meeting using an online conference app launched by each communication terminal 3, or may be data recorded by an online conference tool.

The processing of the next step S255 is similar to the processing of step S107 described in the first embodiment, and therefore will not be described here. Note that the processing of steps S253-S255 may be repeatedly executed at a predetermined time interval (e.g., every 5 seconds) until the next online morning meeting operation end operation is accepted (execution of loop processing).

Next, the operation reception unit 32 of the communication terminal 3 receives an operation to end the online morning meeting (step S256).

Next, the transmitting/receiving unit 31 transmits a morning assembly end notification to the information analyzing device 5 (step S257). As a result, the transmitting/receiving unit 51 of the information analyzing device 5 receives the morning assembly end notification transmitted by the communication terminal 3. At this time, the morning assembly end notification includes end notification button information indicating the end of the morning assembly.

The process from step S258 to step S260 is the same as the process from step S108 to step S110 in the first embodiment, so the explanation is omitted.

[Main Effects of the Modification of the Embodiment]
As described above, according to this embodiment, the communication terminal 3 acquires image data relating to an image including the face of each participant using the communication terminal 3 by an image acquisition unit configured with a Web camera or the like, and audio data relating to the speech of each participant by an audio acquisition unit configured with an audio microphone or the like, and transmits (uploads) the data to the information analysis device 5. As a result, the communication terminal 3 does not need to store the acquired image data and audio data of the participants, which has the effect of making it possible to realize participation in an online event without considering the impact on performance and costs associated with an increase in the storage means of the communication terminal 3, for example.

Third Embodiment
Next, a third embodiment will be described. In the third embodiment, in contrast to the first embodiment, the second embodiment, and the modified example of the embodiment described so far, the information analysis device 5 focuses on the viewpoint of whether data related to each participant who participated in the online morning meeting can be acquired and the viewpoint of handling the analysis results of each participant, and an example of realizing this will be described. Specifically, in the third embodiment, the handling of acquired data related to each participant is set in advance according to the input contents operated on the dashboard of each participant described later. This makes it possible to realize a consensus on the handling of data (information) related to each participant for each of the multiple participants who participated in a specific event held online.

In the third embodiment, the hardware resources constituting the communication terminal 3 and the information analysis device 5 constituting the information analysis system 1 are the same as those in the first embodiment, and therefore a description thereof will be omitted. Furthermore, the functional configurations of the communication terminal 3 and the information analysis device 5 are also the same as those in the embodiment, and therefore a description thereof will be omitted. However, the information analysis device 5 newly manages a personal data usage management DB 5005, which will be described later.

[Functional configuration of the information analysis system]
Next, the functional configuration of this embodiment will be described with reference to Fig. 26 and Fig. 27. Fig. 26 is a diagram showing an example of the overall configuration of an information analysis system according to the third embodiment. Note that some functions of the communication terminal 3 and each functional configuration of the information analysis device 5 are the same as those described in the first embodiment, and therefore description thereof will be omitted.

<<Functional configuration of communication terminal>>
Among the functional components of the communication terminal 3 according to the third embodiment, the display control unit 34 causes the display 307 to display an icon capable of playing an explanatory video for informing each participant of the purpose of analyzing the state of the multiple participants. After the explanatory video is played, the display control unit 34 also causes the display 307 to display an acquisition permission selection unit capable of selecting whether or not to permit acquisition of data for analysis. However, the display control unit 34 may omit displaying the icon capable of playing the explanatory video described above. In that case, the communication terminal 3 may also omit playing the explanatory video. After the analysis is performed by the analysis processing unit 56, the display control unit 34 also causes the display 307 to display a notification reception selection unit capable of selecting whether or not to receive a notification related to the analysis result. The display control unit 34 further causes the display 307 to display a disclosure permission selection unit capable of selecting whether or not to permit disclosure of the analysis result related to a specific participant to the leader of a specific participant.

<<Functional configuration of information analysis device>>
In the information analyzing device 5 according to the third embodiment, a personal data usage management table (personal data usage management DB 5005) is added, which will be described next. This personal data usage management table will be described below.

●Personal data usage management table●
FIG. 27 is a conceptual diagram showing an example of a personal data usage management table according to the third embodiment. Note that the structure of the data table described below is an example, and is not limited thereto. A personal data usage management DB 5005 configured by the personal data usage management table as shown in FIG. 27 is constructed in the storage unit 5000. In the personal data usage management table, items including a user ID, a group (team) name, a user name, permission or denial of data acquisition, receipt of notification, and permission or denial of data disclosure are associated with each tab with an event ID (or event contents) as a tab, and are stored and managed.

Of these, permission to obtain data is an item that manages whether or not acquisition of analysis data related to participants who attended the online morning assembly is permitted. Receipt of notification is an item that manages whether or not notification is received by email or the like when analysis is completed. Permission to disclose data is an item that manages whether or not disclosure of one's analysis results to the leader is permitted. In this way, the personal data usage management table may be used as a data table that manages various permission information related to participants (members) other than the leader of a specified event.

The permission to obtain data, receive notifications, and disclose data is set for each user ID based on the selections and inputs made on the user settings screen displayed on the dashboard of each participant's communication terminal 3, which will be described later.

In this embodiment, the personal data usage management table (personal data usage management DB5005) functions as an example of a personal data usage management means.

<Personal data usage setting processing>
Next, the personal data use setting process will be described. Fig. 28 is a sequence diagram showing an example of the personal data use setting process according to the third embodiment. As shown in Fig. 28, a user (a participant of an online morning meeting) who uses the information analysis system 1 starts a web browser by accessing an online conference app or a specified URL using the communication terminal 3. As a result, the display control unit 34 of the communication terminal 3 displays a login (sign-in) screen on the display 307 of the communication terminal 3 (step S301).

Next, the participants of the online morning assembly perform a login (sign-in) operation. This causes the operation reception unit 32 to accept the login (sign-in) operation by the participants (step S302).

●Screen display example●
Here, an example of a display screen in the communication terminal 3 will be described. FIG. 29 is a diagram showing an example of a sign-in screen in the communication terminal according to the third embodiment. As shown in FIG. 29, a sign-in screen 3221 is displayed on the display 307 of the communication terminal 3 by the display control unit 34. The sign-in screen 3221 displays the title of the morning assembly diagnostic solution started in step S301, input fields for inputting a user name and password, and a sign-in button 3752 for executing sign-in. This allows each participant who plans to participate in the online morning assembly to sign in using their respective communication terminals 3.

Returning to FIG. 28, the display control unit 34 displays a service content introduction video playback icon (step S303). Specifically, the display control unit 34 causes the display 307 to display a screen including the service content introduction video playback icon.

●Screen display example●
FIG. 30 is a diagram showing an example of a video playback screen in a communication terminal according to the third embodiment. As shown in FIG. 30, a service content introduction screen 3222 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. On the service content introduction screen 3222, the title of the morning assembly diagnostic solution started in step S301 and a playback icon (right-facing triangle mark in a square) 3231 for playing the service content video are displayed. Here, the service content video is, for example, a video for informing each participant of the purpose of analyzing the status of multiple participants. Providing the video is one solution (method) that takes into consideration users (participants) who, when various information related to the service content, usage precautions, software license, etc. is given as text information, will proceed with consent without reading to the end. As a result, each participant who plans to participate in the online morning assembly can operate the playback icon 3231 to play the video, and when they finish watching the video, they can operate the "Next" button 3752 to transition to another screen.

Returning to FIG. 28, the display control unit 34 displays the consent registration screen (step S304). Specifically, the display control unit 34 causes the display 307 to display a registration screen related to confirmation and consent.

●Screen display example●
31 is a diagram showing an example of a registration screen related to confirmation and consent in a communication terminal according to the third embodiment. As shown in FIG. 31, a confirmation consent registration screen 3223 is displayed on the display 307 of the communication terminal 3 by the display control unit 34. In the confirmation consent registration screen 3223, the title of the morning assembly diagnostic solution started in step S301, the bibliographic information of the participants (the company name, the surname, the given name), a check box for checking that the usage precautions have been confirmed, and a check box for checking that the software usage contract is agreed to are displayed. As a result, each participant who plans to participate in the online morning assembly can register in the information analysis device 5 that the usage precautions have been confirmed and that the software usage contract is agreed to by operating the "Register" button 3752.

Returning to FIG. 28, the operation reception unit 32 receives the consent registration operation (step S305). Specifically, the operation reception unit 32 receives the input of bibliographic information in the confirmation consent registration screen 3223 shown in FIG. 31, the check input indicating that the usage precautions have been confirmed, the check input indicating that the user agrees to the software license agreement, and the operation of the "Register" button 3752.

Then, the display control unit 34 displays the dashboard screen (step S306). Specifically, starting from the operation of the "Register" button 3752 in the above-mentioned step S305, the display control unit 34 displays the dashboard screen of the participant on the display 307 of the communication terminal 3 of the participant.

●Screen display example●
Fig. 32 is a diagram showing an example of a user setting screen in a communication terminal according to the third embodiment. As shown in Fig. 32, a dashboard screen 3311 is displayed on the display 307 of the communication terminal 3 by the display control unit 34. On the dashboard screen 3311, menu tabs (Report, Schedule, Setting, Sign Out) including the names of each participant and setting screens corresponding to the menu tabs are displayed. In the example shown in Fig. 32, for "Setting" among the menu tabs,
- An area where you can select team names using the pull-down keys, and for each member,
A slide switch button for allowing data acquisition, a slide switch button for receiving email notification when analysis is complete, and a slide switch button for allowing disclosure of one's analysis results to the leader are displayed. The initial state of each selection button is ON ("Allow" or "Receive" in the permission/denial selection). This allows each participant to freely select whether to allow data acquisition, whether to receive notifications, and whether to allow disclosure of analysis results to the leader for a given event such as an online morning meeting, depending on their own physical condition, mental state, etc. on that day.

Returning to FIG. 28 again, the operation reception unit 32 receives a personal data usage setting operation (step S307). Specifically, the operation reception unit 32 receives a selection input of whether or not to allow data acquisition in the dashboard screen 3311 shown in FIG. 32, a selection input of whether or not to allow notification to be received, and a selection input of whether or not to allow disclosure of the analysis results.

Next, the transmission/reception unit 31 transmits a personal data usage setting request to the information analyzing device 5 (step S308). As a result, the transmission/reception unit 51 of the information analyzing device 5 receives the personal data usage setting request transmitted by the communication terminal 3. At this time, the personal data usage setting request includes event information such as the event ID and event name, the email addresses of each participant, and the operation input data input by each participant on the dashboard screen 3311.

Next, the analysis processing unit 56 of the information analysis device 5 performs a personal data usage setting process (step S309). Specifically, the analysis processing unit 56 sets the information included in the personal data usage setting request received in step S308 in the corresponding items managed in the personal data usage management DB 5005 (see FIG. 27) via the storage and reading unit 59. More specifically, the analysis processing unit 56 sets the event information such as the event ID and event name, the email address of each participant, and the operation input data (permission of data acquisition, receipt of notification, and permission of data disclosure) in the corresponding items in the personal data usage management DB 5005.

Next, the transmission/reception unit 51 transmits a personal data usage setting response to the personal data usage setting request received in step S308 to the communication terminal 3 (step S310). As a result, the transmission/reception unit 31 of the communication terminal 3 receives the personal data usage setting response transmitted by the information analysis device 5. At this time, the personal data usage setting response may include a message, a flag, etc. indicating that the personal data usage setting has been completed.

The communication terminals 3 in the personal data usage setting process shown in FIG. 28 are the communication terminals of each participant who will be taking part in a specific event, such as an online morning meeting. Therefore, the process is performed by . In other words, the processes from steps S101 to S310 described above are performed individually by each communication terminal 3.

<Information analysis processing for online events>
Next, the information analysis process in the online event will be described. FIG. 33 is a sequence diagram showing an example of the information analysis process in the online event according to the third embodiment. The process of steps S321-S330 shown in FIG. 33 is similar to the process of steps S101-S110 shown in FIG. 12, and therefore the description will be omitted. However, in the third embodiment, in the process of step S328, a process is added in which the personal data usage setting information set in the personal data usage management DB 5005 (see FIG. 27) in the above-mentioned step S309 is read out, and notification screen data according to the result is generated.

<<Details of information analysis processing>>
Next, the details of the information analysis process will be described. Figures 34A and 34B are flowcharts showing details of the information analysis process in an online event according to the third embodiment. Note that the process of steps S328-1 to S328-11 shown in Figure 34A is the same as the process of steps S108-1 to S108-11 shown in Figure 14A, so the description will be omitted.

The following FIG. 34B is a flowchart showing details of the information analysis process for an online event according to the third embodiment. Note that the process of steps S328-12 to S328-18 shown in FIG. 34B is similar to the process of steps S108-12 to S108-18 shown in FIG. 14A, except for steps S308-14 and S308-18. Therefore, a detailed explanation will be omitted. Note that the process of steps S308-14 and S308-18 will be explained in detail separately.

<Generating report data>
Fig. 35 is a flowchart showing details of the report data generation process according to the third embodiment. Note that each process shown below is an example of the information analysis process, and the third embodiment is not limited to this. As shown in Fig. 35, the information analysis device 5 repeatedly executes the following processes from steps S328-14-1 to S328-14-10 as detailed processes related to step S328-13 shown in Fig. 34B for all participants who attended the event. This also applies to detailed processes related to step S328-18 when there are absent members. Details of step S328-14 will be described below.

Then, the memory readout unit 59 reads out the usage settings for each piece of personal data (step S328-14-2). Specifically, the memory readout unit 59 searches the personal data usage management DB 5005 (see FIG. 27) using the user ID (email address) of the participant who participated in the event as a search key, and reads out the corresponding information on whether data acquisition is permitted, whether notification is received, and whether data disclosure is permitted.

Next, the determination unit 55 determines whether data acquisition is permitted (step S328-14-3). Specifically, the determination unit 55 determines via the memory readout unit 59 whether the item for permitting data acquisition in the information related to the specific participant read out in step S328-14-2 is set to permitted (ON).

If it is determined that data acquisition is not permitted (step S328-14-3: NO), the generation unit 57 generates UI screen data (report data) that does not permit acquisition of the analysis results for the target participant (step S328-14-4), and then proceeds to processing for other participants via step S328-14-10.

On the other hand, if it is determined that data acquisition is permitted (step S328-14-3: YES), the determination unit 55 further determines whether to receive a notification when the analysis is completed (step S328-14-5). Specifically, the determination unit 55 determines via the memory readout unit 59 whether the notification reception item in the information related to the specific participant read out in step S328-14-2 is set to receive (ON).

If it is determined that notification will not be received when the analysis is complete (step S328-14-5: NO), the generation unit 57 generates UI screen data (report data) that does not notify the target participant of the analysis results (step S328-14-6), and then proceeds to processing for other participants via step S328-14-10.

On the other hand, if it is determined that a notification will be received when the analysis is complete (step S328-14-5: YES), the judgment unit 55 further determines via circle B whether to permit notification of the analysis results to the leader (step S328-14-7). Specifically, the judgment unit 55 determines via the memory reading unit 59 whether the item regarding data disclosure permission is set to permission (ON) among the information related to the specific participant read in step S328-14-2.

If it is determined that notification of the analysis results to the leader is permitted (step S328-14-7: YES), the generation unit 57 generates UI screen data (report data) that notifies the target participant and the leader of the analysis results (step S328-14-8), and then proceeds to processing for other participants via step S328-14-10.

On the other hand, if it is determined that notification of the analysis results to the leader is not permitted (step S328-14-7: NO), the generation unit 57 generates UI screen data (report data) that notifies only the target participant of the analysis results (step S328-14-9), and then proceeds to processing for the other participants via step S328-14-10.

In this way, the information analysis device 5 executes the above-mentioned processing of steps S328-14-1 to S328-14-10 for each participant. As a result, the information analysis device 5 repeats the same processing for all participants except the leader who participated in a specific event such as an online morning meeting, and provides each participant or leader with an analysis result according to the information input and set by each participant.

Note that the processing in the flow of FIG. 35 may involve combinations and sequences of judgments other than those described above. This also applies to the detailed processing (subroutine) in step S328-16 shown in FIG. 34B.

In addition, in this embodiment, the flow shown in FIG. 35 may be executed before step S328-1 shown in FIG. 34A. This is expected to have the effect of eliminating the need to perform data acquisition processing for unauthorized users who are not permitted to acquire data at the initial stage, based on the information entered by each participant in FIG. 32.

Returning to FIG. 33, after the processing of step S329, the display control unit 34 of the communication terminal 3 displays the report (step S330). Specifically, the display control unit 34 of the communication terminal 3 causes the display 307 to display the report data related to each participant transmitted by the information analysis device 5.

●Screen display example●
FIG. 36 is a diagram showing an example of a report display screen (energy score) in a communication terminal according to the third embodiment. As shown in FIG. 36, an energy score screen 3411 is displayed by the display control unit 34 on the display 307 of the communication terminal 3. The energy score screen 3411 is different from the energy score screen 3131 according to the first embodiment in that the analysis result of participant B, who has set data acquisition to be prohibited, and the analysis result of participant I, who has set his/her analysis result to be prohibited from disclosure to the leader, are not displayed. Regarding the non-display form, for example, when data acquisition is prohibited, "Acquisition not permitted" may be displayed on the icon of the participant. Also, when disclosure of his/her analysis result to the leader is prohibited, "Disclosure not permitted" may be displayed on the icon of the participant. Furthermore, a comment to the effect that data is not disclosed to participants B and I may be added to the overall evaluation comment. This protects the privacy of participants B and I during the online morning assembly, and the group (team) leader who receives this report will be able to understand that participants B and I do not want the analysis results of this online morning assembly to be collected or disclosed.

[Major Effects of the Third Embodiment]
As described above, according to this embodiment, the intention of participants participating in a predetermined event such as an online morning meeting is confirmed in advance, and the analysis results according to the confirmation result (setting contents) are disclosed or notified only to authorized persons. In addition to the effect of the first embodiment, this has the effect of realizing consensus building regarding the handling of data (information) related to each of multiple participants who participated in a predetermined event such as an online morning meeting held online.

[Supplementary description of the embodiment]
Each function of the above-mentioned embodiment can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a device programmed to execute each function by software, such as a processor implemented by an electronic circuit. This device refers to, for example, a processor, an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), an SOC (System on a chip), a GPU (Graphics Processing Unit), and a conventional circuit module, etc.

So far, we have described an information analysis system, information analysis device, information analysis method, and program according to one embodiment of the present invention, but the present invention is not limited to the above-mentioned embodiment, and can be modified within the scope of what a person skilled in the art can imagine, such as adding, changing, or deleting other embodiments, and any aspect is within the scope of the present invention as long as it provides the functions and effects of the present invention.

■ Summary ■
For example, aspects of the present invention are as follows.

<First aspect>
The information analysis system 1 (an example of an information analysis system; omitted below) as a first aspect has a communication terminal 3 (an example of a communication terminal; omitted below) that can acquire information related to multiple participants who participate in a specific event held online, and an information analysis device 5 (an example of an information analysis device; omitted below) that analyzes the information sent by the communication terminal, the communication terminal 3 has an acquisition unit 33 (an example of an acquisition means; omitted below) that acquires image information and audio information of the multiple participants, and a transmission/reception unit 31 (an example of a first transmission means; omitted below) that transmits the acquired image information and audio information to the information analysis device 5, and the information analysis device 5 has an analysis processing unit 56 (an example of an analysis means; omitted below) that analyzes the status of the multiple participants based on the image information and audio information sent by the communication terminal 3, and a generation unit 57 (an example of a generation means; omitted below) that generates a status notification that can be displayed on the communication terminal 3 based on the analyzed status of the multiple participants.

According to the first aspect, even in online events, it is possible to grasp the state of participants based on not only their speech but also their facial expressions, and to utilize the information obtained from their speech and facial expressions.

<Second aspect>
In the information analysis device 5 in the information analysis system 1 as the second aspect, in the first aspect, the image information represents at least one of information relating to nodding and information relating to smiling faces contained in images of each of the multiple participants during a specified event, and the audio information represents information relating to speech of each of the multiple participants during a specified event, and the analysis processing unit 56 analyzes the state of the multiple participants based on at least one of information relating to nodding and information relating to smiling faces, and information relating to speech, of each of the multiple participants.

According to the second aspect, it is possible to provide information that has a positive effect, such as empathy for each participant and a sense of security for the other person, obtained by at least one of information related to the participant's nod and information related to the participant's smile.

<Third aspect>
In the information analysis device 5 in the information analysis system 1 as the third aspect, in the second aspect, the analysis processing unit 56 analyzes the states of multiple participants by performing analysis using a machine learning model on at least one of information related to nodding and information related to smiling.

According to the third aspect, by using a previously trained learning model, it is possible to obtain more accurate analysis results.

<Fourth aspect>
In the information analysis device 5 in the information analysis system 1 as a fourth aspect, in any of the first to third aspects, the analysis processing unit 56 analyzes the status of multiple participants based on either image information and audio information transmitted by a communication terminal 3 used by a specific participant among the multiple participants, or image information and audio information transmitted by a communication terminal 3 used by each of the multiple participants.

According to the fourth aspect, it becomes possible to increase the variety of ways in which participants can provide (upload) image information and audio information.

<Fifth aspect>
In the information analysis device 5 in the information analysis system 1 as the fifth aspect, in the second or third aspect, the status of the multiple participants represents the facial expression detection state of each of the multiple participants based on at least one of the number of nods and the number of smiles, and the activity state of the multiple participants as a whole based on the facial expression detection state of each of the multiple participants, and a generation unit 57 generates a status notification based on the facial expression detection state and activity state of each of the multiple participants.

According to the fifth aspect, it is possible to provide not only indicators for individual participants, but also the collective activity level of the group (team) to which each participant belongs.

<Sixth aspect>
In the information analysis device 5 in the information analysis system 1 as the sixth aspect, in the fifth aspect, the generation unit 57 generates, as a status notification, a chart (in graph form, table form, etc.) as a result visually representing the facial expression detection status and activity status of each of the multiple participants, and a comment on the results for a specific participant among the multiple participants.

According to the sixth aspect, it is possible to improve the visibility of group activity levels.

<Seventh aspect>
The information analysis device 5 in the information analysis system 1 as the seventh aspect, in any of the first to sixth aspects, has a registration unit 58 (an example of a registration means, omitted hereinafter) that registers image information of multiple participants participating in a specified event and the email addresses of each participant in advance in a user information management DB 5001 (an example of a user information management means, omitted hereinafter) that manages user information, as participant notification information for the specified event, before the specified event is executed.

According to the seventh aspect, by calculating and providing individual vitality and group activity levels only to registered members, it becomes possible to maintain confidentiality of information within an organization.

<Eighth aspect>
The information analysis device 5 in the information analysis system 1 as the eighth aspect, in any of the first to seventh aspects, has a transmission/reception unit 51 (an example of a second transmission means; omitted below) that transmits the generated status notification to a communication terminal 3 (an example of a communication terminal 3; omitted below) that can acquire information related to multiple participants who have participated in a specified event held online.

According to the eighth aspect, feedback for participants who have taken part in an event held online is sent to the communication terminals of the participants, allowing the participants to check the feedback.

<Ninth aspect>
The communication terminal 3 in the information analysis system 1 as a ninth aspect has a display control unit 34 (an example of a display control means; omitted below) that displays the status notification sent by the information analysis device 5 on the display 307 (an example of a display means; omitted below) in any of the first to eighth aspects.

According to the ninth aspect, feedback for participants who have taken part in an online event is displayed on the communication terminals of the participants, thereby ensuring the confidentiality of personal information.

<Tenth aspect>
The information analysis device 5 as a tenth aspect that analyzes information related to multiple participants who participated in a specified event held online has an analysis processing unit 56 (an example of an analysis means; omitted below) that analyzes the status of the multiple participants based on image information and audio information transmitted by a communication terminal capable of acquiring information related to the multiple participants who participated in a specified event held online, and a generation unit 57 (an example of a generation means; omitted below) that generates a status notification that can be displayed on the communication terminal 3 based on the analyzed status of the multiple participants.

According to the tenth aspect, as in the first aspect, even in online events, it is possible to grasp the state of participants based on not only their speech but also their facial expressions, and to utilize the information obtained from their speech and facial expressions.

<Eleventh aspect>
As an eleventh aspect, the information analysis method executed by the information analysis device 5 that analyzes information related to multiple participants who participated in a specified event held online executes processing including an analysis step of analyzing the status of the multiple participants based on image information and audio information transmitted by a communication terminal 3 capable of acquiring information related to the multiple participants who participated in a specified event held online, and a generation step of generating a status notification that can be displayed on the communication terminal 3 based on the analyzed status of the multiple participants.

According to the eleventh aspect, as in the first aspect, even in online events, it is possible to grasp the state of participants based on not only their speech but also their facial expressions, and to utilize the information obtained from their speech and facial expressions.

<Twelfth aspect>
The program as a twelfth aspect causes an information analysis device 5 that analyzes information related to multiple participants who participated in a specified event held online to execute processing including an analysis step of analyzing the status of the multiple participants based on image information and audio information transmitted by a communication terminal 3 that can acquire information related to the multiple participants who participated in a specified event held online, and a generation step of generating a status notification that can be displayed on the communication terminal 3 based on the analyzed status of the multiple participants.

According to the twelfth aspect, as in the first aspect, even in online events, it is possible to grasp the state of participants based on not only their speech but also their facial expressions, and to utilize the information obtained from their speech and facial expressions.

<Thirteenth aspect>
In the ninth aspect, the display control unit 34 of the information analysis device 5 in the information analysis system 1 as the thirteenth aspect causes the display 307 to display an icon capable of playing an explanatory video to communicate to each participant the purpose of analyzing the status of multiple participants.

According to the thirteenth aspect, it is possible to reduce the burden on participants and to help them better understand the purpose of use.

<14th aspect>
In the information analysis system 1 as the fourteenth aspect, the display control unit 34 of the information analysis device 5 in the thirteenth aspect causes the display 307 to display an acquisition permission selection unit that allows the user to select whether or not to allow the acquisition of data for analysis.

According to the 14th aspect, it is possible to more reliably ensure confidentiality of personal information.

<Fifteenth aspect>
In the 14th aspect, the display control unit 34 of the information analysis device 5 in the information analysis system 1 as the 15th aspect displays on the display 307 a notification reception selection unit that allows the user to select whether or not to receive a notification related to the analysis results after analysis is performed by the analysis processing unit 56.

According to the fifteenth aspect, it becomes possible to provide analysis results according to the participant's condition and wishes.

<16th aspect>
In the information analysis system 1 as the 16th aspect, the display control unit 34 of the information analysis device 5 in the 14th aspect displays on the display 307 a disclosure permission selection unit that allows the user to select whether or not to allow the analysis results related to a specific participant to be disclosed to the leader of a specific participant.

According to the 16th aspect, it becomes possible to operate more flexibly with regard to sharing of analysis results.

1 Information analysis system 3 Communication terminal 5 Information analysis device 31 Transmitter/receiver unit (an example of a first receiving means, an example of a first transmitting means)
33 Acquisition unit (an example of an acquisition means)
34 Display control unit (an example of a display control means)
51 Transmitter/receiver unit (an example of a second receiving means, an example of a second transmitting means)
52 Acquisition processing unit (an example of an acquisition means)
56 Analysis processing unit (an example of an analysis means)
57 Generation unit (an example of a generation means)
58 Registration unit (an example of a registration means)
307 Display (an example of a display means)

JP 2022-028539 A

Claims (16)

An information analysis system including: a communication terminal capable of acquiring information related to a plurality of participants who have participated in a predetermined event held online; and an information analysis device that analyzes the information transmitted by the communication terminal,
The communication terminal includes:
An acquisition means for acquiring image information and audio information of the plurality of participants;
a first transmission means for transmitting the acquired image information and audio information to the information analysis device;
having
The information analysis device includes:
an analysis means for analyzing states of the plurality of participants based on the image information and the audio information transmitted by the communication terminal;
a generating means for generating a status notification displayable on the communication terminal based on the analyzed statuses of the plurality of participants;
having
An information analysis system comprising: the image information represents at least one of information related to a nod and information related to a smile included in an image of each of the plurality of participants during the specified event, and the voice information represents information related to an utterance of each of the plurality of participants during the specified event;
The analysis means
Analyzing the states of the plurality of participants based on at least one of the information related to the nods and the information related to the smiles, and the information related to the speech of each of the plurality of participants;
2. The information analysis system according to claim 1 . The analysis means
analyzing at least one of the information related to the nods and the information related to the smiles using a machine learning model to analyze the states of the plurality of participants;
3. The information analysis system according to claim 2. The analysis means
Analyzing states of the plurality of participants based on either the image information and the audio information transmitted by a communication terminal used by a specific participant among the plurality of participants, or the image information and the audio information transmitted by a communication terminal used by each of the plurality of participants;
4. The information analysis system according to claim 1, wherein the information analysis system is a computer-readable storage device. The states of the plurality of participants are:
expressing a facial expression detection state of each of the plurality of participants based on at least one of the number of nods and the number of smiles, and an activity state of the entire plurality of participants based on the facial expression detection states of each of the plurality of participants;
The generating means includes:
generating said status notification based on a facial expression detection state and the activity state of each of said plurality of participants;
4. The information analysis system according to claim 2, wherein the information analysis system is a computer-readable storage medium. The generating means includes:
generating, as the status notification, a chart as a result visually expressing the facial expression detection status and the activation status of each of the plurality of participants, and a comment on the results for a specific participant among the plurality of participants;
6. The information analysis system according to claim 5, The information analysis system according to any one of claims 1 to 3,
The information analysis device further comprises:
a registration means for registering, in advance, the image information of the plurality of participants participating in the predetermined event and email addresses of each of the participants as participant notification information for the predetermined event in a user information management means for managing user information before the predetermined event is executed;
having
An information analysis system comprising: The information analysis system according to any one of claims 1 to 3,
The information analysis device further comprises:
a second transmission means for transmitting the generated status notification to the communication terminal;
having
An information analysis system comprising: The information analysis system according to any one of claims 1 to 3,
The communication terminal further comprises:
a display control means for causing a display means to display the status notification transmitted by the information analyzing device;
having
An information analysis system comprising: An information analysis device that analyzes information related to a plurality of participants who have participated in a predetermined event held online,
an analysis means for analyzing states of the plurality of participants based on the image information and the audio information transmitted by a communication terminal that has acquired the image information and the audio information related to the plurality of participants;
a generating means for generating a status notification displayable on the communication terminal based on the analyzed statuses of the plurality of participants;
having
2. An information analysis device comprising: An information analysis method executed by an information analysis device that analyzes information related to a plurality of participants who have attended a predetermined event held online, comprising:
an analysis step of analyzing states of the plurality of participants based on the image information and the audio information transmitted by a communication terminal that has acquired the image information and the audio information related to the plurality of participants;
generating a status notification displayable on the communication terminal based on the analyzed statuses of the plurality of participants;
Perform a process that includes
1. An information analysis method comprising: An information analysis device that analyzes information related to a plurality of participants who have participated in a specific event held online,
an analysis step of analyzing states of the plurality of participants based on the image information and the audio information transmitted by a communication terminal that has acquired the image information and the audio information related to the plurality of participants;
generating a status notification displayable on the communication terminal based on the analyzed statuses of the plurality of participants;
A program that executes a process including: The display control means
displaying, on the display means, an icon capable of playing an explanatory video for informing each participant of the purpose of analyzing the states of the plurality of participants;
10. The information analysis system according to claim 9. The display control means further
displaying, on the display means, an acquisition permission selection section that allows the user to select whether or not to permit acquisition of data for the analysis;
14. The information analysis system according to claim 13. The display control means
After the analysis is performed by the analysis means, a notification reception selection section is displayed on the display means, allowing the user to select whether or not to receive a notification regarding the analysis result.
15. The information analysis system according to claim 14. The display control means
displaying, on the display means, a disclosure permission selection section that allows a user to select whether or not to permit disclosure of an analysis result related to a specific participant to a leader of the specific participant;
15. The information analysis system according to claim 14.