JP2023047956A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

To improve the usability in remote meetings.SOLUTION: An information processing program according to the present application cause a computer to execute a calculation procedure for calculating, for each of a plurality of speakers, a listening degree indicating the degree of listening to the speech of each of the plurality of speakers by a listener who is listening to the speeches of the plurality of speakers in a remote conference in which a plurality of participants participate, and an auxiliary control procedure for providing an auxiliary function that makes it easier for listeners to hear the speech of a low-attentive speaker whose degree of listening is lower than that of a high-attentive speaker whose degree of listening is high.SELECTED DRAWING: Figure 7

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

2. Description of the Related Art Conventionally, there is known a technology related to a remote conference (teleconference) system in which people in remote locations hold an online conference using a telephone line or an Internet line. For example, in a teleconference system in which a conference is held between a plurality of bases through remote calls, there is known a technique in which each conference participant is provided with rendering processing means for arbitrarily setting the sound image position of the speaker who is heard on the receiving side. ing.

JP-A-2006-279492

However, the conventional technology described above does not necessarily improve usability in a remote conference. For example, in the above conventional technology, the voice of each speaker is virtually freely arranged at each speaker's position on the receiving side. Therefore, it may be difficult for the listener to satisfactorily listen to the speech of a speaker who is relatively not the listener's attention (the listener is not paying attention to) among the plurality of speakers. Therefore, it may not be said that the conventional technology described above has high usability in a remote conference.

An information processing program according to an embodiment, in a remote conference in which a plurality of participants participate, calculates a listening degree indicating a degree of listening to the speech of each of the plurality of speakers by a listener who is listening to the speech of the plurality of speakers. A calculation procedure for calculating each of a plurality of speakers, and assistance for providing an auxiliary function that makes it easier for the listener to hear the speech of the low-attentive speaker whose degree of listening is low compared to the high-attentive speaker whose high degree of listening is. causing a computer to execute a control procedure;

FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment. FIG. 2 is a diagram illustrating a configuration example of an information processing apparatus according to the embodiment; FIG. 3 is a diagram illustrating an example of a screen according to the embodiment; FIG. 4 is a diagram showing an example of three-dimensional arrangement of multiple voices according to the embodiment. FIG. 5 is a diagram for explaining the listening degree according to the embodiment. FIG. 6 is a diagram illustrating an example of an auxiliary function according to the embodiment; FIG. 7 is a diagram illustrating an example of an auxiliary function according to the embodiment; FIG. 8 is a diagram illustrating an information processing procedure according to the embodiment; FIG. 9 is a diagram illustrating an example of an auxiliary function according to a modification; FIG. 10 is a diagram illustrating an example of an auxiliary function according to a modification; FIG. 11 is a diagram illustrating an example of an auxiliary function according to a modification; FIG. 12 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing apparatus.

Embodiments for implementing an information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. The information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. Also, in each of the following embodiments, the same parts are denoted by the same reference numerals, and overlapping descriptions are omitted.

(embodiment)
[1. Configuration example of information processing system]
FIG. 1 is a diagram showing a configuration example of an information processing system 1 according to an embodiment. The information processing system 1 includes an information processing device 100 used by users of the remote conference service, and a distribution server 200 that provides the remote conference service. The information processing device 100 and the distribution server 200 are connected via a predetermined network N so as to be communicable by wire or wirelessly. Note that the information processing system 1 shown in FIG. 1 may include an arbitrary number of information processing apparatuses 100 and an arbitrary number of distribution servers 200 . A case of a web conference (also referred to as an online conference) will be described below as an example of a remote conference.

The information processing device 100 is an information processing device used by users of the web conference service. The information processing apparatus 100 is realized by, for example, a smartphone, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. In addition, hereinafter, among the users of the Web conference service, users participating in a predetermined Web conference are referred to as "participants".

An application for using a web conference service (hereinafter also referred to as a “web conference application”) is installed in the information processing apparatus 100 . The information processing apparatus 100 issues participant identification information (for example, an ID) capable of identifying a user (later participant) who installed the web conference application. In addition to the participant identification information, the information processing apparatus 100 also includes participant information (participant's identity information, participant's role in the conference (“presenter”, “participant”, etc.), device information), which is basic information about the participant. and information about applications, IP addresses, setting keywords, etc.) are sent to the distribution server 200 as metadata.

The distribution server 200 is a server device that provides a web conference service. Specifically, the distribution server 200 receives the participant information of each of the multiple participants from the information processing device 100 of each of the multiple participants. Subsequently, the distribution server 200 generates conference metadata that summarizes the participant information of all the participants. Subsequently, after generating the conference metadata, the distribution server 200 transmits the generated conference metadata to the information processing apparatuses 100 of the respective participants. Further, when the participant information is updated, the distribution server 200 generates updated conference metadata each time, and transmits the updated conference metadata to the information processing apparatuses 100 of the respective participants.

Also, the distribution server 200 acquires the name and organization (company name, department, etc.) of each participant as set keywords based on the participant information of all the participants. Also, the distribution server 200 acquires keywords preset by the participants as set keywords based on the participant information of all the participants. Note that the set keyword is not limited to words, and may be sentences such as phrases. In addition, the distribution server 200 acquires frequently used words in meetings and address words (such as "Would you like a minute?") as setting keywords. It should be noted that the set keyword may be acquired based on the result of machine learning performed by a learning unit provided in the distribution server 200 using keywords set in past web conferences as teacher data. Upon acquiring the set keywords, the distribution server 200 generates a keyword list in which the set keywords of each participant and the participant identification information of each participant are associated with each other. After generating the keyword list, the distribution server 200 transmits the generated keyword list to the information processing devices 100 of the respective participants.

The information processing apparatus 100 also includes devices having various sensor functions such as a camera, a microphone, and a speaker. In the following description, the participant using the information processing device 100 may be referred to as the "principal". For example, the information processing device 100 transmits audio data relating to the participant's (principal's) voice detected by the microphone and the participant's (principal's) image data detected by the camera to the distribution server 200 together with the participant identification information. A case where the image data is a video (also referred to as a moving image) will be described below. Note that the image data may include a still image.

In addition, the distribution server 200 receives audio data relating to the voices of each of the multiple participants participating in the Web conference from the information processing devices 100 of the multiple participants. Subsequently, the distribution server 200 transmits the received voice data to the information processing devices 100 of other participants than the participant (the person himself/herself). Also, the distribution server 200 receives image data of each of the multiple participants from the information processing devices 100 of the multiple participants. Subsequently, the distribution server 200 transmits the received image data to the information processing devices 100 of participants other than the participant (the person himself/herself). When the distribution server 200 does not distribute the image data of the participant (principal), default image data (for example, image data including characters indicating the name or initials of the participant (principal)) or the participant (principal) The registered image data is transmitted to the information processing apparatuses 100 of the other participants.

The information processing apparatus 100 also has a screen, and displays images of other participants received from the distribution server 200 on the screen. Specifically, the information processing apparatus 100 displays on the screen a full-screen image including participant images of each of the plurality of participants in the web conference. Further, the information processing apparatus 100 displays the participant images of each of the plurality of participants in the Web conference in different display areas of the screen.

The information processing apparatus 100 also includes, for example, a plurality of speakers, and outputs the voices of other participants received from the distribution server 200 from each of the plurality of speakers. Specifically, the information processing apparatus 100 outputs the voices of each of the plurality of speakers from each of the plurality of speakers so that the voices of each of the plurality of speakers can be heard from each of the plurality of sound sources arranged at different positions. Here, the number of speakers and the number of speakers may differ. More specifically, among the sound images output from the speakers, the positional relationship of the sound images of the plurality of speakers is localized. For example, even if the sound is localized so that it can be heard from the right speaker, it is possible to make the listener perceive the direction from which the sound is coming by lowering the volume of the left speaker as well or providing a delay. That is, the number of speakers may be less than the number of speakers. Alternatively, the number of speakers may be greater than the number of speakers. In this way, the information processing apparatus 100 arranges the sound sources of the voices of the plurality of speakers at three-dimensionally different positions, thereby making it easier for the listener to distinguish between the voices of the plurality of speakers. Note that the information processing apparatus 100 may include earphones (headphones) instead of the plurality of speakers, and output the voices of other participants received from the distribution server 200 from the earphones (headphones).

For example, at the start of the conference, the information processing apparatus 100 determines the position (also called localization) of the sound source of each speaker's voice (also called localization), volume, and presence/absence of voice processing (reverberation processing, etc.) for each speaker based on the conference metadata. to decide. In addition, during the conference, the information processing apparatus 100 determines the localization of the voices of the plurality of speakers based on the degree of listening that indicates the degree of listening to each of the plurality of speakers by the listener who is listening to the speech of the plurality of speakers. , volume, and the presence or absence of voice processing. In addition, the information processing apparatus 100 can be configured so that the voices of each of the plurality of speakers can be heard from the localization of the voices determined for each voice, with the volume determined for each voice, and in a voice-processed state. Outputs the voice of each speaker. Details of the listening degree will be described later. In addition, the information processing apparatus 100 changes the degree of separation of multiple voices (localization separation, volume, sharpness of reverberation sound) according to user application settings.

Below, the participant identified by the participant ID "U1" may be referred to as "participant U1". Thus, hereinafter, when described as "participant U* (* is an arbitrary number)", it indicates that the participant is identified by the participant ID "U*". For example, when "participant U2" is described, the participant is the participant specified by the participant ID "U2".

Further, hereinafter, the information processing device 100 will be described as the information processing devices 100-1 and 100-2 according to the participants using the information processing device 100. FIG. For example, the information processing device 100-1 is the information processing device 100 used by the participant U1. Further, for example, the information processing device 100-2 is the information processing device 100 used by the participant U2. Further, hereinafter, the information processing apparatuses 100-1 and 100-2 will be referred to as the information processing apparatus 100 when they are not distinguished from each other.

In the above-described embodiment, the remote conference is a web conference, but the remote conference according to this embodiment is not limited to the web conference. For example, the remote conference according to this embodiment may be a video conference or a telephone conference in addition to the web conference.

[2. Configuration example of information processing device]
FIG. 2 is a diagram illustrating a configuration example of the information processing apparatus 100 according to the embodiment. As shown in FIG. 2 , information processing apparatus 100 includes communication section 110 , storage section 120 , display section 130 , audio output section 140 , detection section 150 and control section 160 . Note that the information processing apparatus 100 may have an input unit (for example, a keyboard, a mouse, etc.) that receives various operations from the user of the information processing apparatus 100 or the like.

(Communication unit 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card) or the like. Also, the communication unit 110 is connected to a network N (not shown) by wire or wirelessly, and transmits and receives information to and from the distribution server 200 and other information processing apparatuses 100, for example.

(storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 120 stores various programs (corresponding to an example of an information processing program). For example, the storage unit 120 stores a web conference application program.

In addition, the storage unit 120 stores various data. For example, the storage unit 120 stores conference metadata acquired by the conference control unit 161 . The storage unit 120 also stores the progress information generated by the generation unit 163 . The storage unit 120 also stores the voice data of the participants input to the microphone. The storage unit 120 also stores image data of the participants captured by the camera. The storage unit 120 also stores recorded data of images of participants. The storage unit 120 also stores voice data of other participants acquired by the conference control unit 161 . The storage unit 120 also stores image data of other participants acquired by the conference control unit 161 . The storage unit 120 also stores recorded data of images of other participants acquired by the conference control unit 161 .

(Display unit 130)
The display unit 130 is implemented by an image output device such as a display. The display unit 130 displays various information under the control of the conference control unit 161 or the auxiliary control unit 165 . Note that when the information processing apparatus 100 employs a touch panel, the input unit and the display unit 130 are integrated. Also, in the following description, the display unit 130 may be referred to as a screen.

Specifically, the display unit 130 displays the participant images of each of the plurality of participants in the web conference in different display areas. For example, the display unit 130 displays the participant images of the plurality of participants acquired by the conference control unit 161 in different display areas under the control of the conference control unit 161 .

Moreover, the display unit 130 further displays a full-screen image including participant images of each of the plurality of participants in the web conference. For example, the display unit 130 displays a full-screen image including the participant images of each of the multiple participants acquired by the conference control unit 161 under the control of the conference control unit 161 .

(Audio output unit 140)
Audio output unit 140 is realized by an audio output device such as a speaker. The audio output unit 140 is implemented by, for example, two speakers arranged on the left and right sides of the listener. For example, the audio output unit 140 includes a speaker (hereinafter also referred to as a left speaker) installed at a predetermined distance to the left of the information processing device 100 of the listener and a speaker located at a predetermined distance to the right of the information processing device 100 main body. This is realized by a speaker (hereinafter also referred to as a right speaker) installed at the position. For example, the audio output unit 140 outputs audio through two left and right speakers in a stereo system.

In addition, the audio output unit 140 outputs the voices of the plurality of speakers so that the voices of the plurality of speakers in the Web conference can be heard from the plurality of sound sources arranged at different positions. For example, under the control of the conference control unit 161, the audio output unit 140 outputs a plurality of utterances so that each of the plurality of speakers' voices acquired by the conference control unit 161 can be heard from each of a plurality of sound sources arranged at different positions. output the voice of each person. Also, in the following description, the audio output unit 140 may be referred to as a speaker.

(Detector 150)
The detection unit 150 is realized by various sensor devices. For example, the detection unit 150 is realized by a sound collecting device such as a microphone, which is a sound sensor. The sound sensor collects the voices of the participants and outputs the collected voice data to the control unit 160 . Also, in the following description, the sound sensor may be referred to as a microphone.

Also, the detection unit 150 is implemented by an imaging device such as a camera, which is an image sensor. The image sensor captures images of participants and the like, and outputs captured image data to the control unit 160 . Also, in the following description, the image sensor may be referred to as a camera.

Further, the detection unit 150 detects the line of sight of the listener. Specifically, the detection unit 150 detects the line of sight of the listener using a known line of sight detection technique. For example, the detection unit 150 identifies the listener's line of sight direction based on the positional relationship between the inner corners of the listener's eyes and the iris included in the listener's image acquired by the camera.

Alternatively, detection unit 150 includes an infrared LED and an infrared camera. The detector 150 photographs the listener's face with an infrared camera while the infrared LED illuminates the listener's face. The detection unit 150 may identify the listener's line of sight direction based on the positional relationship between the corneal reflection of the listener's eye and the pupil included in the listener's image acquired by the infrared camera.

(control unit 160)
The control unit 160 is a controller, and for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like controls the information processing apparatus 100. Various programs (corresponding to an example of an information processing program) stored in the internal storage device are executed by using a storage area such as a RAM as a work area.

As shown in FIG. 2, the control unit 160 includes a conference control unit 161, a calculation unit 162, a generation unit 163, an acquisition unit 164, and an auxiliary control unit 165. realize or perform Note that the internal configuration of the control unit 160 is not limited to the configuration shown in FIG. 2, and may be another configuration as long as it performs information processing described later.

(Meeting control unit 161)
The conference control unit 161 activates the web conference application upon receiving the operation of the participant via the input unit. Also, when the web conference application is activated, the conference control unit 161 activates the camera and the microphone. Subsequently, the conference control unit 161 transmits audio data relating to the voice of the participant (principal) detected by the microphone and image data of the participant (principal) detected by the camera to the distribution server 200 together with the participant identification information.

In addition to the participant identification information, the conference control unit 161 also controls the participant information, which is basic information about the participants (personal information of the participants, roles of the participants in the conference (“presenter”, “participant”, etc.), device and information about the application, IP address, etc.) are transmitted to the distribution server 200 as metadata.

Also, the conference control unit 161 acquires conference metadata from the distribution server 200 . For example, the conference control unit 161 acquires conference metadata of a web conference by four participants, participant U11 and participants U21 to U23. After acquiring the conference metadata, the conference control unit 161 determines the layout of the participant images of each of the multiple speakers at the start of the conference, based on the conference metadata.

FIG. 3 is a diagram illustrating an example of a screen according to the embodiment; In FIG. 3, in a Web conference in which four participants, a participant U11 and participants U21 to U23, participate, participants U21 to U23 (hereinafter also referred to as speakers U21 to U23) who are three speakers An example of the screen of the information processing device 100-11 of the participant U11 (hereinafter also referred to as the listener U11) who is listening to the speech is shown. Conference control unit 161 displays a screen as shown in FIG. 3 on display unit 130-11.

In FIG. 3, the conference control unit 161 acquires the participant images G21 to G23 of the three participants U21 to U23 in the web conference from the distribution server 200, respectively. Subsequently, the conference control unit 161 displays the participant images G21 to G23 of the three participants U21 to U23 respectively in different display areas F21 to F23 of the screen. In the example shown in FIG. 3, the conference control unit 161 generates participant images G21 to G23 of the participants U21 to U23 at the start of the conference based on the information indicating the roles of the participants in the conference among the conference metadata. are determined.

For example, the conference control unit 161 determines to arrange the participant image G21 of the participant U21 at the start of the conference in the central display area F21 based on the fact that the role of the participant U21 is "presenter". . The conference control unit 161 also displays an icon G211 indicating the position of the sound source of the participant U21 in the display area F21.

Further, the conference control unit 161 changes the participant image G22 of the participant U22 (participant U23) at the start of the conference based on the fact that the role of the remaining participant U22 (participant U23) is "participant". (Participant image G23) is determined to be arranged in the display area F22 on the right of the center (the display area F23 on the left of the center). The conference control unit 161 also displays an icon G221 (icon G231) indicating the position of the sound source of the participant U22 (participant U23) in the display area F22 (display area F23).

Further, the conference control unit 161 displays on the screen a full-screen image G11 including the participant images G21 to G23 of the participants U21 to U23, respectively. In the full-screen image G11, materials related to the conference are displayed in areas other than the participant images G21 to G23, but drawing is omitted in FIG.

FIG. 4 is a diagram showing an example of three-dimensional arrangement of multiple voices according to the embodiment. In FIG. 4, in a web conference in which four participants, the participant U11 and the participants U21 to U23, participate, three speakers for the participant U11 who is a listener listening to the statements of the three speakers U21 to U23 shows an example of the arrangement of the sound sources of the respective speakers U21 to U23.

In the example shown in FIG. 4, a left speaker 140-11-L is arranged at a predetermined distance to the left of the information processing device 100-11 of the listener U11. A right speaker 140-11-R is arranged at a predetermined distance to the right of the information processing device 100-11 of the listener U11. Participant images G21 to G23 of speakers U21 to U23 are displayed in different display areas on display unit 130-11 of information processing device 100-11. Further, a camera constituting a detection unit 150-11 is installed above the display unit 130-11 of the information processing device 100-11.

In FIG. 4, the conference control unit 161 acquires from the distribution server 200 the voice data of each of the three speakers U21 to U23 in the web conference. Subsequently, the conference control unit 161 outputs the voices of the plurality of speakers U21 to U23 so that the voices of the plurality of speakers U21 to U23 can be heard from the plurality of sound sources arranged at different positions. In the example shown in FIG. 4, the conference control unit 161 determines the position (orientation) of the sound source of each of the plurality of speakers at the start of the conference, based on the information indicating the roles of the participants in the conference among the conference metadata. ), volume, and presence/absence of audio processing (reverberation processing, etc.).

For example, based on the fact that the role of the participant U21 is "presenter", the conference control unit 161 places the sound source of the voice of the participant U21 at the start of the conference in front of the listener U11 (in the center), where it is easiest for the listener U11 to hear. Decide to put it in position. Further, based on the fact that the role of the remaining participants U22 (participant U23) is "participant", the conference control unit 161 selects the sound source of the voice of the participant U22 (participant U23) at the start of the conference. It is decided to place it at a position to the right of the center (position to the left of the center) where it is harder to hear than the voice of the participant U21.

Further, based on the fact that the role of the participant U21 is "presenter", the conference control unit 161 sets the volume of the voice of the participant U21 at the start of the conference to the loudest volume (for example, " 10”). Further, the conference control unit 161 adjusts the volume of the voice of the participant U22 (participant U23) at the start of the conference based on the fact that the role of the remaining participant U22 (participant U23) is "participant". Decide to set the volume (for example, "7") to be smaller than the volume of the voice of the participant U21.

In addition, it is generally known that if a part of a plurality of sounds is subjected to sound processing (for example, reverberation processing), the brain will be more likely to concentrate on the unprocessed sound. ing. Therefore, the conference control unit 161 determines not to perform reverberation processing on the voice of the participant U21 at the start of the conference, based on the fact that the role of the participant U21 is "presenter". Further, the conference control unit 161 performs reverberation processing on the voice of the participant U22 (participant U23) at the start of the conference based on the fact that the role of the remaining participant U22 (participant U23) is "participant". decide to do

(Calculation unit 162)
In a Web conference in which a plurality of participants participate, the calculation unit 162 calculates, for each of the plurality of speakers, the listening degree indicating the degree of listening to the speech of each of the plurality of speakers by the listener who is listening to the speech of the plurality of speakers. calculate. Specifically, the calculator 162 calculates the degree of attentive listening based on the direction of the line of sight detected by the detector 150 . More specifically, the calculation unit 162 identifies the speaker being watched by the listener based on the line-of-sight direction detected by the detection unit 150, and calculates the degree of listening of the identified speaker to other speakers. It is calculated to be higher than the listening degree of

FIG. 5 is a diagram for explaining the listening degree according to the embodiment. FIG. 5 explains the degree of listening in FIG. In FIG. 5, the calculation unit 162 identifies the speaker U21 being watched by the listener U11 based on the direction of the listener U11's line of sight detected by the detection unit 150, and calculates the listening degree of the identified speaker U21. It is calculated to be higher than the listening degree of the other speakers U22 and U23. For example, the calculation unit 162 calculates the listening degree of the speaker U21, who is being watched by the listener U11, as "100", which is higher than the listening degrees of the other speakers U22 and U23, which are "70". Also, hereinafter, a speaker whose degree of listening is relatively high compared to other speakers is referred to as a "highly listening speaker". In addition, a speaker whose degree of listening is relatively low compared to other speakers is described as a “speaker with low listening intent”. In the examples shown in FIGS. 4 and 5, the speaker U21, whose degree of listening is relatively high compared to the other speakers U22 and U23, is the high listening speaker. In addition, speakers U22 and U23, whose degree of listening is relatively low compared to other speaker U21, are low-listening speakers.

(Generating unit 163)
The generation unit 163 generates proceedings progress information in which character information obtained by converting the statement into text, the statement time of the statement, and the speaker of the statement are recorded in association with each other for each statement in the Web conference. For example, the generating unit 163 uses a known voice recognition technology based on the voice data of the other participants acquired by the conference control unit 161 and the voice data of the participants detected by the microphone of the detecting unit 150 to generate each utterance. Character information is generated by converting the voice data corresponding to the character. Further, the generation unit 163 identifies the speaker of each utterance based on the participant identification information of the other participants acquired by the conference control unit 161 together with the voice data of the other participants. In addition, the generation unit 163 generates the acquisition time (corresponding to the speech time) of the voice data of the other participants acquired by the conference control unit 161 and the acquisition time of the voice data of the participant detected by the microphone of the detection unit 150 (the speech time). ), the utterance time of each utterance is specified. Subsequently, the generation unit 163 generates proceedings progress information in which character information obtained by character conversion of each utterance, utterance time of each utterance, and speaker of each utterance are recorded in association with each other. After generating the proceedings progress information, the generation unit 163 stores the generated proceedings progress information in the storage unit 120 .

(Acquisition unit 164)
The acquisition unit 164 acquires set keywords, such as keywords and phrases set in advance, before the Web conference. For example, the acquisition unit 164 acquires a keyword list from the distribution server 200. FIG. Subsequently, when acquiring the keyword list, the acquisition unit 164 acquires set keywords associated with the participant identification information of the participants based on the acquired keyword list.

(Auxiliary control unit 165)
The auxiliary control unit 165 provides an auxiliary function that makes it easier for listeners to hear the speech of a low-attentive speaker with a low degree of listening than a high-attentive speaker with a high degree of listening. Specifically, when a character string that matches a preset character string is detected from the character information obtained by converting the utterance of the low-attentive speaker, the auxiliary control unit 165 notifies the listener of Emphasis processing is performed to call attention to the remarks of the speaker who listens less attentively.

FIG. 6 is a diagram illustrating an example of an auxiliary function according to the embodiment; FIG. 6 differs from FIG. 4 in that the utterance of speaker U22 includes "keyword #1" (for example, "keyword #1" may be the name of listener U11), which is the set keyword of listener U11. different. The auxiliary control unit 165 refers to the storage unit 120, and selects "keyword #1", which is the set keyword of the listener U11, from the character information obtained by converting the utterance of the speaker U22, who is a speaker with low listening attention in FIG. Find strings that match .

When the auxiliary control unit 165 detects a character string that matches the set keyword of the listener U11, "keyword #1", in the utterance of the speaker U22, the auxiliary control unit 165 performs an example of the emphasizing process. The voice output unit 140 is controlled so that the voice volume of U21 and U23 is louder than the voices of the other speakers U21 and U23. For example, the auxiliary control unit 165 controls the voice output unit 140 so that the volume of the voice of the speaker U22, who is a speaker with low listening comprehension, is set to the loudest volume (for example, "10") that is easy for the listener U11 to hear. . In addition, the auxiliary control unit 165 controls the audio output unit 140 so that the volume of the voices of the other speakers U21 and U23 is set to be lower than the volume of the voice of the speaker U22 (for example, "7"). do.

In addition, as an example of emphasis processing, the auxiliary control unit 165 controls the audio output unit 140 to output the voice of the speaker U22 without performing reverberation processing on the voice of the speaker U22 who is a low-attentive speaker. In addition, as an example of emphasis processing, the auxiliary control unit 165 controls the audio output unit 140 to output the voices of the other speakers U21 and U23 while performing reverberation processing on the voices of the other speakers U21 and U23. do.

In addition, as an example of the emphasis processing, the auxiliary control unit 165 causes the display unit 130 to visually emphasize and display the participant image G22 of the speaker U22, who is a low-attentive speaker, among the plurality of participant images. Control. For example, the auxiliary control unit 165 controls the display unit 130 to highlight or blink the participant image G22 of the speaker U22 who is a low-attentive speaker.

Although not shown in FIG. 6, the auxiliary control unit 165 may control the display unit 130 to display information about the detected character string as an example of the highlighting process. In the example of FIG. 6, the auxiliary control unit 165 controls the display unit 130 to display the detected setting keyword "keyword #1".

FIG. 7 is a diagram illustrating an example of an auxiliary function according to the embodiment; In FIG. 7, the position of the sound source corresponding to the voice of speaker U22, who is a low-attentive speaker in FIG. 4, and the position of the sound source corresponding to the voice of U21, who is a high-attentive speaker in FIG. different from 6.

When the auxiliary control unit 165 detects a character string that matches the set keyword of the listener U11, "keyword #1", in the utterance of the speaker U22, the auxiliary control unit 165 performs an example of the emphasizing process. from the original position (the "right" position in FIGS. 4 and 6) to the position of the sound source corresponding to the voice of speaker U21, who is a speaker with high listening attention (in FIGS. 4 and 6). control the audio output unit 140 to change to the “center” position).

In addition, as an example of the emphasis processing, the auxiliary control unit 165 shifts the display position of the participant image G22 of the speaker U22 who is a speaker with low listening attention among the plurality of participant images to the original position ( The display unit 130 is controlled to change from the "right" position) to the display position of the participant image G21 of the speaker U21, who is a speaker with high listening intent (the "center" position in FIGS. 4 and 6).

Note that in the above-described embodiment, when the auxiliary control unit 165 detects a character string that matches the set keyword in the utterance of the low-attentive speaker, Although a case has been described in which the voice output unit 140 is controlled to change from the position of the speaker to the position of the sound source corresponding to the speaker with high listening attention, the trigger for changing the localization position of the speaker is not limited to the detection of the set keyword. Specifically, the detection unit 150 detects a change in the direction of the listener's line of sight. For example, the detection unit 150 detects that the direction of the listener's line of sight has changed from a speaker with high listening attention to a speaker with low listening attention. The auxiliary control unit 165 identifies the low-attentive speaker who is starting to be watched by the listener, based on the change in the direction of the listener's line of sight detected by the detection unit 150 . Subsequently, the auxiliary control unit 165 determines whether or not the length of time during which the listener watches the specified low-attentive speaker has exceeded a predetermined threshold. If the auxiliary control unit 165 determines that the length of time during which the listener is gazing at the specified low-attentive speaker exceeds a predetermined threshold value, the auxiliary control unit 165 restores the position of the sound source corresponding to the low-attentive speaker's voice to the original position. The voice output unit 140 is controlled to change from the position to the position of the sound source corresponding to the speaker who listens highly.

Instead of the auxiliary control unit 165, the calculation unit 162 may identify a low-attentive speaker who is starting to be watched by the listener, based on the change in the direction of the listener's line of sight detected by the detection unit 150. Subsequently, the calculation unit 162 determines whether or not the length of time during which the listener watches the specified low-attentive speaker exceeds a predetermined threshold. Subsequently, when the calculation unit 162 determines that the length of time during which the listener is paying attention to the specified low-attentive speaker has exceeded a predetermined threshold value, the calculation unit 162 calculates the listening degree of the specified low-attentive speaker to another It is calculated higher than the degree of listening of the speaker. That is, the calculation unit 162 calculates the degree of listening of a speaker whose length of time that the listener is paying attention to exceeds a predetermined threshold value to be higher than the degree of listening of other speakers. In other words, a speaker whose length of time being watched by listeners exceeds a predetermined threshold may be changed from a low-attentive speaker to a high-attentive speaker. Based on the degree of attentive listening calculated by the calculating unit 162, the auxiliary control unit 165 calculates the position of the sound source corresponding to the voice of the speaker newly changed to the speaker with high attentive listening (original speaker with low attentive listening). , to the position of the sound source corresponding to the original high-attentive speaker.

[3. Information processing procedure]
FIG. 8 is a diagram illustrating an information processing procedure according to the embodiment; As shown in FIG. 8, the detection unit 150 of the information processing apparatus 100 detects the line of sight of a listener who is listening to the statements of a plurality of speakers in a remote conference in which a plurality of participants participate (step S101).

The calculation unit 162 of the information processing device 100 calculates, for each of the plurality of speakers, the degree of listening that indicates the degree of listening to the speech of each of the plurality of speakers (step S102). For example, the calculation unit 162 identifies the speaker being watched by the listener based on the direction of the line of sight detected by the detection unit 150, and sets the listening degree of the identified speaker higher than the listening degree of the other speakers. Calculate high.

The auxiliary control unit 165 of the information processing apparatus 100 provides an auxiliary function that makes it easier for listeners to hear the speech of the low-attentive speaker whose degree of listening is lower than that of the high-attentive speaker whose high listening degree is (step S103). For example, if a character string matching a preset character string is detected from the character information obtained by converting the utterance of the speaker with low attentive listening, the auxiliary control unit 165 Emphasis processing is performed to call attention to the person's remarks.

[4. Modification]
The information processing system 1 according to the above-described embodiments may be implemented in various different forms other than the above-described embodiments. Therefore, other embodiments of the information processing system 1 will be described below. In addition, the same code|symbol is attached|subjected to the same part as embodiment, and description is abbreviate|omitted.

[4-1. Time shift playback]
FIG. 9 is a diagram illustrating an example of an auxiliary function according to a modification; In FIG. 9, it is assumed that the listener U11 has missed the speech of the speaker U22, who is a speaker with low listening comprehension. Therefore, the listener U11 performs an operation to rewind and reproduce at high speed the individually recorded image G22' in which the participant image of the speaker U22 who has missed to hear is recorded. For example, it is assumed that the listener U11 performs an operation (an operation such as clicking or tapping) to select the image G22 of the speaker U22 who is a low-attentive speaker on the screen shown in FIG. Note that the operation by the listener U11 to designate the speaker U22 is not limited to the operation of selecting the image G22 of the speaker U22. For example, the listener U11's operation to specify the speaker U22 may be performed by the listener U11 inputting information that enables the speaker U22 to be identified in a dedicated field.

During the Web conference, the auxiliary control unit 165 selects the speaker U22, who is a low-attentive speaker designated by the listener U11, among the plurality of individually recorded images obtained by recording the participant images of the respective participants. The display unit 130 is controlled to reproduce and display the corresponding individual recorded image G22'. Specifically, when the auxiliary control unit 165 receives from the listener U11 the specification of the speaker whose individual recorded image is to be reproduced, the auxiliary control unit 165 transmits the individual recorded image G22' corresponding to the speaker U22 specified by the listener U11 from the distribution server 200. get. Subsequently, the auxiliary control unit 165 controls the display unit 130 to reproduce and display the individual recorded image G22' in the image display area F22 of the speaker U22, who is the low-attentive speaker designated by the listener U11. In addition, the distribution server 200 stores a plurality of individually recorded images obtained by recording the participant images of each of the plurality of participants. Then, the distribution server 200 may stream-distribute the individual recorded image specified by the listener to the information processing apparatus 100 . For example, when the auxiliary control unit 165 receives from the listener U11 the specification of a speaker whose individual recorded image is to be reproduced, the auxiliary control unit 165 controls the display unit 130 to display the individual recorded image G22′ streamed from the distribution server 200.

Further, the auxiliary control unit 165 controls the display unit 130 to reproduce and display the individual recorded image G22' at the reproduction speed "1.5x" specified by the listener U11. The individually recorded image G22' shown in FIG. 9 includes an icon G222 indicating the playback speed "1.5x speed" specified by the listener U11. The individually recorded image G22' also includes an icon G223 indicating a selectable playback speed of "1.0x speed" and an icon G224 indicating a selectable playback speed of "2.0x speed", which are not specified by the listener U11. .

Further, the auxiliary control unit 165 controls the display unit 130 to reproduce and display the individual recorded image G22' at the reproduction time "30 seconds" specified by the listener U11. The individual recorded image G22' shown in FIG. 9 includes an icon G225 indicating the reproduction time "30 seconds" specified by the listener U11. The individual recorded image G22' also includes an icon G226 indicating a selectable reproduction time of "10 seconds" although not specified by the listener U11. Note that the reproduction time is lengthened in proportion to the number of times the listener U11 selects the icon G225 or the icon G226.

The individual recorded image G22' also includes a button G227 that can switch from the recorded image to the original Web conference image (live image) when selected by the listener U11.

In the above example, the auxiliary control unit 165 controls the display unit 130 to reproduce and display the individual recorded image during the Web conference. , the display unit 130 may be controlled to reproduce and display the entire recorded image obtained by recording the full screen image. Specifically, the display unit 130 displays the proceedings progress information during the web conference according to the operation of the participants. When the auxiliary control unit 165 accepts the speech time of the speech specified by the listener as the reproduction start time from the proceedings progress information, the auxiliary control unit 165 transmits the entire recorded image obtained by recording the full screen image from the received reproduction start time to the distribution server. 200. Subsequently, the auxiliary control unit 165 controls the display unit 130 to reproduce and display the acquired whole recorded image. As a result, the information processing apparatus 100 can help a listener who is 20 minutes late to participate in the previous meeting to catch up by reproducing only the important part at double speed.

[4-2. Listening Cluster Representation]
FIG. 10 is a diagram illustrating an example of an auxiliary function according to a modification; Generally, in a web conference, unlike a face-to-face meeting, it is difficult to feel the line of sight of the other party. Therefore, the generation unit 163 identifies a super-listening speaker whose degree of listening exceeds a predetermined threshold for each of a plurality of listeners, classifies each listener into a cluster of super-listening speakers specified for each listener, and classifies Generate cluster information for each cluster.

In the example shown in FIG. 9 , the generating unit 163 determines that four listeners “ayyapan”, “doigaki”, “ishige”, and “yamada” are super-listening speakers whose listening degrees exceed a predetermined threshold. The speaker U21 is specified, the four listeners are classified into the cluster CL21 of the speaker U21, and cluster information about the classified cluster CL21 is generated. For example, the generation unit 163 generates an image G31 that includes icons that allow recognition of emotions such as facial expressions and backtracking of each of the four listeners, in addition to the degree of listening indicated by the cluster CL21.

In addition, the generation unit 163 determines that the degree of listening for each of the three listeners "iwaki", "tonoma", and "yamaoka" who are participating in the same Web conference as the participant of the cluster CL21 exceeds the predetermined threshold. A speaker U22 who is a speaker who listens attentively is specified, three listeners are classified into a cluster CL22 of the speaker U22, and cluster information about the classified cluster CL22 is generated. For example, the generation unit 163 generates an image G32 that includes, in addition to the degree of attentive listening indicated by the cluster CL22, icons capable of recognizing emotions such as facial expressions and backtracking of each of the three listeners.

Auxiliary control unit 165 controls display unit 130 to display image G31 and image G32 generated by generation unit 163 during the Web conference.

As a result, the information processing apparatus 100 can visually recognize the line of sight of each of the plurality of participants in the remote conference. In addition, the information processing apparatus 100 enables each of a plurality of participants to participate in the same Web conference, divide into a plurality of groups for discussion, and freely move between groups.

[4-3. Voice quality health check]
FIG. 11 is a diagram illustrating an example of an auxiliary function according to a modification; In general, in a web conference, unlike a face-to-face conference, one does not know whether the other party has received the voice normally. Therefore, in FIG. 11, first voice feature data indicating the features of the first voice data output from the information processing device 100-1 of the participant U1 who is the person (voice output source), and other conference participants ( If the second voice feature data indicating the feature of the second voice data output from the information processing device 100-2 of the participant U2 (the voice output destination) does not match, the participant U1 is notified.

Specifically, the conference control unit 161 of the information processing device 100-1 acquires the person's voice data (first voice data) in the web conference. After acquiring the first audio data, the conference control unit 161 transmits the first audio data to the distribution server 200 . When distribution server 200 acquires the first audio data from information processing device 100-1, distribution server 200 transmits the acquired first audio data to information processing device 100-2. Further, when distribution server 200 acquires the first audio data from information processing device 100-1, distribution server 200 generates audio feature data indicating the characteristics of the acquired first audio data. Here, the sound feature data is, for example, data obtained by characterizing (digitizing) the amount of change in sound amplitude over time or the number of continuous average amplitude passages.

Further, generation unit 163 of information processing device 100-1 generates first sound feature data indicating the characteristics of the first sound data based on the first sound data acquired by conference control unit 161. FIG. After generating the first sound feature data, generation unit 163 transmits the first sound feature data to distribution server 200 . When the distribution server 200 acquires the first sound feature data from the information processing device 100-1, the distribution server 200 compares the acquired first sound feature data with the sound feature data representing the feature of the acquired first sound data, and determines whether the two are the same. If they match, the first sound feature data is transmitted to information processing device 100-2.

Conference control unit 161 of information processing apparatus 100-2 outputs the first audio data from audio output unit 140 upon obtaining the first audio data from distribution server 200. FIG. Acquisition unit 164 of information processing device 100-2 acquires the second audio data output from audio output unit 140. FIG. Generation unit 163 of information processing device 100-2 generates second sound feature data indicating the characteristics of the second sound data based on the second sound data acquired by acquisition unit 164. FIG. Auxiliary control unit 165 of information processing device 100 - 2 also acquires the first sound feature data from distribution server 200 . The auxiliary control unit 165 notifies the participant U1 when the second sound feature data generated by the generation unit 163 and the first sound feature data do not match.

The above-described information processing system 1 has a mechanism for reliable reception (retransmission until TCP and ACK confirmation) because comparison cannot be performed if loss occurs along the network path in the same way as voice.

As a result, the information processing apparatus 100 can easily estimate where the voice packet is dropped from the match/unmatch results of the voice feature data at a plurality of points. Note that the unmatched result, including the presentation of the suspected section, may be shared not only with the person himself but also with all the participants.

[4-4. Number and position of sound sources]
Further, in the above-described embodiment, the information processing apparatus 100 arranges the voices of the three speakers so that the voices of the three speakers can be heard from the sound sources arranged at different positions on the left, right, and center of the listener. , but the number of sound sources is not limited to three. Specifically, the information processing apparatus 100 arranges the voices of two or less speakers so that the voices of the two or fewer speakers can be heard from each of two or fewer sound sources arranged at different positions according to the number of speakers. You may output the voice of each person. In addition, the information processing apparatus 100 is configured so that each of the four or more speakers can be heard from each of four or more sound sources arranged at different positions according to the number of speakers. Audio may be output.

More specifically, the information processing apparatus 100 controls the sound of each of the four or more speakers so that each of the four or more speakers can be heard from each of four or more sound sources arranged on a straight line in front of the listener. may be output. For example, the auxiliary control unit 165 arranges the position of the sound source corresponding to the speaker whose listening degree calculated by the calculation unit 162 is high, closer to the listener. In other words, the auxiliary control unit 165 arranges the position of the sound source corresponding to the speaker whose degree of attentive listening calculated by the calculation unit 162 is low, at a position farther from the listener. Note that the positions of the sound sources may be arranged at equal intervals on a straight line in front of the listener, for example.

In addition, the information processing apparatus 100 is arranged so that the voices of the four or more speakers can be heard from four or more sound sources arranged on the circumference of a circle with the listener at the center. may be output. The positions of the sound sources may, for example, be evenly spaced on the circumference of a circle centered at the listener. For example, the auxiliary control unit 165 arranges the position of the sound source corresponding to the speaker with the highest listening degree calculated by the calculation unit 162 in front of the listener. Further, the auxiliary control unit 165 arranges the positions of the sound sources corresponding to the speaker with the next highest listening degree calculated by the calculation unit 162 to the right and left positions in front of the listener. Further, the auxiliary control unit 165 arranges the position of the sound source corresponding to the speaker whose degree of attentive listening calculated by the calculation unit 162 is relatively low behind the listener. Also, it is generally known that in a surround system, it is difficult for the listener to hear sounds behind the listener (almost inaudible). Therefore, the auxiliary control unit 165 may intentionally set the volume of the voice of the speaker whose degree of attentive listening calculated by the calculation unit 162 to be relatively low to zero.

As described above, the information processing apparatus 100 outputs the voices of the plurality of speakers so that the voices of the plurality of speakers can be heard from the plurality of sound sources arranged at different positions.

[4-5. number of speakers]
In the above-described embodiment, the information processing apparatus 100 outputs the voices of a plurality of speakers from two speakers arranged on the left and right sides of the listener, but the number of speakers is limited to two. can't For example, the information processing device 100 includes three or more (for example, six or eight) speakers arranged to surround the listener, and the surround system allows the voices of a plurality of speakers to be played at different positions. The voices of each of the plurality of speakers may be output from each of three or more speakers so that they can be heard from each of the plurality of arranged sound sources. Information processing apparatus 100 may output the voices of a plurality of speakers from a single speaker so that the voices of the speakers can be heard from a plurality of sound sources arranged at different positions.

[5. effect〕
As described above, the information processing apparatus 100 according to the embodiment includes the calculator 162 and the auxiliary controller 165 . In a remote conference in which a plurality of participants participate, the calculation unit 162 calculates, for each of the plurality of speakers, the listening degree indicating the degree of listening to the speech of each of the plurality of speakers by a listener who is listening to the speeches of the plurality of speakers. calculate. The auxiliary control unit 165 provides an auxiliary function that makes it easier for listeners to hear the speech of a low-attentive speaker with a low degree of listening than a high-attentive speaker with a high degree of listening.

In this way, the information processing apparatus 100 can make it easier for the listener to hear the speaker who is not the listener's attention (the listener is not paying attention). As a result, the information processing apparatus 100 can enable participants to have substantial discussions in a remote conference in which a plurality of participants participate. In addition, the information processing apparatus 100 can enable participants to spend time comfortably during a remote conference in which a plurality of participants participate. Therefore, the information processing apparatus 100 can improve usability in the remote conference.

Further, the calculation unit 162 calculates the listening degree based on the line-of-sight direction detected by the detection unit 150 that detects the line-of-sight of the listener.

As a result, the information processing apparatus 100 calculates the degree of attentive listening based on the line of sight of the listener, so that the degree of attentive listening can be calculated appropriately.

Further, the calculation unit 162 identifies the speaker being watched by the listener based on the line-of-sight direction detected by the detection unit 150, and compares the listening degree of the identified speaker to the listening degree of the other speakers. calculated to be higher.

As a result, the information processing apparatus 100 calculates the degree of listening to the speaker that the listener is paying attention to higher than the degree of listening to other speakers, so that the degree of listening can be calculated appropriately.

Further, when a character string matching a preset character string is detected from the character information obtained by converting the utterance of the speaker with low attentive listening, the auxiliary control unit 165 instructs the listener to make the utterance with low attentive listening. Emphasis processing is performed to call attention to the person's remarks. For example, the preset character string is a character string set based on the results of machine learning using character strings preset in past remote meetings as teacher data.

As a result, when a preset keyword or the like appears in the utterance of a speaker who is not the listener's attention (the listener is not paying attention), the information processing apparatus 100 is not the listener's attention ( The listener can be alerted to pay attention to the speaker (whom the listener is not paying attention to). Therefore, the information processing apparatus 100 allows the listener to hear the speech of a speaker who is not the listener's attention (the listener is not paying attention to) only when necessary, without preventing the listener from concentrating on the speech of the listener's attention. can be made easier to hear.

In addition, as an emphasis process, the auxiliary control unit 165 outputs the voices of the plurality of speakers output from the voice output unit 140 so that the voices of the plurality of speakers can be heard from each of the plurality of sound sources arranged at different positions. Among them, the voice output unit 140 is controlled so that the volume of the voice of the low-attentive speaker is louder than the volume of the voices of the other speakers.

As a result, the information processing apparatus 100 increases the volume of the voice of the speaker who is not the listener's attention (the listener's attention is not focused) to a volume that is easy for the listener to hear. not) can make it easier for listeners to hear what the speaker is saying.

In addition, as an emphasis process, the auxiliary control unit 165 outputs the voices of the plurality of speakers output from the voice output unit 140 so that the voices of the plurality of speakers can be heard from each of the plurality of sound sources arranged at different positions. Among them, the voice output unit 140 is controlled to change the position of the sound source corresponding to the voice of the low-attentive speaker from the original position to the position of the sound source corresponding to the voice of the high-attentive speaker.

As a result, the information processing apparatus 100 changes the position of the sound source corresponding to the voice of the speaker who is not the listener's attention (the listener is not paying attention) to a position that is easy for the listener to hear. It is possible to make it easier for the listener to hear the speaker who is not (the listener does not pay attention to).

In addition, as an emphasis process, the auxiliary control unit 165 outputs the voices of the plurality of speakers output from the voice output unit 140 so that the voices of the plurality of speakers can be heard from each of the plurality of sound sources arranged at different positions. Of these, the voice output unit 140 is controlled so as to output the voice of the low-attentive speaker without performing reverberation processing on the voice of the low-attentive speaker.

In general, it is known that if some voices out of a plurality of voices are subjected to voice processing (for example, reverberation processing), the brain will be more likely to concentrate on the voices that have not undergone voice processing. . As a result, the information processing apparatus 100 can make it easier for the listener to hear the speaker who is not the listener's attention (the listener is not paying attention).

In addition, the auxiliary control unit 165 performs emphasis processing, among the participant images of each of the plurality of participants displayed in the different display areas of the display unit 130, to display the participant image of the low-attentive speaker as the participant image of the other speaker. The display unit 130 is controlled so as to visually emphasize and display the participant image.

As a result, the information processing apparatus 100 visually emphasizes and displays the participant image of the speaker who is not the listener's attention (the listener is not paying attention). is not paying attention to).

Further, the auxiliary control unit 165 controls the display unit 130 to display information about the detected character string as the highlighting process.

As a result, the information processing apparatus 100 displays the set keyword to appeal to the listener's visual sense, so it is possible to call attention to the statement of the speaker who is not the listener's attention (the listener is not paying attention to). .

In addition, the auxiliary control unit 165 selects, by the listener, among a plurality of individually recorded images obtained by recording the participant images of the respective participants displayed in different display areas of the display unit 130 during the remote conference. The display unit 130 is controlled so as to reproduce and display the individual recorded image corresponding to the designated low-attentive speaker.

As a result, even if the listener fails to hear the speech of a speaker who is not the listener's attention (the listener is not paying attention), the information processing apparatus 100 can reproduce the missed speech during the conference. It can be made possible to catch up.

Further, the auxiliary control unit 165 controls the display unit 130 to reproduce and display the individual recorded image at the reproduction time specified by the listener.

As a result, the information processing apparatus 100 allows the listener to specify the playback time of the individual recorded image, thereby improving usability for the listener.

Further, the auxiliary control section 165 controls the display section 130 to reproduce and display the individual recorded image at the reproduction speed specified by the listener.

As a result, the information processing apparatus 100 allows the listener to specify the playback speed of the individual recorded image, thereby improving usability for the listener.

Further, the auxiliary control unit 165 controls the display unit 130 to reproduce and display the individual recorded image in the display area corresponding to the low-attentive speaker designated by the listener.

As a result, the information processing apparatus 100 enables the listener to listen to the speech of the speaker who has missed it while listening to the speech of the other speaker.

Also, during the remote conference, the auxiliary control unit 165 reproduces the entire recorded image obtained by recording the full-screen images including the participant images of each of the plurality of participants displayed in different display areas of the display unit 130. The display unit 130 is controlled to display the

Thereby, the information processing apparatus 100 can enable the listener to catch up during the conference, for example, the content of the entire conference that the listener joined late.

Information processing apparatus 100 further includes generation unit 163 . The generation unit 163 generates proceedings progress information in which character information obtained by converting the statement into characters, the statement time of the statement, and the speaker of the statement are associated and recorded for each statement in the remote conference. During the remote conference, the auxiliary control unit 165 sets the utterance time of the utterance specified by the listener from the progress information displayed on the display unit 130 as the playback start time, and reproduces the entire recorded image from the playback start time. Control the display unit 130 to reproduce and display.

As a result, the information processing apparatus 100 enables the listener to catch up during the conference, for example, the content of the entire conference in which the listener participates late, and the content after the statement that the listener considers important. can.

Information processing apparatus 100 further includes generation unit 163 . The generation unit 163 identifies, for each of a plurality of listeners, a speaker whose listening degree exceeds a predetermined threshold, classifies each listener into a cluster of speakers who are super-listening and classifies each cluster. Generate cluster information about . The auxiliary control unit 165 controls the display unit 130 to display the cluster information generated by the generation unit 163 during the remote conference.

As a result, the information processing apparatus 100 can visually recognize the line of sight of each of the plurality of participants in the remote conference. In addition, the information processing apparatus 100 enables each of a plurality of participants to participate in the same Web conference, divide into a plurality of groups for discussion, and freely move between groups.

Information processing apparatus 100 further includes acquisition unit 164 and generation unit 163 . The acquisition unit 164 acquires voice data of participants in the remote conference. The generation unit 163 generates first sound feature data indicating the feature of the sound data based on the sound data. The auxiliary control unit 165 notifies the participant when the second audio feature data indicating the characteristics of the audio data received by the other information processing device 100 of the other participant does not match the first audio feature data. conduct.

As a result, the information processing apparatus 100 can easily estimate where the voice packet is dropped from the match/unmatch results of the voice feature data at a plurality of points.

[6. Hardware configuration]
Further, the information processing apparatus 100 according to the embodiment described above is implemented by a computer 1000 configured as shown in FIG. 12, for example. FIG. 12 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing apparatus 100. As shown in FIG. Computer 1000 includes CPU 1100 , RAM 1200 , ROM 1300 , HDD 1400 , communication interface (I/F) 1500 , input/output interface (I/F) 1600 and media interface (I/F) 1700 .

The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each section. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. Communication interface 1500 receives data from another device via a predetermined communication network, sends the data to CPU 1100, and transmits data generated by CPU 1100 to another device via a predetermined communication network.

The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, through an input/output interface 1600 . CPU 1100 acquires data from an input device via input/output interface 1600 . CPU 1100 also outputs the generated data to an output device via input/output interface 1600 .

Media interface 1700 reads programs or data stored in recording medium 1800 and provides them to CPU 1100 via RAM 1200 . CPU 1100 loads such a program from recording medium 1800 onto RAM 1200 via media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. etc.

For example, when the computer 1000 functions as the information processing apparatus 100 according to the embodiment, the CPU 1100 of the computer 1000 implements the functions of the control unit 160 by executing programs loaded on the RAM 1200 . CPU 1100 of computer 1000 reads these programs from recording medium 1800 and executes them, but as another example, these programs may be obtained from another device via a predetermined communication network.

As described above, some of the embodiments of the present application have been described in detail based on the drawings. It is possible to carry out the invention in other forms with modifications.

[7. others〕
Further, among the processes described in the above embodiments and modifications, all or part of the processes described as being performed automatically can be performed manually, or described as being performed manually. All or part of the processing can also be performed automatically by known methods. In addition, information including processing procedures, specific names, various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the one shown in the figure, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

For example, in the above-described embodiment, an example in which the information processing system 1 is a central processing type (also referred to as a centralized type) computing system has been described, but the configuration of the information processing system 1 is not limited to this. For example, the information processing system 1 may be a distributed computing system composed of a plurality of information processing apparatuses 100 . In this case, the information processing system includes a plurality of information processing apparatuses 100 connected to each other via a network. Further, in this case, in the information processing system, the functions of the information processing device 100 described with reference to FIG. 2 are implemented in each information processing device 100 .

The information processing apparatus 100 described above may be implemented by a plurality of server computers, and depending on the function, may be implemented by calling an external platform or the like using an API (Application Programming Interface), network computing, or the like. can be changed flexibly.

Also, the above-described embodiments and modifications can be appropriately combined within a range that does not contradict the processing contents.

Also, the above-mentioned "section, module, unit" can be read as "means" or "circuit". For example, the auxiliary control section can be read as auxiliary control means or an auxiliary control circuit.

1 information processing system 100 information processing device 110 communication unit 120 storage unit 130 display unit 140 audio output unit 150 detection unit 160 control unit 161 conference control unit 162 calculation unit 163 generation unit 164 acquisition unit 165 auxiliary control unit

An information processing program according to an embodiment, in a remote conference in which a plurality of participants participate, calculates a listening degree indicating a degree of listening to the speech of each of the plurality of speakers by a listener who is listening to the speech of the plurality of speakers. A calculation procedure for calculating each of a plurality of speakers, and assistance for providing an auxiliary function that makes it easier for the listener to hear the speech of the low-attentive speaker whose degree of listening is low compared to the high-attentive speaker whose high degree of listening is. causing a computer to execute a control procedure;
In addition, the calculating step calculates the degree of attentive listening based on the direction of the line of sight detected by the detection unit that detects the line of sight of the listener.
Further, the calculation step identifies a speaker being watched by the listener based on the direction of the line of sight detected by the detection unit, and calculates the degree of listening of the identified speaker to the It is calculated higher than the listening degree.
Further, the auxiliary control procedure instructs the listener, when a character string matching a preset character string is detected from the character information obtained by converting the utterance of the low-attentive speaker, to the listener. Emphasis processing is performed to call attention to the remarks of the speaker who listens less attentively.
Further, the character string set in advance is a character string set based on the result of machine learning using character strings set in advance in past remote meetings as teacher data.
Further, the auxiliary control procedure includes, as the emphasizing process, outputting the voice of each of the plurality of speakers from a sound output unit so that the voice of each of the plurality of speakers can be heard from each of a plurality of sound sources arranged at different positions. out of the voices, the voice output unit is controlled so that the volume of the voice of the low-attentive speaker is louder than the voices of the other speakers.
Further, the auxiliary control procedure includes, as the emphasizing process, outputting the voice of each of the plurality of speakers from a sound output unit so that the voice of each of the plurality of speakers can be heard from each of a plurality of sound sources arranged at different positions. and controlling the voice output unit to change the position of the sound source corresponding to the voice of the low-attentive speaker from the original position to the position of the sound source corresponding to the voice of the high-attentive speaker.
Further, the auxiliary control procedure includes, as the emphasizing process, outputting the voice of each of the plurality of speakers from a sound output unit so that the voice of each of the plurality of speakers can be heard from each of a plurality of sound sources arranged at different positions. The voice output unit is controlled so as to output the voice of the low-attentive speaker without performing reverberation processing on the voice of the low-attentive speaker.
Further, the auxiliary control procedure includes, as the emphasizing process, the participant image of the low-attentive speaker among the participant images of the plurality of participants displayed in different display areas of the display unit. The display unit is controlled so as to visually emphasize and display the participant image of the speaker.
Further, the auxiliary control procedure controls the display unit to display information about the detected character string as the highlighting process.
In addition, the auxiliary control procedure includes, during the remote conference, among the plurality of individually recorded images obtained by recording the participant images of the plurality of participants respectively displayed in different display areas of the display unit, the The display unit is controlled so as to reproduce and display the individual recorded image corresponding to the low-attentive speaker designated by the listener.
Also, the auxiliary control procedure controls the display unit to reproduce and display the individually recorded image at the reproduction time designated by the listener.
Also, the auxiliary control procedure controls the display unit to reproduce and display the individually recorded image at a reproduction speed designated by the listener.
Further, the auxiliary control procedure controls the display unit to reproduce and display the individual recorded image in the display area corresponding to the low-attentive speaker designated by the listener.
Further, the auxiliary control procedure reproduces an entire recorded image obtained by recording a full-screen image including participant images of each of the plurality of participants displayed in different display areas of a display unit during the remote conference. The display unit is controlled to display as
Further, the method further includes a generation procedure for generating proceedings progress information in which character information obtained by converting the utterance into characters, the utterance time of the utterance, and the speaker of the utterance are recorded in association with each other for each utterance in the remote conference. , the auxiliary control procedure includes, during the remote conference, using the utterance time of the utterance specified by the listener as a reproduction start time from among the proceedings progress information displayed on the display unit, and performing the reproduction. The display unit is controlled to reproduce and display the entire recorded image from the start time.
Further, a super-listening speaker whose degree of listening exceeds a predetermined threshold is specified for each of the plurality of listeners, each listener is classified into clusters of the super-listening speaker specified for each listener, and each classified cluster. and a generating procedure for generating cluster information relating to, wherein the auxiliary control procedure controls a display to display the cluster information generated by the generating procedure during the remote conference.
The method further comprises: an acquisition procedure for acquiring voice data of a participant in the remote conference; and a generation procedure for generating first voice feature data indicating features of the voice data based on the voice data. The control procedure notifies the participant when the second audio feature data indicating the characteristics of the audio data received by the other information processing device of the other participant does not match the first audio feature data. conduct.
Further, an information processing method according to an embodiment is an information processing method executed by a computer, in which, in a remote conference in which a plurality of participants participate, a listener who is listening to the speeches of the plurality of speakers a calculating step of calculating, for each of the plurality of speakers, a degree of attentive listening that indicates the degree of listening to each utterance; and an auxiliary control step for providing auxiliary functions to facilitate listening to the listener.
Further, the information processing apparatus according to the embodiment provides a degree of listening that indicates the degree of listening to each of the plurality of speakers by a listener who is listening to the statements of the plurality of speakers in a remote conference in which a plurality of participants participate. for each of the plurality of speakers, and an auxiliary function that makes it easier for the listener to hear the utterances of the low-attentive speaker with a low degree of listening compared to the high-attentive speaker with a high degree of listening. and an auxiliary control unit for

Claims (20)

In a remote conference in which a plurality of participants participate, a calculation for calculating, for each of the plurality of speakers, a listening degree indicating the degree of listening to the speech of each of the plurality of speakers by a listener who is listening to the speeches of the plurality of speakers. a procedure;
an auxiliary control procedure for providing an auxiliary function that makes it easier for the listener to hear the speech of the low-attentive speaker whose degree of listening is lower than that of the high-attentive speaker whose degree of listening is high;
An information processing program that causes a computer to execute The calculation procedure is
calculating the degree of attentive listening based on the direction of the line of sight detected by the detection unit that detects the line of sight of the listener;
The information processing program according to claim 1. The calculation procedure is
Based on the line-of-sight direction detected by the detection unit, the speaker being watched by the listener is identified, and the listening degree of the identified speaker is calculated higher than the listening degree of other speakers. ,
The information processing program according to claim 2. The auxiliary control procedure is
When a character string matching a preset character string is detected from the character information obtained by converting the utterance of the speaker with low attentiveness to listening, the listener is instructed to respond to the utterance of the speaker with low attentiveness. Perform emphasis processing to call attention,
The information processing program according to any one of claims 1 to 3. The preset string is
A character string set based on the results of machine learning using character strings set in advance in past remote meetings as teacher data.
The information processing program according to claim 4. The auxiliary control procedure is
As the emphasizing process, among the voices of the plurality of speakers output from the voice output unit so that the voices of the plurality of speakers can be heard from each of the plurality of sound sources arranged at different positions, the low-level listening is performed. controlling the audio output unit to output the volume of the speaker's voice louder than the volume of the other speaker's voice;
6. The information processing program according to claim 4 or 5. The auxiliary control procedure is
As the emphasizing process, among the voices of the plurality of speakers output from the voice output unit so that the voices of the plurality of speakers can be heard from each of the plurality of sound sources arranged at different positions, the low-level listening is performed. controlling the audio output unit to change the position of the sound source corresponding to the speaker's voice from the original position to the position of the sound source corresponding to the high-attentive speaker's voice;
The information processing program according to any one of claims 4 to 6. The auxiliary control procedure is
As the emphasizing process, among the voices of the plurality of speakers output from the voice output unit so that the voices of the plurality of speakers can be heard from each of the plurality of sound sources arranged at different positions, the low-level listening is performed. controlling the audio output unit to output the voice of the low-attentive speaker without performing reverberation processing on the speaker's voice;
The information processing program according to any one of claims 4 to 7. The auxiliary control procedure is
As the emphasizing process, among the participant images of the plurality of participants displayed in different display areas of the display unit, the participant image of the low-attentive speaker is selected from the participant images of the other speakers. Control the display unit to visually emphasize and display
The information processing program according to any one of claims 4 to 8. The auxiliary control procedure is
controlling the display unit to display information about the detected character string as the highlighting process;
The information processing program according to claim 9 . The auxiliary control procedure is
The low listening level designated by the listener among a plurality of individually recorded images obtained by recording participant images of the plurality of participants respectively displayed in different display areas of a display unit during the remote conference. controlling the display unit to reproduce and display an individual recorded image corresponding to the speaker;
The information processing program according to any one of claims 1 to 10. The auxiliary control procedure is
controlling the display unit to reproduce and display the individually recorded image at the reproduction time specified by the listener;
The information processing program according to claim 11. The auxiliary control procedure is
controlling the display unit to reproduce and display the individually recorded image at the reproduction speed specified by the listener;
The information processing program according to claim 11 or 12. The auxiliary control procedure is
controlling the display unit to reproduce and display the individually recorded image in the display area corresponding to the low-attentive speaker designated by the listener;
The information processing program according to any one of claims 11 to 13. The auxiliary control procedure is
During the remote conference, the display unit reproduces and displays an entire recorded image obtained by recording full-screen images including participant images of the plurality of participants displayed in different display areas of the display unit. to control the
The information processing program according to any one of claims 1 to 14. further comprising a generating procedure for generating proceedings progress information in which character information obtained by converting the utterance into characters, the utterance time of the utterance, and the speaker of the utterance are associated and recorded for each utterance in the remote conference;
The auxiliary control procedure is
During the remote conference, from the progress information displayed on the display unit, the speech time of the speech specified by the listener is set as a playback start time, and the entire recorded image is started from the playback start time. controlling the display to play and display
The information processing program according to claim 15. A super-listening speaker whose degree of listening exceeds a predetermined threshold is specified for each of the plurality of listeners, each listener is classified into a cluster of the super-listening speaker specified for each listener, and clusters for each cluster are classified. a generation procedure for generating information;
further comprising
The auxiliary control procedure is
during the remote conference, controlling a display unit to display the cluster information generated by the generating procedure;
The information processing program according to any one of claims 1 to 16. an acquisition procedure for acquiring voice data of participants in the remote conference;
a generation procedure for generating first audio feature data indicating a feature of the audio data based on the audio data;
further comprising
The auxiliary control procedure is
Notifying the participant when the second audio feature data indicating the characteristics of the audio data received by another information processing program of another participant does not match the first audio feature data,
The information processing program according to any one of claims 1 to 17. A computer-executed information processing method comprising:
In a remote conference in which a plurality of participants participate, a calculation for calculating, for each of the plurality of speakers, a listening degree indicating the degree of listening to the speech of each of the plurality of speakers by a listener who is listening to the speeches of the plurality of speakers. process and
an auxiliary control step of providing an auxiliary function that makes it easier for the listener to hear the speech of the low-attentive speaker whose degree of listening is lower than that of the high-attentive speaker whose degree of listening is high;
Information processing method including. In a remote conference in which a plurality of participants participate, a calculation for calculating, for each of the plurality of speakers, a listening degree indicating the degree of listening to the speech of each of the plurality of speakers by a listener who is listening to the speeches of the plurality of speakers. Department and
an auxiliary control unit that provides an auxiliary function that makes it easier for the listener to hear the speech of the low-attentive speaker whose degree of listening is lower than that of the high-attentive speaker whose degree of listening is high;
Information processing device.

Images