WO2022137547A1 - Système d'aide à la communication - Google Patents

Système d'aide à la communication Download PDF

Info

Publication number
WO2022137547A1
WO2022137547A1 PCT/JP2020/048863 JP2020048863W WO2022137547A1 WO 2022137547 A1 WO2022137547 A1 WO 2022137547A1 JP 2020048863 W JP2020048863 W JP 2020048863W WO 2022137547 A1 WO2022137547 A1 WO 2022137547A1
Authority
WO
WIPO (PCT)
Prior art keywords
participant
user
output
intimacy
support system
Prior art date
Application number
PCT/JP2020/048863
Other languages
English (en)
Japanese (ja)
Inventor
契 宇都木
Original Assignee
株式会社日立製作所
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社日立製作所 filed Critical 株式会社日立製作所
Priority to PCT/JP2020/048863 priority Critical patent/WO2022137547A1/fr
Publication of WO2022137547A1 publication Critical patent/WO2022137547A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer

Definitions

  • the present invention relates to a communication support system.
  • remote conferences and remote collaborative creation activities have the problem that communication does not proceed smoothly because it is difficult to understand the real reaction of the other party compared to activities in the actual environment. Therefore, in remote communication involving a large number of people, it is important to provide appropriate feedback on the reactions of participants according to the situation and to support remote communication.
  • Patent Document 1 is known as a background technique of the present invention.
  • speech information is extracted based on voice data collected from participants in a plurality of meetings, activity data in the meeting of the participants is generated, and the dialogue status of the participants is described by the size of a circle or a line.
  • visualizing and displaying using the thickness etc. and by acquiring the voices of multiple participants during the meeting and displaying the ever-changing conversation situation in real time, more active discussions can be held while observing the situation. Techniques that can be induced are disclosed.
  • Patent Document 1 the problems in terms of voice in remote communication involving a large number of people such as remote conferences and remote collaborative creation activities can be solved, but nonverbal communication by visual information such as facial expressions is lacking. The problem of the point to be done has not been solved. Therefore, in remote communication, it is necessary to more appropriately grasp the reaction of each participant.
  • the communication support system of the present invention is a communication support system that supports remote communication performed between a plurality of participants, and includes an input reception unit that receives input from the participants participating in the communication support system, and the above-mentioned.
  • An intimacy estimation unit that estimates the intimacy between the first participant and the other participants based on the information about the first participant and the other participants among the plurality of participants.
  • Output priority calculation that determines the priority of the output contents output from the other participants to the first participant based on the intimacy estimated by the intimacy estimation unit. It has a unit and an output unit that outputs the output content to the first participant based on the output priority calculated by the output priority calculation unit.
  • Configuration of users / devices / networks in a remote communication system Configuration of general information equipment used in FIG.
  • User information stored on the server.
  • a modified example of FIG. Analysis example of user's face image. An example of data showing the judgment value of the analysis of FIG. The whole processing flowchart which concerns on 1st Embodiment. Feedback data sent to the server on a regular basis.
  • FIG. 1 is a configuration of a user, a device, and a network in a remote communication system according to the first embodiment of the present invention.
  • the server 1 is connected to the presenting user 3 and the listening users 6a to 6f via the net line of the general communication network, and analyzes the contents of the remote communication between these users.
  • the presentation user 3 makes a presentation to the listening users 6a to 6f through the information terminal (PC) 2.
  • Each of the listening users 6a to 6f is watching the presentation of the presentation user 3 via the information terminal (PC) 2 and the information terminal (pad) 7.
  • the listening users 6a to 6f are divided into a first group 4 and a second group 5, and this grouping is, for example, a group for each company, a group for each department in the organization, a group for each age, and the like. , It is freely set according to the closeness of the relationship of the group of users who participate in remote communication.
  • each group there is a conversation that is shared only within the group, and the specific content of the conversation, the impression of the presentation content of the presentation user 3, and the content of emotional expression for the presentation are shared by text messages and the like.
  • the concrete conversation content is not transmitted to the outside of each group in principle, and instead, each audit for the presentation of the presentation user 3 is performed between the first group 4 and the second group 5. Only the output contents of emotional expressions such as emoticon icons (non-verbal reactions) representing the reactions of users 6a to 6f are shared.
  • FIG. 2 is a configuration of a general information device used in FIG.
  • the server 1 is composed of a CPU, a main memory for storing program data, and an external storage (device) such as a memory card.
  • the CPU has an input reception unit that accepts input from participants participating in the communication support system, an intimacy estimation unit that estimates the intimacy between users, and an output priority that calculates the output priority based on the intimacy between users. It is equipped with a degree calculation unit, an index calculation unit that calculates an index related to communication between users, and an output unit that determines and outputs output contents based on an output priority and an index related to communication. Details will be described later.
  • the presenting user 3 or the listening users 6a to 6f are output devices (keyboard, mouse, touch panel) such as a display, a camera, a microphone, and an input device (keyboard, mouse, touch panel) connected to a general communication network by an external bus such as USB.
  • the information terminal 2 (PC) or the information terminal 7 (pad) is used to view or transmit audio and video.
  • FIG. 3 is user information stored in the server.
  • FIG. 3A is a data table of user information
  • FIG. 3B is a data table related to the history of FIG. 3A.
  • FIG. 3 (a) profile information about the user who uses the remote communication system is managed.
  • This user information shown in FIG. 3 is an example of an inter-individual distance table from the viewpoint of user A.
  • the items in the data table of FIG. 3A include the group to which the user belongs, the relationship between a certain user A and each user (d is an identification number and has nothing to do with numerical calculation), and the history of text comments by comment number (for details, see FIG. 3 (b)) shows the intimacy coefficient between the user A and another user.
  • This intimacy coefficient is a value indicating the distance between individuals between users, and the intimacy changes depending on the settings, input contents, facial expressions, etc. of each user, which will be described later.
  • the intimacy coefficient shown in FIG. 3A indicates that the smaller the value, the higher the intimacy with the target partner. Further, as shown in FIG. 1, this user information is stored in the server 1 as being common to all users regardless of the presenter or the audience.
  • the intimacy between users can be judged from the contents of the history of text comments in FIG. 3 (b). For example, the number of histories between certain users (number of conversations) and how to convey a message to the other party in a text comment (determined by the frequency of use of a specific positive word) can be used as judgment materials.
  • the intimacy between users can be determined by grasping the relationship between users and the emotions of the other user based on the information set by the screens of FIGS. 4 to 7 described below.
  • the intimacy coefficient set according to the intimacy between the users determined in this way is used in the server 1 to determine the output content for each user, and affects the content output to the user.
  • FIG. 4 is an example of a screen for managing relationships between users.
  • Terminal 2 (7) outputs a screen for managing information about each user.
  • the user of the terminal 2 (7) can select, set, and edit the relationship with each user on the output screen according to the classification (classification of colleagues, customers, friends, acquaintances, etc.) and the degree of familiarity.
  • the familiarity value that can be set for each user is an element for calculating the intimacy coefficient (FIG. 3 (a)).
  • the value of familiarity shown in FIG. 4 indicates that the larger the value, the higher the intimacy. Even if the person is not close, if the other person is an important person (VIP, etc.), setting the familiarity value to a high value can affect the intimacy coefficient (importance coefficient). ..
  • FIG. 5 is an example of a screen for selecting information for the transmission target user.
  • Terminal 2 (7) outputs a screen for setting transmission information.
  • transmission information For example, in FIG. 5, for "colleagues", voice, text, anonymous keywords (specific keywords used without displaying their name to the target user are shared), facial expression information, emotion classification data ( It is set to be able to send (icon related to one's facial expression) and anonymous emotions (only emotions are shared without displaying one's name to the target user), and is not shared for facial expressions.
  • emotion classification data It is set to be able to send (icon related to one's facial expression) and anonymous emotions (only emotions are shared without displaying one's name to the target user), and is not shared for facial expressions.
  • extremely private information such as facial footage
  • the relationship between users is determined by the setting of this transmission information, and a value according to the relationship between users is added to the intimacy in the calculation in the intimacy estimation unit.
  • each information is not sent to the server 1, so that the complete confidentiality of the user information is ensured.
  • FIG. 6 is an example of a screen for setting the feedback amount for each user group.
  • the terminal 2 (7) outputs a screen for setting feedback weighting in order to incorporate highly important opinions and reactions to the group to which the user belongs and to reduce unimportant opinions and reactions as much as possible. There is.
  • the value of this display weighting is an input value that determines how much the reaction of which group is displayed. Therefore, a group in which this value is set to a large value will have a large display on its own information terminal. For example, on the setting screen of FIG. 6, since the display weight of a customer who is in a financial group is 0.9 (maximum value is 1), comments and reactions are greatly displayed to the user of this terminal. The presence on the screen of the terminal becomes large. However, since the display weighting of the group of subordinates of the company is set to 0.3, the comments and reactions on this terminal are displayed small, and the presence on the screen becomes small. In this way, the relationship between users is determined by the display weighting, and the intimacy estimation unit adds a value corresponding to this relationship to the intimacy.
  • FIG. 7 is an example of a screen for setting the amount of feedback regarding emotional expression.
  • Terminal 2 (7) outputs a screen for setting the aggregation of the emotions of the other user.
  • This is a screen for inputting and setting what kind of reaction is important. For example, as shown in FIG. 7, when you want to aggregate positive emotions, you can select a positive item in the emotion group to relate to positive emotions.
  • display weighting values can be set for each of facial expressions, voices, languages, and inputs. The larger the display weighting value, the greater the influence on the calculation result of the intimacy.
  • the display weighting value can be customized for each group or user, and each user's emotions are parameterized by the facial expressions judged from each user's facial image and the input from the UI on each user's terminal. be able to.
  • the emotional parameterization is performed by the index calculation unit calculating an index related to communication.
  • the familiarity value set on the screen of FIG. 4 indicating the relationship for each user, the type of transmission information set on the screen of FIG. 5, and the display weighting set on the screens of FIGS. 6 and 7, respectively.
  • the value of is used in the calculation of the intimacy of each user performed by the intimacy estimation unit in the server 1 (FIG. 2). That is, in the server 1, the intimacy estimation unit estimates the intimacy according to the relationship between the users by using these setting information.
  • FIG. 8 is an example of a model screen of the communication support system according to the first embodiment.
  • the presentation screen is output and displayed on the terminal 2 (7). Further, on the lower side of the presentation screen, the feedback time transition graph 12 and the video 13 of another user participating are projected.
  • the feedback time transition graph 12 shows the reaction (emotion) of each user to the presentation as parameters, and the graph flows to the left with the passage of time. In this graph, the larger the number (there are many good reactions to the presentation), the higher the graph.
  • the quantified feedback from the feedback time transition graph 12 allows the presenter to visually understand the current excitement of the audience, which allows the presenter to proceed with the conference while feeling a realistic reaction. More specifically, the index calculation unit calculates an index according to the reaction of each user based on the value of the emotion display weighting described with reference to FIG. 7, and the output unit feeds back using the value of this index as a parameter. By determining the display content of the time transition graph 12, the reaction of each user is parameterized and reflected in the feedback time transition graph 12.
  • the facial image 13 of another user is a real image or image of a face taken by a camera, a facial expression reproduction model (CG or virtual model that reproduces a facial expression based on the movement of a real person's facial expression), and a facial expression reproduction icon (existing). Based on the movement of the person's facial expression, an icon with a similar facial expression is displayed). Which display is to be displayed is determined by the server 1 for each user based on the output priority calculated by the output priority calculation unit based on the intimacy between the users. For example, users with higher intimacy and higher output priority display images that are closer to real images and images, and by changing the display so that the lower the output priority, the higher the level of abstraction, intimacy and anonymity. It is possible to display both sexes.
  • Word cloud 18 that aggregates and describes trend words in conversation and a text field 19 that expresses chat shared during the presentation are displayed.
  • Word cloud 18 can display the trending words in a large size by aggregating the comments of the users who are participating in the conference, and what kind of keywords are used by the users who are watching the presentation as a whole. You can visualize what you are doing.
  • the comments of the users who are participants are displayed in the text field 19, some users may not be displayed due to the transmission information setting (FIG. 5) described above.
  • a video 16 displaying one's facial expression and a transmission data 17 indicating transmission information are displayed below the Word cloud 18. Further, a voice ON / OFF button 14 and a video ON / OFF button 15 for setting ON / OFF of one's own voice and facial expression are provided.
  • FIG. 9 is a modified example of FIG.
  • FIG. 9 the Word cloud 18 and the text field 19 in FIG. 8 are replaced with the feedback 2-axis graph 20.
  • the feedback 2-axis graph 20 a plot 21 representing the group importance is displayed based on the definition set in the 2-axis graph.
  • FIG. 9 For example, as shown in FIG. 9, "concentration” (how many listening users are listening without showing away, whether the number of nods to the speaker is high) and “positiveness” (speaker's).
  • There is a two-axis graph (such as inputting a specific keyword that has a favorable impression on the content of the presentation, the time of a smile that can be read from the facial expression of the listening user, etc.).
  • This graph shows the result of totaling the indexes related to communication between users calculated by the index calculation unit for each user, and is based on the output priority calculated by the output priority calculation unit based on the intimacy between users.
  • the display content is determined on the server 1.
  • model screen examples in FIGS. 8 and 9 are drawn assuming a PC terminal, a pad terminal may also be used.
  • FIG. 10 is an analysis example of the user's face image. Further, FIG. 11 is an example of data showing the determination value of the analysis of FIG. Note that FIG. 11A is an example of feature point data to be transmitted as “facial expression information”, and FIG. 11B is an example of data to be transmitted as “emotion classification data”.
  • the output unit By analyzing the facial expression of the user in the index calculation unit of the server 1 using the camera captured image 22, it is possible to determine the emotion of the user corresponding to the facial expression and calculate the index related to communication. Based on the index calculated in this way, the output unit generates a facial expression reproduction model or a facial expression reproduction icon.
  • the analysis method creates a data table by digitizing and extracting a set number of points (feature points) that are characteristic of facial expressions in x-coordinates and y-coordinates. As a result, as shown in FIG. 11A, the coordinate values for each feature point in the feature point extraction image 23 are extracted. Then, the facial expression reproduction model 24 of FIG.
  • the facial expression reproduction model 24 may be generated by changing the facial expression reproduction model 24 according to the user's emotions to further understand the user's emotions.
  • FIG. 11A only four feature points are shown, but in reality, it is determined by 30 or more feature points.
  • the index calculation unit can calculate an index related to communication from the feature point extraction image 23 using a machine learning discriminator, and can determine the facial expression icon 25 based on the calculation result. As shown in FIG. 11B, the facial expression icon 25 is determined by calculating a determination value as an index related to communication for each facial expression item such as a smile or a nod.
  • the machine learning discriminator is, for example, SVM (Support vector machine), NN (Neural Network), or the like.
  • the determination value (index) for each facial expression item calculated when the facial expression icon 25 is determined may be used in the calculation of the intimacy of each user performed by the intimacy estimation unit. For example, among the weighted values of the display for each detailed item set on the screen of FIG. 7, the weighted value related to the facial expression can be determined whether or not to be adopted in the calculation of intimacy based on this determination value.
  • the facial expression reproduction model 24 and the facial expression icon 25 created as described above are images that reproduce the facial expressions of each user based on the index related to communication between users. These are transmitted from the server 1 to the terminal of each user, and are displayed as a video 13 on the model screen shown in FIGS. 8 and 9, for example, to output to each user. Note that FIG. 10 shows the output based on facial expression analysis. Similarly, based on the input from the voice, the sound is fed back by applying the sound to a machine learning classifier or the like based on the recognition of the laughing voice and the volume of the voice. It also has a mechanism to do it.
  • cultural differences can be absorbed by setting conversion rules for each nationality of participating users in the recognition mechanism. For example, it is possible to eliminate the difference by setting different gesture and smile thresholds for each country and reflecting emotions in the parameters.
  • FIG. 12 is an overall processing flowchart according to the first embodiment.
  • the personal client operation is the content operated by the terminal of the PC or the pad.
  • the information logged in in step S2 is also shared with the server (described later in step S104).
  • Steps S3 to S12 are flows (loop processing) that are repeated at regular time intervals, and the loop processing is started in step S3.
  • step S4 the voice information obtained by the microphone recording operation at the time of the user using the terminal is acquired.
  • step S5 the voice information of the user using the terminal is processed by converting the recorded information of the microphone into text and estimating the emotional information from the recorded information and the text.
  • step S6 the user using the terminal acquires the video information obtained by recording the camera at the time.
  • step S7 the facial features obtained from the video information of the user using the terminal are recognized from the image information of the camera acquired in step S6, and the information related to emotions is estimated according to the degree of the features. ..
  • step S8 the input information to be transmitted to the server is acquired based on the information acquired / processed in steps S4 to S7.
  • step S9 Send the input information to the server in step S9.
  • the information to be transmitted is input by the input reception unit of the server.
  • This transmission is an asynchronous communication in which the data transmitted to the buffer is read at the timing of reading the audio / video information and is processed.
  • the transmission of step S9 or the reception of step S10 is periodically transmitted / received each time the loop processing of steps S3 to S12 is performed.
  • step S11 the terminal outputs an image (face image, facial expression reproduction model, facial expression icon) based on the information received from the server.
  • Server 1 is started in step S101.
  • step S102 the personal information database (DB) is read.
  • DB personal information database
  • Steps S103 to S112 are flows (loop processing) that are repeated at regular time intervals, and the loop processing is started in step S103.
  • step S104 in the operation of the individual client, the login information of step S2 is received and a new login is accepted.
  • step S105 the loop process is started for a certain user A among the logged-in users.
  • step S106 the information periodically transmitted from the terminal in step S9 is received.
  • the input receiving unit accepts the input of the terminal.
  • step S107 the inter-individual distance between the user A and a plurality of other users is confirmed, the output content is determined, and the loop process of transmitting to the terminal of the user A is started.
  • step S108 the distance between the individual user A and another user is calculated and confirmed.
  • the index calculation unit calculates the index
  • the intimacy estimation unit calculates the intimacy
  • the output priority calculation unit outputs based on the calculated intimacy. The priority is calculated
  • the output unit determines the output content at the terminal based on the calculated output priority.
  • the output content determined here includes the facial expression reproduction model and display reproduction icon of each user displayed as the video 13 of FIGS. 8 and 9, the display content in Word cloud 18 and the text field 19.
  • step S109 the information of the output content determined in step S108 is transmitted to the terminal of the user A. This transmission is asynchronous communication similar to step S9.
  • step S110 the distance between the individual user A and another user is confirmed, the output content is determined, and the loop process of transmitting to the terminal of the user A is terminated.
  • each user follows the output priority according to the personal distance (intimacy) between the user A and each other user.
  • the output content to be output to the user A is determined, and the information of the output content is transmitted from the server 1 to the terminal of the user A.
  • step S111 the loop process targeting user A is terminated.
  • the loop processing of steps S105 to S111 for each user the output content in the terminal of each user is determined, and the information of the output content is transmitted from the server 1 to the terminal of each user.
  • step S112 the loop processing of steps S103 to S112 is terminated.
  • FIG. 13 is an example of feedback data periodically transmitted to the server according to the first embodiment.
  • the feedback data shown in FIG. 13 is the information transmitted in step S9 of FIG. Each item will be explained.
  • a flag as to whether or not to disclose the voice information of the user A is shown to 12 users other than the user A who are participating in the conference, such as [10010 ...]. ing. In this way, it is determined to whom information about oneself can be disclosed. It is not disclosed to the group whose corresponding flag is 0.
  • the file type is recorded in the voice information data column. Similar to the voice information disclosure flag field, the text information disclosure flag column shows a flag as to whether or not the user A discloses voice information to 12 users other than the user A. In the text information data field, the text content input by the user A is recorded.
  • the face image disclosure flag column is the same as the voice information disclosure flag column and the text information disclosure flag column. Face image data is recorded in the face image data column.
  • the facial expression information disclosure flag column is the same as the facial image disclosure flag column, the voice information disclosure flag column, and the text information disclosure flag column.
  • a coordinate value list is recorded in the facial expression information data column. This is a value used for constructing the above-mentioned facial expression reproduction model (see FIG. 10).
  • the emotional information disclosure flag column is the same as the facial expression information disclosure flag column, the face image disclosure flag column, the voice information disclosure flag column, and the text information disclosure flag column.
  • the determination value for the facial expression icon (see FIG. 10) analyzed from the facial image is recorded.
  • the above feedback data is sent to the server, and the intimacy estimation unit determines the intimacy coefficient, which is information regarding the distance between each user and the user A.
  • the above information of each user is totaled by the display weighted value set on the emotion aggregation setting screen (FIG. 7) in the index calculation unit, and the emotion aggregation value for each user is calculated.
  • This purpose-specific emotion aggregate value is transmitted from the server 1 to each user's terminal, and is used for graph display on the screen as shown in the feedback 2-axis graph 20 of FIG. 9 in each terminal. , Expressed as a plot group as the degree of concentration. Further, the correlation with the reception display side user is calculated and used for the calculation of the sympathy value in the inter-individual distance (described later in FIG. 14).
  • FIG. 14 is a flowchart showing the process of calculating the inter-individual distance according to the first embodiment. The flowchart will be described below.
  • the inter-individual distance table created in FIG. 14 is used in the calculation / confirmation of the inter-individual distance in step S108 of FIG. 12, and is created independently of the flow of FIG.
  • step S20 user A (screen viewer) is selected.
  • step S21 a loop process for determining the distance between individuals with each viewer (user X) participating in the conference is started.
  • step S22 it is confirmed whether or not there is target person information. If this target person information does not exist in the information of user A, it is determined that there is no relationship.
  • step S23 the data of the classification group of the target person is acquired.
  • step S24 the importance information is input for each group acquired in step S23.
  • the importance information referred to here is the display weighting (FIG. 6) in the feedback weighting setting.
  • step S25 a numerical value of "friendliness" preset for each user is input from the management screen of the terminal (FIG. 4).
  • step S26 the intimacy in the conversation is calculated from the number of times the user A has talked with each user. The numerical value and the calculated value input in the flow from step S24 to step S26 are used in the calculation in which the intimacy estimation unit determines the inter-individual distance (intimacy).
  • step S27 the correlation with the emotion data of the user A regarding the user X is calculated, and the sympathy value between the user A and the user X is calculated.
  • the feedback data of FIG. 13 is also used to calculate this sympathy value. If the positive correlation is higher than a certain threshold, the empathy value is added to the calculation of the inter-individual distance.
  • step S28 the output priority calculation unit determines the inter-individual distance between the user A and the user X based on the flow of steps S22 to S27.
  • step S29 the loop processing of steps S21 to S29 is terminated.
  • the data table regarding the inter-individual distance of the user A is completed in the step S30. This inter-personal distance data table is created for each user from a certain user A, and the output is determined based on the aggregated data table.
  • the intimacy between users is estimated from the relationship between users and the state of communication, and based on the estimation result, from the user to other users.
  • Output (comments, utterances) can be prioritized. For example, by clearly indicating or outputting a large comment to a participant who has a high degree of intimacy, the comment that is of high interest to the participant can be output preferentially, so that remote communication with a sense of presence can be realized. Can be done. It also calculates indicators related to communication from the user to other users (for example, agree, disagree, understand), and based on this and output priority, outputs a graph expressing the value of the indicator, or an image based on the indicator. Images (for example, user's face image, facial expression reproduction model, facial expression reproduction icon, etc.) can be output.
  • FIG. 15 is an example of a model screen of the communication support system according to the second embodiment.
  • the point of the communication support system of the second embodiment is for exhibitions on VR (Virtual Reality), and is remote communication performed in VR space.
  • VR Virtual Reality
  • the points common to the first embodiment will be omitted, and the differences will be mainly described.
  • the first exhibit 28 In the VR space image 27 output by the terminal 2 (7), the first exhibit 28, the facial expression reproduction icon 29 of the user who is looking at the first exhibit, and the second exhibit 30 are projected. ..
  • each user (avatar) in the VR space can grasp the direction, position, and distance of the line of sight in the virtual space three-dimensionally on the screen.
  • the screen of the facial expression information 32 of the user who is watching can also be used to visualize the atmosphere of the user who is watching the same object. This makes it possible to understand the facial expressions and atmospheres of the users participating in the VR space.
  • FIG. 16 is feedback data periodically transmitted to the server according to the second embodiment.
  • a column for avatar position operation information and a column for avatar angle operation information have been added to the data table described with reference to FIG.
  • the column of avatar position operation information the position in the VR space by the XYZ coordinates is shown.
  • the column of the avatar angle operation information the numerical value regarding the line-of-sight direction in which the avatar is looking at the object in the VR space is shown. That is, the operation information of the avatar is sent together with the feedback data.
  • FIG. 17 is a calculation of the inter-individual distance according to the second embodiment.
  • step S28A The difference from the calculation of the inter-individual distance of the first embodiment shown in FIG. 14 is step S28A and step S29A.
  • step S28A if the user X sees the same object as the user A, the value of the intimacy according to the relationship between the users is added in the calculation of the interpersonal distance between the user A and the user X. ..
  • step S29A if the user X is within a certain distance from the user A in the VR, the intimacy according to the relationship between the users is calculated in the calculation of the interpersonal distance between the user A and the user X. Add the values. This completes the inter-individual distance table in the VR space (step S32A).
  • FIG. 18 is a configuration example of a model of the communication support system according to the third embodiment.
  • the point in the third embodiment is remote communication via the avatar robot. Similar to the description of the second embodiment, the points common to the first embodiment will be omitted, and the differences will be mainly described.
  • the avatar robot 37 is equipped with an omnidirectional camera 34 and is operated at the presentation site 36 set up in the real space.
  • the omnidirectional image 33 taken by the avatar robot 37 using the omnidirectional camera 34 is a projection of the on-site presenter 35, the on-site presenter 39, and the like existing around the avatar robot 37.
  • the avatar robot 37 transmits and projects the captured omnidirectional image 33 to each user 6aB to 6dB.
  • Each user 6aB to 6dB can view the omnidirectional image 33 by an output terminal such as a PC, a pad, a smartphone, or a VR headset.
  • Each user 6aB to 6dB is watching the omnidirectional video 33, but the user 6bB, the user 6cB, and the user 6dB are watching the same object (object video 33b) in the omnidirectional video 33, and the user 6aB is all. You are looking at another object (object image 33a) in the surrounding image 33. Therefore, in FIG. 18, the user 6aB does not participate in the feedback from the user 6bB to the user 6dB to the site 36.
  • the server creates a facial expression reproduction model, a facial expression icon, etc. from the appearance and reaction of the user 6bB to the user 6dB watching the omnidirectional image 33, and feeds back to the site 36.
  • the output of the feedback content is transmitted as feedback to the screen 38 in front of the on-site presenter 39 and the screen 41 in front of the on-site presenter 35 on the 360-degree four-sided display 40 installed in the avatar robot 37, respectively.
  • the feedback content is, for example, screen 42.
  • the content is, for example, the impression of the user 6 dB on the content of the presentation by the on-site presenter 35.
  • FIG. 19 is a user viewing content, which is an example of a user viewing screen model according to the third embodiment. Similar to the description of FIG. 18, the points common to the first embodiment and the second embodiment will be omitted, and the differences will be mainly described.
  • the difference between the first embodiment (see FIGS. 8 and 9) and the second embodiment (see FIG. 15) is that the user's viewpoint image 46 in the avatar robot is used for the screen of the terminal 2 (7). That is.
  • the user's viewpoint image 46 displays a presenter A39, a user B's avatar 44, a facial expression reproduction icon 45 representing the user B's feedback content, a user C's avatar 48, and a face image icon 47 representing the user C's feedback content.
  • the user's viewpoint image 46 is accompanied by another user's avatar (Avatar 44 in FIG. 19). 48) is coming to appear.
  • the viewing angle of another user moves to a viewing angle deviating from a certain viewing angle of the user, the icon of the other user disappears from the user's screen.
  • FIG. 20 is an example of a model screen of the avatar robot according to the third embodiment.
  • FIG. 20 shows the state of the avatar robot as seen from the presenter A.
  • the emotion icon 45 which is the feedback content of the user B
  • the face image icon 47 which is the feedback content of the user C
  • the facial expression information 49 which is the feedback content of the user A
  • FIG. 21 is feedback data transmitted periodically according to the third embodiment.
  • FIG. 21 shows that the difference between the first embodiment and the second embodiment is that the viewpoint robot selection information and the line-of-sight angle information are included in the items.
  • the viewpoint robot selection information column the identification number of the robot is recorded by selecting one robot from the plurality of robots in the field by the user.
  • the line-of-sight angle information information about the line-of-sight angle currently viewed by the user from the robot having the identification number selected in the viewpoint robot selection information field is recorded.
  • FIG. 22 is a calculation of the inter-individual distance according to the third embodiment.
  • FIG. 22 shows the difference between the first embodiment (see FIG. 14) and the second embodiment (see FIG. 17) in step S28B and step S29B.
  • step S28B when the user X is looking at the camera of the same robot among the plurality of avatar robots having the user A, that is, the avatar robot that outputs the image to be viewed by the user X and the avatar robot visually recognized by the user A.
  • the value of the intimacy according to the relationship between these users is added.
  • step 29B based on step 28B, if the line-of-sight direction of user X is close to the line-of-sight direction of user A, that is, when the line-of-sight direction of user X is within a predetermined error range with the line-of-sight direction of user A, the user.
  • the value of the intimacy according to the relationship between these users is added.
  • the inter-individual distance table is completed by the flow of steps S20B to 32B including this.
  • the server 1 is an input reception unit that receives input from participants participating in the communication support system, and among a plurality of participants.
  • the intimacy estimation unit that estimates the intimacy between the first participant and the other participants based on the information about the first participant and the other participants, and the parent estimated by the intimacy estimation unit.
  • An output priority calculation unit that calculates an output priority that determines the priority of output contents output from other participants to the first participant based on the density, and an output calculated by the output priority calculation unit. It has an output unit that outputs the output contents to the first participant based on the priority.
  • the server 1 has an index calculation unit that calculates an index related to communication between the first participant and other participants.
  • the output unit determines the output content based on the output priority calculated by the output priority calculation unit and the index calculated by the index calculation unit. By doing so, it is possible to give feedback according to the communication situation between the participants to the presenter who is making a presentation to other participants.
  • the output unit is taken as an input received by the input reception unit, for example, with a camera.
  • the image of the face of another participant is output as it is, and if it is lower than the predetermined value, for example, a facial expression reproduction model or a facial expression reproduction icon is output as an image based on an index related to communication.
  • the intimacy estimation unit sets the relationship from the first participant to the other participants by the screen of FIG. 4, and from the first participant to the other participants by the screen of FIG.
  • the intimacy can be estimated based on the type of transmission information set for the subject and the amount of feedback set from the first participant to the group to which the other participants belong from the screen of FIG. .. By doing so, it is possible to estimate the intimacy in consideration of the relationship between users.
  • the image based on the index related to communication includes a facial expression reproduction model 24 which is a model image that reproduces the facial expressions of the participants.
  • the output unit can create a facial expression reproduction model 24 from a camera-photographed image 22 of each participant's face. By doing so, it is possible to create a sense of intimacy while ensuring anonymity for a person who has a certain sense of intimacy.
  • the image based on the index related to communication includes the facial expression reproduction icon 25, which is an icon that reproduces the facial expression of the participant.
  • the output unit can create the facial expression reproduction icon 25 from the camera-captured image 22 of each participant's face. By doing so, it is possible to give feedback by expressing emotions that ensure anonymity to a person who is not acquainted.
  • the intimacy estimation unit is used when the avatars of other participants are looking at the same target as the avatars of the first participant, or other participants.
  • the first When at least one of the avatars of the first participant is within a certain distance from the avatar of the first participant, in the calculation of the intimacy between the first participant and the other participant, the first. Add the values according to the relationship between one participant and other participants. By doing so, the intimacy can be estimated from the user's avatars in the VR space.
  • the intimacy estimation unit visually recognizes the avatar robot that outputs the video to be viewed by other participants and the first participant among the plurality of avatar robots.
  • the first is when at least one of the following is satisfied when the avatar robot is the same as the robot, or when the viewing direction of another participant is within a predetermined error from the viewing direction of the first participant.
  • a value corresponding to the relationship between the first participant and the other participant is added. By doing so, even when the avatar robot is used, it is possible to calculate the inter-individual distance between the users and estimate the intimacy between the users.
  • the present invention is not limited to the above embodiment, and various modifications and other configurations can be combined within a range that does not deviate from the gist thereof. Further, the present invention is not limited to the one including all the configurations described in the above-described embodiment, and includes the one in which a part of the configurations is deleted.
  • Avatar (user) looking at the 1st exhibit Facial expression reproduction icon 30 ... 2nd exhibit 31 ... Feedback information 32 ... Looking at the 1st exhibit User's facial expression information 33 ... All-around image of the avatar robot 33a, 33b ... Object 34 ... All-around camera 35 ... On-site presenter B reflected in the all-around image of the avatar robot 36 ... Site 37 ... Abata (operating) robot 38 ... Screen in front of site presenter A 39 ... Site presenter A 40 ... 360-degree four-sided display 41 ... Screen in front of presenter B on site 42 ... Screen on which the facial expression feedback of the viewing user is reflected 44 ... Avatar 45 of user B ... Facial expression reproduction icon 46 of user B ... User's viewpoint image (Abata robot) ) 47 ... Face image icon of user C 48 ... Avata of user C 49 ... Facial expression reproduction model of user A

Abstract

La présente invention concerne un système d'aide à la communication servant à apporter une assistance dans une communication à distance à réaliser par une pluralité de participants. Le système d'aide à la communication comprend: une unité de réception d'entrée qui reçoit une entrée provenant des participants qui participent au système d'aide à la communication; une unité d'estimation de niveau d'intimité qui estime un niveau d'intimité entre un premier participant et un autre participant parmi la pluralité de participants d'après des informations se rapportant au premier participant et à l'autre participant; une unité de calcul de priorité de sortie qui, d'après le niveau d'intimité estimé par l'unité d'estimation de niveau d'intimité, calcule un priorité de sortie servant à déterminer la priorité d'un contenu de sortie à délivrer de l'autre participant au premier participant; et une unité de sortie qui délivre le contenu de sortie au premier participant d'après la priorité de sortie calculée par l'unité de calcul de priorité de sortie.
PCT/JP2020/048863 2020-12-25 2020-12-25 Système d'aide à la communication WO2022137547A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/048863 WO2022137547A1 (fr) 2020-12-25 2020-12-25 Système d'aide à la communication

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/048863 WO2022137547A1 (fr) 2020-12-25 2020-12-25 Système d'aide à la communication

Publications (1)

Publication Number Publication Date
WO2022137547A1 true WO2022137547A1 (fr) 2022-06-30

Family

ID=82158011

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/048863 WO2022137547A1 (fr) 2020-12-25 2020-12-25 Système d'aide à la communication

Country Status (1)

Country Link
WO (1) WO2022137547A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024009623A1 (fr) * 2022-07-04 2024-01-11 パナソニックＩｐマネジメント株式会社 Système d'évaluation, dispositif d'évaluation et procédé d'évaluation

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004258819A (ja) * 2003-02-24 2004-09-16 Fujitsu Ltd 電子コミュニケーション支援サーバ
JP2005235142A (ja) * 2004-01-21 2005-09-02 Nomura Research Institute Ltd ユーザ間親密度測定システム及びユーザ間親密度測定プログラム
JP2006050370A (ja) * 2004-08-06 2006-02-16 Sony Corp 情報処理装置および方法、記録媒体、並びにプログラム
JP2008107895A (ja) * 2006-10-23 2008-05-08 Nomura Research Institute Ltd 仮想空間提供サーバ、仮想空間提供システム及びコンピュータプログラム
JP2009124606A (ja) * 2007-11-16 2009-06-04 Sony Corp 情報処理装置、情報処理方法、プログラム及び情報共有システム
JP2017229060A (ja) * 2016-06-22 2017-12-28 富士ゼロックス株式会社 会議コンテンツを表現する方法、プログラム、及び装置
JP2018089227A (ja) * 2016-12-06 2018-06-14 株式会社コロプラ 情報処理方法、装置、および当該情報処理方法をコンピュータに実行させるためのプログラム
JP2019219987A (ja) * 2018-06-21 2019-12-26 日本電信電話株式会社 集団状態推定装置、集団状態推定方法及び集団状態推定プログラム

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004258819A (ja) * 2003-02-24 2004-09-16 Fujitsu Ltd 電子コミュニケーション支援サーバ
JP2005235142A (ja) * 2004-01-21 2005-09-02 Nomura Research Institute Ltd ユーザ間親密度測定システム及びユーザ間親密度測定プログラム
JP2006050370A (ja) * 2004-08-06 2006-02-16 Sony Corp 情報処理装置および方法、記録媒体、並びにプログラム
JP2008107895A (ja) * 2006-10-23 2008-05-08 Nomura Research Institute Ltd 仮想空間提供サーバ、仮想空間提供システム及びコンピュータプログラム
JP2009124606A (ja) * 2007-11-16 2009-06-04 Sony Corp 情報処理装置、情報処理方法、プログラム及び情報共有システム
JP2017229060A (ja) * 2016-06-22 2017-12-28 富士ゼロックス株式会社 会議コンテンツを表現する方法、プログラム、及び装置
JP2018089227A (ja) * 2016-12-06 2018-06-14 株式会社コロプラ 情報処理方法、装置、および当該情報処理方法をコンピュータに実行させるためのプログラム
JP2019219987A (ja) * 2018-06-21 2019-12-26 日本電信電話株式会社 集団状態推定装置、集団状態推定方法及び集団状態推定プログラム

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024009623A1 (fr) * 2022-07-04 2024-01-11 パナソニックＩｐマネジメント株式会社 Système d'évaluation, dispositif d'évaluation et procédé d'évaluation

Similar Documents

Publication Publication Date Title
EP3874350B1 (fr) Agents intelligents pour la gestion des données associées à des objets tridimensionnels
Kirk et al. Home video communication: mediating'closeness'
US8791977B2 (en) Method and system for presenting metadata during a videoconference
US8243116B2 (en) Method and system for modifying non-verbal behavior for social appropriateness in video conferencing and other computer mediated communications
US20120204120A1 (en) Systems and methods for conducting and replaying virtual meetings
US20120204118A1 (en) Systems and methods for conducting and replaying virtual meetings
JP5959748B2 (ja) オーケストレーション・モデルを実施するビデオ会議システム
Terken et al. Multimodal support for social dynamics in co-located meetings
US20120204119A1 (en) Systems and methods for conducting and replaying virtual meetings
US9299178B2 (en) Generation of animated gesture responses in a virtual world
TWI795759B (zh) 線上會議系統
US11651541B2 (en) Integrated input/output (I/O) for a three-dimensional (3D) environment
WO2022137547A1 (fr) Système d'aide à la communication
JP2003323628A (ja) 発言者特定映像装置およびそのプログラムならびに発言者特定映像表示方法
Otsuka Multimodal conversation scene analysis for understanding people’s communicative behaviors in face-to-face meetings
JP2021192186A (ja) コンピュータプログラム、情報処理装置及び会合開催方法
Tanaka et al. Appearance, motion, and embodiment: unpacking avatars by fine‐grained communication analysis
JP2016162001A (ja) 業務支援装置および業務支援プログラム
JP6901190B1 (ja) 遠隔対話システム、遠隔対話方法及び遠隔対話プログラム
Myodo et al. Issues and solutions to informal communication in working from home using a telepresence robot
JP2023146875A (ja) 情報処理装置、情報処理方法、及びプログラム
WO2012109006A2 (fr) Systèmes et procédés de conduite et de relecture de réunions virtuelles
Colburn et al. Graphical enhancements for voice only conference calls
RU2218593C2 (ru) Способ телекоммуникаций в компьютерных сетях
JP7465040B1 (ja) コミュニケーション可視化システム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20967033

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20967033

Country of ref document: EP

Kind code of ref document: A1