JP7512774B2 - COMMUNICATION SYSTEM, USER TERMINAL, PROGRAM, AND METHOD - Google Patents

Description

The present invention relates to a communication system, a user terminal, a program, and a method.

In recent years, online conferencing systems have become popular as a means for users belonging to companies and other organizations to share their situations online and communicate with each other.

For example, Patent Document 1 discloses an online conference system consisting of a terminal used by users and a server that manages online conference reservations, progress, and task processing. In managing reservations, the organizer of the online conference uses the terminal to access the server and registers the conference reservation. The server sends a conference hosting notification to the email addresses of those who plan to attend. The server estimates the level of human relationships within the organization based on the frequency of communication exchanged during the conference.

Patent Document 2 also discloses a method for storing behavior and situations (topics, meeting types, etc.) in association with each other based on meeting minutes data, and estimating the level of human relationships in each situation based on the behavior and situations stored in association with each other.

JP 2003-153224 A JP 2013-182560 A

However, in the online conference system described in Patent Document 1, it is difficult for a user who wants to host a conference to understand the other party's situation (whether they are busy or not, whether they are concentrating or not, whether they are talking to someone or not, whether they are in a bad mood or not, etc.), so there is a concern that the user may hesitate to hold a conference. This concern is not resolved even by the method described in Patent Document 2.

The present invention has been made in consideration of the above problems, and the object of the present invention is to provide a new and improved communication system, user terminal, program, and method that can support the smooth start of communication.

In order to solve the above problem, according to one aspect of the present invention, there is provided a communication system having a first user terminal used by a first user and one or more second user terminals used by one or more second users, the first user terminal including an operation unit that detects an operation to set a sharing level for each of the one or more second users, and a display control unit that generates, for each of the second users, a display screen on which screen elements showing the status of the second user are arranged in a manner corresponding to the sharing level set for the second user.

The display screen may include a movable display corresponding to each of the second users, and the operation unit may detect an operation of moving the movable display as an operation of setting the sharing level, and the sharing level set for the second users may change in response to the movement of the movable display.

On the display screen, the screen element for the second user may be positioned within a movable display corresponding to the second user.

On the display screen, the screen element for the second user may be arranged in a screen area different from the movable display corresponding to the second user.

When a first sharing level is set for the second user, an image of the second user is placed on the display screen as a screen element indicating the status of the second user, and the first user can speak to the second user, and when a second sharing level is set for the second user, text indicating the status of the second user may be placed on the display screen as a screen element indicating the status of the second user.

The system may also include a sharing level setting unit that matches the sharing level set by the first user to a second user when the sharing level set by the second user to the first user is different.

The device may also include a sharing level setting unit that limits the sharing level that each of the second users can set for the first user when a predetermined operation is detected by the operation unit of the first user terminal.

The first user terminal may further include a conference unit that controls the startup of a conference system in which the first user and the second user participate when a conference start operation is detected by the operation unit when a sharing level is set at which the first user and the second user can communicate with each other.

The system may also include a human relationship estimation unit that estimates human relationships between multiple users among the first user and the one or more second users based on information indicating operations on the first user terminal and each of the one or more second user terminals.

The human relationship estimation unit may estimate the human relationships between the multiple users based on at least one of the following: a history of sharing levels set by each of the multiple users for the other users, a user designating a specific other user and limiting the sharing level that the other user can set for the user, or a combination of users who participated in the implemented conference system.

The system may also include a sharing level setting unit that changes the sharing level set by the first user to the second user to a higher sharing level.

A user terminal is provided that operates as a first user terminal used by a first user, and includes an operation unit that detects an operation to set a sharing level for each of one or more second users, and a display control unit that generates, for each of the second users, a display screen on which screen elements indicating the status of the second user are arranged in a manner corresponding to the sharing level set for the second user.

In addition, in order to solve the above problem, according to another aspect of the present invention, a program is provided for causing a computer to function as a user terminal that operates as a first user terminal used by a first user, and that includes an operation unit that detects an operation to set a sharing level for each of one or more second users, and a display control unit that generates, for each of the second users, a display screen on which screen elements indicating the status of the second user are arranged in a manner corresponding to the sharing level set for the second user.

In order to solve the above problem, according to another aspect of the present invention, there is provided a method executed by a user terminal operating as a first user terminal used by a first user, the method including: detecting an operation to set a sharing level for each of one or more second users; and generating, for each of the second users, a display screen on which screen elements indicating the status of the second user are arranged in a manner corresponding to the sharing level set for the second user.

The present invention described above can help facilitate a smooth start to communication.

FIG. 1 is an explanatory diagram illustrating a communication system according to an embodiment of the present invention. FIG. 1 is an explanatory diagram showing a functional configuration of a communication system according to an embodiment of the present invention. 4 is a flowchart showing the overall operation of the communication system according to the embodiment of the present invention. FIG. 11 is an explanatory diagram showing a specific example of a virtual space screen. 13 is an explanatory diagram showing a specific example of spatial operations in the icon operation area 64. FIG. 13 is an explanatory diagram showing a specific example of spatial operations in the icon operation area 64. FIG. 11 is an explanatory diagram showing a specific example of screen elements arranged within an icon. FIG. 11 is an explanatory diagram showing a specific example of screen elements arranged within an icon. FIG. 11 is an explanatory diagram showing a specific example of screen elements arranged within an icon. FIG. FIG. 11 is an explanatory diagram showing an example of icon movement for starting a conference. FIG. 1 is an explanatory diagram showing the startup of the conference system. FIG. 11 is an explanatory diagram showing an example of icon movement for adding a participant to a conference. FIG. 13 is an explanatory diagram showing adding a participant to a conference. FIG. 13 is an explanatory diagram showing an example of a screen after the end of a conference; 1 is an explanatory diagram showing an example of an estimation result of human relationships between four users (user 1, user 2, user 3, and user 4). FIG. 1 is an explanatory diagram showing an example of an estimation result of human relationships between four users (user 1, user 2, user 3, and user 4). FIG. 2 is a block diagram showing the hardware configuration of a user terminal 20.

The following describes in detail an embodiment of the present invention with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals to avoid redundant description.

In addition, in this specification and drawings, multiple components having substantially the same functional configuration may be distinguished by adding different letters or numbers after the same reference symbol. For example, multiple components having substantially the same functional configuration or logical meaning may be distinguished as necessary, such as user terminals 20-1, 20-2, and 20-3. However, if there is no need to particularly distinguish between multiple components having substantially the same functional configuration, only the same reference symbol is assigned to each of the multiple components. For example, if there is no need to particularly distinguish between user terminals 20-1, 20-2, and 20-3, each user terminal will be simply referred to as user terminal 20.

<1. Overview of the communication system>
An embodiment of the present invention relates to a communication system in which a plurality of user terminals communicate, and in particular to a communication system in which a plurality of user terminals communicate for communication between users. An overview of the communication system according to an embodiment of the present invention will be described below with reference to Fig. 1. Note that, although an example in which the communication system according to an embodiment of the present invention is applied to communication in a company will be mainly described below, the communication system according to an embodiment of the present invention may be applied to other uses.

Figure 1 is an explanatory diagram showing a communication system according to one embodiment of the present invention. As shown in Figure 1, the communication system according to one embodiment of the present invention includes a plurality of user terminals 20, a plurality of overhead cameras 30, a server 40, and a human relationship display terminal 50.

The multiple user terminals 20, the multiple overhead cameras 30, and the human relationship display terminal 50 are connected to the server 40 via a network 12. The network 12 is a wired or wireless transmission path for information transmitted from devices connected to the network 12. For example, the network 12 may include public line networks such as the Internet, telephone line networks, and satellite communication networks, as well as various LANs (Local Area Networks) and WANs (Wide Area Networks) including Ethernet (registered trademark). The network 12 may also include dedicated line networks such as IP-VPN (Internet Protocol-Virtual Private Network).

(User terminal 20)
User terminal 20 is an information terminal used by a user. FIG. 1 shows user terminal 20-1 used by user 1, user terminal 20-2 used by user 2, and user terminal 20-3 used by user 3. Each user may be located at a remote location. For example, user 1 may be located at a different location from users 2 and 3. User 1 is an example of a first user, and user terminal 20-1 is an example of a first user terminal. The other users, user 2 and user 3, are examples of a second user, and user terminal 20-2 and user terminal 20-3 are examples of a second user terminal.

According to one embodiment of the present invention, even when multiple users are located at remote locations, smooth communication between the multiple users can be achieved by using the functions of the user terminal 20 and server 40, which will be described later.

Note that, although FIG. 1 shows a notebook PC (Personal Computer) as an example of the user terminal 20, the user terminal 20 may be other information terminals such as a desktop PC, a smartphone, or a tablet terminal.

(Overhead camera 30)
The overhead camera 30 is an imaging device that captures overhead video data. For example, the overhead camera 30-1 is installed at the base where the user 1 is located, and captures overhead video data including the video of the user 1. The overhead camera 30-2 is installed at the base where the users 2 and 3 are located, and captures overhead video data including the video of the users 2 and 3. Each overhead camera 30 transmits the captured overhead video data to the server 40 via the network 12.

(Server 40)
The server 40 collects media data indicating the status of each user from the user terminal 20 and the overhead camera 30 of each user, and transmits the media data to each user terminal 20. Examples of the media data include video data of the user captured by the user terminal 20, audio data collected by the user terminal 20, presence data indicating the status of the user in text, and overhead video data captured by the overhead camera 30. The server 40 according to one embodiment of the present invention transmits, among these media data, media data corresponding to a sharing level set by a certain user for other users to the user terminal 20 of that user.

The server 40 also has a function of estimating interpersonal relationships between multiple users based on operations on each user terminal 20 or information indicating the usage status of each user terminal 20.

(Human relationship display terminal 50)
The human relationship display terminal 50 is a display terminal that receives information indicating the human relationships between a plurality of users from the server 40 and displays a screen indicating the human relationships between the plurality of users.

(Identifying issues)
As a first technique related to one embodiment of the present invention, an online conference system can be considered, which comprises a terminal used by a user and a server that manages the reservation, progress, and task processing of an online conference. In managing the reservation, the organizer of the online conference uses the terminal to access the server and registers the reservation for the conference. The server sends a conference opening notification to the email addresses of the prospective participants. The server estimates the level of human relations within the organization based on the frequency of communication exchanged at the conference.

A second technique related to one embodiment of the present invention is a method of storing behavior and situations (topics, meeting type, etc.) in association with each other based on meeting minutes data, and estimating the level of human relationships in each situation based on the behavior and situations stored in association with each other.

However, the first technology above presupposes that a conference be booked, and there is a problem that if a user wants to hold an impromptu conference on the spot, the other party may not immediately notice the conference notification even if the reservation is made on the spot. In addition, there is a concern that users may hesitate to hold a conference because it is difficult to grasp the other party's situation (whether they are busy or not, whether they are concentrating or not, whether they are talking to someone or not, whether they are in a bad mood or not, etc.).

In the second technology described above, in order to estimate the level of human relationships, it is necessary to analyze words and actions and estimate both the statement level (a number indicating whether the statement is positive or negative) and who made the statement and to whom, for each statement, which poses the problem that the accuracy of the estimation cannot always be ensured.

The inventor of the present invention has come up with one embodiment of the present invention with the above circumstances in mind. One embodiment of the present invention is capable of resolving the problems of the first and second technologies. That is, one embodiment of the present invention makes it possible to visualize the other party's situation and smoothly start spontaneous communication. Also, one embodiment of the present invention makes it possible to estimate the interpersonal relationships between multiple users with a high degree of accuracy. The configuration and operation of such a communication system according to one embodiment of the present invention will be described in detail below.

<2. Configuration>
FIG. 2 is an explanatory diagram showing the functional configuration of the communication system according to an embodiment of the present invention.

(Configuration of user terminal 20)
As shown in FIG. 2, the user terminal 20 according to the embodiment of the present invention includes an operation unit 220 , a media input unit 230 , a media playback unit 240 , and a conference unit 250 .

The operation unit 220 detects operations performed by the user. For example, the operation unit 220 detects a setting operation in which the user sets a sharing level for other users (described later), a predetermined operation in which other users limit the sharing level that can be set for that user, and a conference start operation in which a conference is started using the conference system. The operation unit 220 outputs spatial operation data indicating these detected operations to the server 40.

The media input unit 230 outputs media data to the server 40 according to the media input parameters input from the server 40. The media input parameters are attribute data indicating at least one of video data, audio data, and presence data.

When the media input parameters indicate video data, the media input unit 230 outputs video data of the user captured by the user terminal 20 to the server 40. When the media input parameters indicate audio data, the media input unit 230 outputs audio data of the user collected by the user terminal 20 to the server 40. When the media input parameters indicate presence data, the media input unit 230 outputs presence data of the user estimated by the user terminal 20 to the server 40. Note that the media input parameters input to the media input units 230 of all user terminals 20 may be attribute data indicating at least audio data for audio calls.

The presence data of a user is useful for determining whether or not it is possible to call out to the user. For example, the user terminal 20 determines whether or not the user is in front of the user terminal 20 by using an image captured by the built-in camera of the user terminal 20. The user terminal 20 determines whether or not the user is busy based on the amount of operation of the user terminal 20 and schedule information (acquired from an external system). The user terminal 20 measures the amount of conversation using the sound from the built-in microphone of the user terminal 20 and determines whether or not the user is talking. The user terminal 20 measures the amount of gaze toward the terminal based on the image of the built-in camera, and determines whether or not the user is concentrating (concentrated/relaxed) based on the measurement result of the amount of gaze and the measurement result of the amount of conversation. The user terminal 20 then determines whether or not it is possible to call out to the user based on the various judgment results described above. The presence data of a user may be data indicating the results of such judgment.

The media playback unit 240 is an example of a display control unit that generates a display screen to be displayed by the display unit of the user terminal 20. For example, the media playback unit 240 plays media data about other users input from the server 40, and generates a display screen on which the played media data is arranged. The display screen corresponds to, for example, a virtual space screen, which will be described later.

The conference unit 250 controls the startup of the conference system based on the detection of a conference start operation by the operation unit 220.

(Configuration of Server 40)
As shown in FIG. 2, the server 40 according to an embodiment of the present invention includes a control unit 420 and a human relationship estimation unit 430 .

The control unit 420 has functions such as a sharing level setting unit that sets the sharing level between each user based on the spatial operation data input from each user terminal 20, a media control unit that controls the output of media input parameters to each user terminal 20 and the output of media data to each user terminal 20, and a conference control unit that outputs conference parameters to the user terminal 20 of the user who will be holding the conference. The control unit 420 also outputs the spatial operation data input from each user terminal 20 to the human relationship estimation unit 430.

The human relationship estimation unit 430 estimates the human relationships between multiple users based on spatial operation data input from the control unit 420, and outputs the estimation results to the human relationship display terminal 50.

<3. Operation>
Next, the operation of the communication system according to the embodiment of the present invention will be described. First, the processing of each step constituting the overall operation will be briefly summarized with reference to Fig. 3, and then the processing of each step will be described in detail with reference to the subsequent figures.

3 is a flowchart showing the overall operation of a communication system according to an embodiment of the present invention. As shown in FIG. 3, the overall operation of a communication system according to an embodiment of the present invention has the following steps of processing. Note that the following description focuses on the user terminal 20-1 used by user 1.
S1: Virtual space generation S2: Spatial operation detection (movement specification, meeting specification)
S3: Determination of media input parameters S4: Input of media data S5: Playback of media data S6: Control of conference S7: Estimation of human relationships S8: Display of human relationships S9: End determination

(S1: Virtual space generation)
When the user 1 instructs the user terminal 20-1 to display a virtual space screen (display screen) by operating the operation unit 220, the server 40 generates a virtual space screen based on a follower list (described later) stored for each user, and transmits a virtual space image to the user terminal 20-1. The media playback unit 240 of the user terminal 20-1 displays the virtual space screen on the display unit.

Figure 4 is an explanatory diagram showing a specific example of a virtual space screen. As shown in Figure 4, the virtual space screen includes a follower list area 61, a follower list area 62, a user search area 63, and an icon operation area 64.

The follower list area 61 is an area where the follower list, which is a list of other users who are of interest to user 1, is placed. The follower list area 62 is an area where the follower list, which is a list of other users who are of interest to user 1, is placed. The user search area 63 is a search area for adding other users who are not currently included in the follower list to the follower list. The icon operation area 64 is an area where the icon of user 1 himself and the icons of other users included in the follower list are placed. User 1 can move the icons of other users placed in the icon operation area 64 by operating the operation unit 220. In other words, the icon is an example of a movable display.

When user 1 uses this system for the first time, there is no corresponding person on the follower list. User 1 searches for other users in the user search area 63 and adds them to the follower list, and the icons of the other users are displayed in the icon operation area 64.

When user 1 uses the system for the second or subsequent time, the screen resumes from the previous state. In the icon operation area 64, user 1's icon (icon 1 in the figure) is fixedly positioned in the center, with the icons of other users positioned around it. User 1 can move the icons of the other users. User 1 can also perform operations to enlarge or reduce the area within the icon operation area 64.

In FIG. 4, three concentric regions are shown centered on the icon 1 of user 1. Different sharing levels (level 0, level 1, level 2, level 3, from the inside out) are defined for the regions surrounded by adjacent concentric circles. The above sharing levels can be thought of as the level of the means by which user 1 checks the status of other users and communicates with them. In one embodiment of the present invention, a smaller value for the sharing level indicates a higher sharing level, i.e., a closer means of communication.

The control unit 420 of the server 40 also manages whether other users included in the follower list are viewing user 1's video. Whether other users included in the follower list are viewing user 1's video can be determined from the display of the other user's icon. For example, as shown in FIG. 4, the icon 2 of user 2 who is not viewing user 1's video is displayed in a different manner from the icon 3 of user 3 who is not viewing user 1's video.

Furthermore, the control unit 420 of the server 40 manages whether or not other users included in the follower list are calling out to user 1. Whether or not other users included in the follower list are calling out to user 1 can be determined from the display of the icon of the other user. For example, as shown in FIG. 4, icon 4 of user 4 who is calling out to user 1 is displayed in a different manner from the icons of other users.

The above-mentioned viewing of the video of user 1, speaking to user 1, and the display content within the icon will be explained in detail in "S4: Input of media data" and "S5: Playback of media data".

(S2: spatial operation detection (movement designation, meeting designation))
When user 1 performs a move operation to move the icon of another user within the icon operation area 64 displayed on user terminal 20-1, the operation unit 220 detects the move operation as an operation to set/change the sharing level for other users to the sharing level corresponding to the destination area.

Furthermore, when user 1 performs an operation to select user 1's icon 1 within icon operation area 64, operation unit 220 detects this operation as a barrier setting operation for all other users. Barrier setting is a setting that restricts other users from viewing user 1's video data and from speaking to user 1. User 1 can set the restriction level and time length.

When the restriction level is set to 0, other users will be unable to place the icon of user 1 in the level 1 area in the icon operation area 64 displayed on their own user terminal 20, and will be restricted from speaking to user 1. When the restriction level is set to 1, other users will be unable to place user 1 in the level 0 and level 1 areas in the icon operation area 64 displayed on their own user terminal 20, and will be restricted from viewing images of user 1 and from speaking to user 1. When the restriction level is set to 2, other users will be unable to place user 1 in the level 0, level 1, and level 2 areas in the icon operation area 64 displayed on their own user terminal 20, and will be restricted from sharing all information, including viewing presence information about user 1.

The operation unit 220 of the user terminal 20-1 transmits the above-mentioned icon movement information and barrier setting information to the server 40 as spatial operation data. The control unit 420 of the server 40 collects barrier setting information from all user terminals 20 and outputs this as barrier setting data to the operation units 220 of all user terminals 20. The operation unit 220 of each user terminal 20 restricts other users' icon movement operations based on the barrier setting data input from the control unit 420 of the server 40.

Figures 5A and 5B are explanatory diagrams showing specific examples of spatial operations in the icon operation area 64. Figure 5A shows an example in which an icon located in the level 2 (position X) area has been moved to the level 0 (position Y) area. As will be described later, the voice data of user 1 is input from user terminal 20-1 to the user terminal 20 of the other user corresponding to the icon moved to the level 0 area, so that user 1 can speak to the other user corresponding to the icon moved to the level 0 area.

On the other hand, FIG. 5B shows an example in which a barrier area is set on an icon located in the level 2 (position X) area. As shown in FIG. 5B, when a barrier area is set on an icon located in the level 2 (position X) area, user 1 cannot move the icon to the level 0 area, and the icon is forcibly moved to the level 1 (position Z) area.

(S3: Determining media input parameters)
When spatial operation data relating to the movement of an icon is input from the operation unit 220 of the user terminal 20-1, the control unit 420 of the server 40 sets the sharing level of the other user corresponding to the icon to the level of the area including the moved icon. Then, the control unit 420 determines attribute data of the media data to be acquired from the user terminal 20 of the other user according to the sharing level set for the other user, and outputs the determined attribute data to the user terminal 20 of the other user as a media input parameter.

For example, if the sharing level of the other user is level 0 or level 1, the control unit 420 sets attribute data consisting of video data and presence data parameters as the media input parameters. If the sharing level of the other user is level 2, the control unit 420 sets attribute data consisting only of presence data parameters as the media input parameters. Note that, in one embodiment of the present invention, all media input parameters may include voice data parameters for voice calls.

(S4: Input of media data)
The media input unit 230 of each user terminal 20 generates only the data indicated by the media input parameters based on the media input parameters input from the server 40 and transmits the data to the server 40 .

For example, if the media input parameters include parameters for video data, the media input unit 230 transmits encoded data of the video data captured by the built-in camera of the user terminal 20 (in this specification, the encoded data may also be referred to simply as video data).

In addition, when the media input parameters include parameters for audio data, the media input unit 230 transmits encoded data of the audio data picked up by the built-in microphone of the user terminal 20 (in this specification, the encoded data may also be referred to simply as audio data).

If the media input parameters include a presence data parameter, the media input unit 230 transmits the user's presence data. As described above, the user presence data is useful for determining whether or not it is possible to contact the user.

In addition, each overhead camera 30 transmits encoded data of the overhead video data acquired by imaging (in this specification, the encoded data may also be referred to simply as overhead video data) to the server 40.

(S5: Playing back media data)
The control unit 420 of the server 40 selects media data corresponding to the sharing level of each icon indicated by the spatial operation data input from the user terminal 20-1 from among the media data collected from each user terminal 20 and the overhead video data collected from the overhead camera 30. Then, the control unit 420 outputs the selected media data to the user terminal 20.

The media playback unit 240 of the user terminal 20 plays/decodes the media data for each of the other users input from the server 40, and places the resulting screen elements within the corresponding icons in the icon operation area 64 of the virtual space screen. The screen elements for the other users can be said to be screen elements that show the status of the other users in a manner according to the sharing level set for the other users. Specific examples of the screen elements are described below.

Figures 6A to 6C are explanatory diagrams showing specific examples of screen elements arranged within an icon. Figure 6A shows the icon of another user whose sharing level is set to level 0. As shown in Figure 6A, the screen elements arranged on the icon of another user whose sharing level is set to level 0 include, in addition to the name of that other user, video data 71 of that other user and presence data 72 of that other user. The video data 71 is video data captured by the built-in camera of the other user's user terminal 20, and makes it possible to check the facial expression of the other user in detail.

Figure 6B shows the icon of another user whose sharing level is set to level 1. As shown in Figure 6B, the screen elements placed on the icon of another user whose sharing level is set to level 1 include, in addition to the name of that other user, overhead video data 73 including video of that other user, and presence data 72 of that other user. In the overhead video data 73, it is possible to check the general actions of the other user.

Figure 6C shows the icon of another user whose sharing level is set to level 2. As shown in Figure 6C, the screen elements placed on the icon of another user whose sharing level is set to level 2 include presence data 74 and do not include video data.

Since there is no information to share with other users whose sharing level is set to level 3, the icons of other users whose sharing level is set to level 3 are not displayed in the icon operation area 64.

When user 1 detects another user who has set the sharing level to level 0, the control unit 420 of the server 40 outputs media data including user 1's voice data to the user terminal 20 of the other user. This allows user 1 to speak to other users who have set the sharing level to level 0 by voice.

(S2: spatial operation detection (conference designation))
The process when a user moves the icon of another user within the virtual space screen has already been described. Here, the process when a user holds a spontaneous meeting on the virtual space screen will be described with reference to the drawings.

(a) Confirmation of intention to participate in a conference Fig. 7A is an explanatory diagram showing an example of icon movement for starting a conference. When user 1 holds an impromptu conference with other users (three people), user 1 moves the icons corresponding to the other users (icon 2, icon 3, icon 4) close to user 1's icon (icon 1) in the virtual space screen and places them in the level 0 area, as shown in Fig. 7A. At this point, user 1 can call out to the other three users. When the other users notice the call and respond to user 1 by voice or the like, they can confirm their intention to participate in the conference.

(b) Starting a conference Fig. 7B is an explanatory diagram showing the startup of the conference system. When user 1 designates the icons of the other three users on the virtual space screen and performs an operation to start the conference, the conference unit 250 of the user terminal 20 starts the conference system. As a result, the conference screen is displayed as shown in Fig. 7B, and the conference begins. If there are icons other than those of conference participants in the level 0 area of the virtual space screen, the conference unit 250 forcibly moves such icons to level 1.

(c) Confirmation of intention to participate in the meeting (additional)
FIG. 8A is an explanatory diagram showing an example of icon movement for adding a participant to a conference. User 1 can also invite other users who have not yet participated in the conference to the conference during the conference. For example, user 1 brings an icon (icon 5) corresponding to another adder who has not yet participated in the conference close to user 1's icon (icon 1) in the virtual space screen and places it in the level 0 area. At this point, user 1 can call out to the other four users including the adder. When the adder notices the call and responds to user 1 by voice or the like, the user can confirm his or her intention to participate in the conference.

(d) Adding a participant to a conference Fig. 8B is an explanatory diagram showing the addition of a participant to a conference. When the user 1 designates the icon of the person to be added in the virtual space screen and performs an operation to add a conference, the conference unit 250 adds the person to the conference. As a result, the conference screen with the person to be added is displayed as shown in Fig. 8B.

(e) Exiting from a conference, terminating a conference FIG. 9 is an explanatory diagram showing an example of a screen after the end of a conference. When user 1 designates the icon of user 1 (icon 1) in the virtual space screen and operates the end of the conference, the conference screen disappears as shown in FIG. 9, and the conference unit 250 ends the conference. At this point, the icons of the five people who participated in the conference are in the level 0 area, so the five people can continue their voice conversation. Note that when user 1 designates an icon corresponding to one of the users in the virtual space screen and operates the exit from the conference, the conference unit 250 causes the user corresponding to the icon to exit the conference. In this case, the conference unit 250 forcibly moves the icon of the person who has left to the level 1 area.

(S6: Control of the conference)
The control executed based on the above-mentioned conference-related screen operations will now be described in detail.

When starting a conference, the operation unit 220 of the user terminal 20 includes participant information indicating the users participating in the conference in spatial operation data and sends it to the server 40. Based on the participant information, the control unit 420 of the server 40 sends parameters required for holding the conference (such as the conference URL and ID) to each participant's user terminal 20 as conference parameters. The conference unit 250 of each user terminal 20 starts operation based on the conference parameters, and exchanges conference data (video, audio) with the other user terminals 20 via the control unit 420 of the server 40.

When a participant is added to a conference, the operation unit 220 of the user terminal 20 includes adder information indicating the adder in the spatial operation data and sends it to the server 40. Based on the adder information, the control unit 420 of the server 40 sends parameters required for adding a participant (such as the conference URL and ID) to the adder's user terminal 20 as conference parameters. The conference unit 250 of the adder's user terminal 20 starts operation based on the conference parameters, and exchanges conference data (video, audio) with other user terminals 20 via the control unit 420 of the server 40.

When leaving a conference, the operation unit 220 of the user terminal 20 includes exit information indicating the person leaving the conference in the spatial operation data and transmits it to the server 40. Based on the exit information, the control unit 420 of the server 40 transmits data notifying the user terminal 20 of the person leaving the conference as conference parameters. The conference unit 250 of the user terminal 20 of the person leaving the conference executes the exit from the conference based on the conference parameters received from the server 40.

When the conference ends, the operation unit 220 of the user terminal 20 includes conference end information in the spatial operation data and outputs it to the control unit. The control unit 420 of the server 40 transmits data notifying the end of the conference as conference parameters to each participant's user terminal 20 based on the conference end information. The conference unit 250 of each participant's user terminal 20 ends the conference operation based on the conference parameters received from the server 40.

(S7: Human Relationship Estimation)
The control unit 420 of the server 40 accumulates the spatial operation data input from each user terminal 20. Then, the control unit 420 outputs the accumulated spatial operation data to the human relationship estimation unit 430 periodically (for example, once a week).

The human relationship estimation unit 430 calculates two types of human relationship indicators based on the spatial operation data input from the control unit 420, and outputs these to the human relationship display terminal 50 as human relationship estimation data indicating the human relationships between multiple users. Below, a specific example of a human relationship estimation method is described with reference to Figures 10 and 11.

(a) Human Relationships Based on Virtual Space Distance As described above, the space operation data includes information indicating operations in which each user set a sharing level for another user. The human relationship estimation unit 430 can calculate the average sharing level between two people in a predetermined period of time in the past (e.g., one week) and use this as an index of human relationships. For example, the human relationship estimation unit 430 calculates the average sharing level by setting sharing level 0 as "0", sharing level 1 as "1/3", sharing level 2 as "2/3", and sharing level 3 as "3/3".

Figure 10 is an explanatory diagram showing an example of the estimated results of human relationships between four users (user 1, user 2, user 3, user 4). For example, a numerical value of "0.2" from user 1 to user 2 indicates that the average index related to the sharing level set by user 1 for user 2 was "0.2" (halfway between sharing level 0 and sharing level 1). In the example shown in this figure, it can be inferred that user 1 and user 2 have a close relationship, user 1 and user 4 have a loose relationship, user 1 and user 3 have an unbalanced relationship, and user 4 does not interact much with other users.

(b) Human Relationships Based on Meeting Time As described above, the spatial operation data includes operation information related to the user's participation in a meeting. The human relationship estimation unit 430 can calculate the average of the meeting time between two people in a predetermined period of time in the past (e.g., one week) and use this as an index of human relationships. For example, the human relationship estimation unit 430 calculates the human relationship index by setting a total of 8 hours of meeting time in the past week as "0" and a total of 0 hours of meeting time as "1."

Figure 11 is an explanatory diagram showing an example of the estimated results of human relationships between four users (user 1, user 2, user 3, and user 4). For example, a numerical value of "0.0" for the combination of user 1 and user 2 indicates that user 1 and user 2 hold frequent meetings, and a numerical value of a combination including user 4 that is generally close to "1.0" indicates that user 4 does not hold many meetings with other users.

(S8: Display of human relationships)
When the human relationship estimation data is input from the human relationship estimation unit 430, the human relationship display terminal 50 displays the human relationship estimation data or an analysis result of the human relationship estimation data.

(S9: End determination)
When the user 1 instructs the operation unit 220 of the user terminal 20-1 to end the display of the virtual space screen, the user terminal 20-1 ends the operation related to this system. If the user 1 does not instruct the user to end the display, the process from S2 is repeated.

<4. Effects>
According to the embodiment of the present invention described above, various operational effects can be obtained. For example, in one embodiment of the present invention, the user terminal 20 of the user displays a virtual space screen on which an icon corresponding to the user himself/herself and icons corresponding to other users are arranged, and a sharing level for the other users is set according to the distance between the icon corresponding to the user himself/herself and the icon corresponding to the other users. When the distance between the icon corresponding to the user himself/herself and the icon corresponding to the other users is short and the sharing level is set to level 0, the user can check the state of the other users by viewing the video data of the other users and start a conversation by calling out to them. When the distance between the icon corresponding to the user himself/herself and the icon corresponding to the other users is long and the sharing level is set to level 2, the user checks the state of the other users based on the presence data. In order for the user to check the state of the other users or call out to them, the user only needs to pull the icon corresponding to the other users toward the icon of the user himself/herself.

According to one embodiment of the present invention, a user can obtain consent to hold a conference with other users whose icons are located near the user's own icon using video and audio data, and can smoothly start an unscheduled, spontaneous conference. In this way, the status of other users can be visualized at a sharing level according to the relationship between the user and the other users, and spontaneous communication can be started smoothly. This makes it possible to prevent situations in which the other user does not notice the conference notification when holding an impromptu conference, or hesitates to hold a conference because they do not know the status of the other users.

In addition, when a user wants to avoid communication with other users, such as when he or she wants to concentrate on work, the user can use a barrier setting operation to prevent the user's icon from being placed in the level 0 area of the other users' user terminals 20. This makes it possible to restrict other users from viewing the user's video data and from speaking to the user.

According to one embodiment of the present invention, it is possible to estimate the psychological distance and interpersonal relationships between users based on the distance between the icons of each user on the virtual space screen, the amount of time the conference system is used, and the like. This estimation method has the advantage of being able to estimate interpersonal relationships with a higher degree of accuracy than other methods that require estimation of the content of statements and the person to whom the statements are made.

5. Modifications
An embodiment of the present invention has been described above. In the following, some modified examples of the above-described embodiment will be described. Note that each modified example described below may be applied alone to the above-described embodiment, or may be applied in combination to the above-described embodiment. Also, each modified example may be applied in place of the configuration of the above-described embodiment, or may be applied in addition to the configuration of the above-described embodiment.

For example, in the above, an example was described in which the sharing levels in the virtual space screen are level 0, level 1, level 2, and level 3. Also, an example was described in which the shared content at level 0 is viewing the other party's video data, viewing the other party's presence data, and a voice call, the shared content at level 1 is viewing the other party's video data and viewing the other party's presence data, the shared content at level 2 is viewing the other party's presence data, and there is no shared content at level 3. However, the number of sharing levels and the shared content at each level are not limited to the above example. For example, even if the number of sharing levels increases or decreases, it is possible to obtain the same effect as in the above-mentioned embodiment. Also, the difference in the shared content at each level may be the resolution of the other party's video. Also, the shared content for communication with the other party may include text messages, illustrations, symbols, etc. that express intentions and feelings other than video data and audio data. In this case, it is also possible to obtain the same effect as in the above-mentioned embodiment.

In the above, a case where each user sets a sharing level for each of the other users has been described. However, if two arbitrary users set different sharing levels for each other, the level of information shared and the way of communication will be different. Therefore, in order to make the level of information sharing between users equal, when two arbitrary users set different sharing levels for each other, the control unit 420 of the server 40 may set a new sharing level based on these two sharing levels. For example, when a first user sets level 0 for a second user and the second user sets level 1 for the first user, the control unit 420 may unify the sharing levels of both users to level 0 or level 1. With this configuration, the first user and the second user can communicate in both directions at an equal sharing level.

In the above, an example was described in which screen elements showing the status of other users are displayed within an icon arranged on the virtual space screen. In another configuration, the screen elements showing the status of other users may be displayed in any area of the display screen, not within the icon of the other user. For example, the display screen may include a virtual space area and a communication area for each sharing level, and information (screen elements) of other users may be displayed in the communication area for the corresponding sharing level among the communication areas for each sharing level in response to the movement of the icon in the virtual space area. Even with such a configuration, it is possible to obtain the same effects as in the above-mentioned embodiment.

Although an example in which icons corresponding to other users are arranged on a virtual space screen has been described above, it is possible to obtain the same effect as in the above-mentioned embodiment without using a virtual space screen. For example, icons corresponding to other users may be arranged on the display screen of the user terminal 20, and a sharing level may be set for each icon. In addition, the display screen of the user terminal 20 may be divided into areas for each sharing level, and icons corresponding to other users may be arranged in each area to set the sharing level.

In the above, an example was described in which the psychological distance and interpersonal relationships between users are estimated based on the distance between the icons of each user on the virtual space screen, the usage time of the conference system, and the like. Furthermore, when estimating interpersonal relationships, the history of barrier setting operations within the virtual space screen may be used. That is, when a user performs a barrier setting operation for a specific other user, it can be considered that the psychological distance from the user to the specific other user is long. In such a case, the control unit 420 of the server 40 may increase the numerical value shown in FIG. 10 from the user to the specific other user. As a result, it is expected that the accuracy of estimating interpersonal relationships will be improved.

The control unit 420 of the server 40 may also select at least some of the other users for whom a certain user has set the sharing level to level 3 (not displayed on the virtual space screen) and level 2 (only presence information is displayed), and temporarily raise the sharing level for the selected other users by one level. In this case, icons corresponding to other users for whom level 3 was set are placed in the level 2 area, and icons corresponding to other users for whom level 2 was set are moved to the level 1 area. This provides an opportunity for users to review the sharing level for other users. Users can then review the placement of icons in the virtual space screen, thereby improving the accuracy of the estimation of human relationships.

6. Hardware Configuration
The above describes the embodiments of the present invention. The above-described information processing, such as displaying the virtual space screen and controlling the conference, is realized by cooperation between software and the hardware of the user terminal 20 described below.

Figure 12 is a block diagram showing the hardware configuration of the user terminal 20. The user terminal 20 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and a host bus 204. The user terminal 20 also includes a bridge 205, an external bus 206, an interface 207, an input device 208, a display device 209, an audio output device 210, a storage device (HDD) 211, a drive 212, and a network interface 215.

The CPU 201 functions as an arithmetic processing device and control device, and controls the overall operation of the user terminal 20 in accordance with various programs. The CPU 201 may also be a microprocessor. The ROM 202 stores programs and arithmetic parameters used by the CPU 201. The RAM 203 temporarily stores programs used in the execution of the CPU 201 and parameters that change appropriately during the execution. These are interconnected by a host bus 204 consisting of a CPU bus and the like. The functions of the operation unit 220, media input unit 230, media playback unit 240, and conference unit 250 described above can be realized by the cooperation of the CPU 201, ROM 202, and RAM 203 with software.

The host bus 204 is connected to an external bus 206, such as a PCI (Peripheral Component Interconnect/Interface) bus, via a bridge 205. Note that the host bus 204, bridge 205, and external bus 206 do not necessarily need to be configured separately, and these functions may be implemented on a single bus.

The input device 208 is composed of input means for the user to input information, such as a mouse, keyboard, touch panel, button, microphone, sensor, switch, and lever, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 201. By operating the input device 208, the user of the user terminal 20 can input various data to the user terminal 20 and instruct processing operations.

The display device 209 includes display devices such as a liquid crystal display (LCD) device, a projector device, an OLED (Organic Light Emitting Diode) device, and a lamp. The audio output device 210 includes audio output devices such as a speaker and headphones.

The storage device 211 is a data storage device configured as an example of a storage unit of the user terminal 20 according to this embodiment. The storage device 211 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, and a deleting device that deletes data recorded on the storage medium. The storage device 211 is configured, for example, with a HDD (Hard Disk Drive) or SSD (Solid Storage Drive), or a memory with equivalent functions. This storage device 211 drives the storage and stores the programs executed by the CPU 201 and various data.

The drive 212 is a reader/writer for storage media, and is built into the user terminal 20 or attached externally. The drive 212 reads information recorded on a removable storage medium 24, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 203 or the storage device 211. The drive 212 can also write information to the removable storage medium 24.

The network interface 215 is, for example, a communication interface configured with a communication device for connecting to the network 12. The network interface 215 may be a wireless LAN (Local Area Network) compatible communication device or a wired communication device that performs wired communication.

The hardware configuration of the user terminal 20 described above can also be applied to the server 40.

＜７． Supplementary Information＞
Although the preferred embodiment of the present invention has been described in detail above with reference to the accompanying drawings, the present invention is not limited to such an example. It is clear that a person having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present invention.

For example, each step in the processing of the user terminal 20 and the server 40 in this specification does not necessarily have to be processed chronologically in the order described in the flowchart. For example, each step in the processing of the user terminal 20 and the server 40 may be processed in an order different from the order described in the flowchart, or may be processed in parallel.

It is also possible to create a computer program for causing the hardware, such as the CPU, ROM, and RAM, built into the user terminal 20 and the server 40 to perform functions equivalent to those of the respective components of the user terminal 20 and the server 40 described above. A storage medium on which the computer program is stored is also provided.

20 User terminal 220 Operation unit 230 Media input unit 240 Media playback unit 250 Conference 30 Overhead camera 40 Server 420 Control unit 430 Human relationship estimation unit 50 Human relationship display terminal

Claims (14)

A communication system having a first user terminal used by a first user and one or more second user terminals used by one or more second users,
The first user terminal comprises:
an operation unit that detects an operation for setting a sharing level for each of the one or more second users;
a display control unit that generates, for each of the second users, a display screen on which screen elements indicating the status of the second users are arranged in a manner corresponding to a sharing level set for the second users; and
A communication system comprising: the display screen includes a movable display corresponding to each of the second users;
the operation unit detects an operation of moving the movable display as an operation of setting the sharing level,
The communication system according to claim 1 , wherein the sharing level set for the second user changes in response to movement of the movable display. The communication system according to claim 2, wherein on the display screen, the screen element for the second user is arranged within a movable display corresponding to the second user. The communication system according to claim 2, wherein on the display screen, the screen element for the second user is arranged in a screen area different from within the movable display corresponding to the second user. When a first sharing level is set for the second user, an image of the second user is arranged on the display screen as a screen element showing the status of the second user, and the first user can speak to the second user;
A communication system according to any one of claims 1 to 4, wherein when a second sharing level is set for the second user, text indicating the status of the second user is placed on the display screen as a screen element indicating the status of the second user. The communication system according to any one of claims 1 to 5, further comprising a sharing level setting unit that matches the sharing level set by the first user to a second user when the sharing level set by the second user to the first user differs from the sharing level set by the second user to the first user. The communication system according to any one of claims 1 to 5, further comprising a sharing level setting unit that limits a sharing level that each of the second users can set for the first user when a predetermined operation is detected by the operation unit of the first user terminal. The communication system according to any one of claims 1 to 7, wherein the first user terminal further comprises a conference unit that controls the start of a conference system in which the first user and the second user participate when a sharing level is set at which the first user and the second user can communicate with each other and a conference start operation is detected by the operation unit. The communication system according to any one of claims 1 to 8, further comprising a human relationship estimation unit that estimates human relationships between the first user and a plurality of users among the one or more second users, based on information indicating operations on the first user terminal and each of the one or more second user terminals. The communication system according to claim 9, wherein the human relationship estimation unit estimates the human relationships between the plurality of users based on at least one of the following: a history of the sharing levels set by each of the plurality of users for the other users, a certain user designating a specific other user and restricting the sharing level that the other user can set for the user, or a combination of users who participated in an implemented conference system. The communication system according to any one of claims 1 to 5, further comprising a sharing level setting unit that changes the sharing level set by the first user to the second user to a higher sharing level. A user terminal that operates as a first user terminal used by a first user,
an operation unit that detects an operation for setting a sharing level for each of one or more second users;
a display control unit that generates, for each of the second users, a display screen on which screen elements indicating the status of the second users are arranged in a manner corresponding to a sharing level set for the second users; and
A user terminal comprising: Computer,
A user terminal that operates as a first user terminal used by a first user,
an operation unit that detects an operation for setting a sharing level for each of one or more second users;
a display control unit that generates, for each of the second users, a display screen on which screen elements indicating the status of the second users are arranged in a manner corresponding to a sharing level set for the second users; and
A program for functioning as a user terminal. 1. A method performed by a user terminal operating as a first user terminal used by a first user, comprising:
detecting an operation to set a sharing level for each of one or more second users;
generating, for each of the second users, a display screen on which screen elements indicating the status of the second users are arranged in a manner corresponding to a sharing level set for the second users;
A method comprising:

Images