JP2018036844A - Information processing system, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow each user to know a transition of interest of another user among a plurality of display objects shared among users.SOLUTION: An information processing system has a plurality of terminal devices, and each of the plurality of terminal devices displays a plurality of common display objects. The information processing system has a display control part for generating a display screen including the plurality of display objects, and the display screen includes a display showing a transition of a display object in which the user of each of the plurality of terminal devices is interested among the plurality of display objects.SELECTED DRAWING: Figure 8

Description

  The present invention relates to an information processing system, an information processing method, and a program.

  In the near future, a remote conference system for realizing a remote conference in which a plurality of separated users participates has become widespread. With regard to the remote conference system, a device for obtaining a sense of reality such as a face-to-face conference has been studied. For example, Patent Literature 1 discloses a technique for each user to know who is requesting eye contact from himself / herself in a remote conference.

JP-A-4-351188

  In face-to-face meetings, shared materials are distributed to each participant, and communication by each participant is promoted using the shared materials. In the face-to-face meeting, each participant can know the transition of the shared material that the other participant is interested in based on the line of sight of the other participant. For this reason, each participant can advance communication smoothly by taking into account the transition of other participants' interests. However, in the conventional remote conference system, the device regarding the display object displayed as the shared material for each user is insufficient, and it has been difficult to realize smooth communication in consideration of each user's interest.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to make each user interested in other users among a plurality of display objects shared among the users. It is an object of the present invention to provide a new and improved information processing system, information processing method, and program capable of knowing changes.

  In order to solve the above-described problem, according to an aspect of the present invention, there is provided an information processing system having a plurality of terminal devices and displaying a plurality of display objects common to each of the plurality of terminal devices, A display control unit configured to generate a display screen including the plurality of display objects, wherein the display screen is a transition of a display object in which each user of the plurality of terminal devices is interested in the plurality of display objects; An information processing system is provided that includes a display indicating.

  The display may include, for each user, a user display arranged in association with a position corresponding to a transition of a display object in which the user is interested.

  The display may include a linked display that connects the user display and a display object in which the user is interested.

  The information processing system is configured to calculate a coordinate position expressed by coordinate values on a coordinate axis corresponding to each display object from a time when the user is interested in each display object among the plurality of display objects. The display control unit may arrange the user display at a position corresponding to the coordinate position on the display screen.

  The information processing system includes: an abstract interest vector calculation unit that calculates an abstract interest vector that connects the coordinate position and a predetermined value on a coordinate axis corresponding to a display object that the user is currently interested in; and the abstract interest vector Is further converted to a projective interest vector having a number of dimensions that can be expressed on the display screen, and the display control unit displays a display object in which the user is currently interested in the projected interest vector. You may arrange | position the said user display in the position of the other end when it is set as one end.

  The display screen is used for a remote conference by users of the plurality of terminal devices, and the position calculation unit is obtained by dividing a time when the user is interested in a display object by a scheduled time of the remote conference. A coordinate value on the coordinate axis corresponding to the display object may be calculated by multiplying the value by the predetermined value on the coordinate axis corresponding to the display object.

  The information processing system may further include an adjustment unit that adjusts a display mode of the user display for each user.

  The display screen may be used for a remote conference by users of the plurality of terminal devices, and the adjustment unit may perform the adjustment according to the degree of interest of each user in the remote conference.

  The adjustment unit may perform the adjustment so that a user having a higher degree of interest in the remote conference has a higher visibility of the user display of the user.

  The adjustment unit may perform the adjustment according to a length of time that each user of the plurality of terminal devices has been interested in the display screen as a degree of interest of each user in the remote conference.

  The adjustment unit, as the degree of interest of each user in the remote conference, the time when face recognition succeeded for each user from a captured image in a range in which the display screen displayed on each of the plurality of terminal devices can be visually recognized The adjustment may be performed according to the length.

  The adjustment unit may perform the adjustment according to the utterance amount of each user as the degree of interest of each user in the remote conference.

  The display screen is used for a remote conference by users of the plurality of terminal devices, and the information processing system displays the coordinate position of each user calculated by the location calculation unit for each user to the remote conference. You may further provide the adjustment part adjusted according to the degree of interest.

  The display control unit may change the arrangement or size of the plurality of display objects on the display screen based on an operation from the user.

  In order to solve the above-mentioned problem, according to another aspect of the present invention, each of the plurality of terminal devices includes displaying a display screen including a plurality of common display objects, and the display screen includes: There is provided an information processing method including a display showing a transition of a display object in which each user of a plurality of terminal devices is interested in the plurality of display objects.

  In order to solve the above-described problem, according to another aspect of the present invention, there is provided an information processing system having a plurality of terminal devices and displaying a plurality of display objects common to each of the plurality of terminal devices. A program for realizing the computer, the computer including the plurality of display objects, and a display indicating a transition of the display objects in which each user of the plurality of terminal devices is interested in the plurality of display objects. A program is provided to function as a display control unit that generates a display screen.

  As described above, according to the present invention, it is possible for each user to know a change in interest of other users among a plurality of display objects shared among users.

It is explanatory drawing which shows the structure of the remote conference system by embodiment of this invention. It is explanatory drawing which shows the hardware constitutions of a terminal device. It is explanatory drawing which shows the hardware constitutions of a server. It is explanatory drawing which shows the specific example of the meeting screen displayed by this embodiment. It is explanatory drawing which shows the structure of the server by the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the interest transition history memorize | stored in an interest transition history memory | storage part. It is explanatory drawing which shows the information space by which an abstract interest vector is expressed. It is explanatory drawing which shows the structure of the terminal device by 1st Embodiment. It is explanatory drawing which shows the specific example of the meeting screen displayed on the display part of the terminal device which user A uses. It is explanatory drawing which shows the other example of the conference screen displayed on the display part of the terminal device which user A uses. It is explanatory drawing which shows the other example of the conference screen displayed on the display part of the terminal device which user A uses. It is explanatory drawing which shows the specific example of the meeting screen displayed on the display part of the terminal device which the user B uses. It is explanatory drawing which shows the variation of the layout of a meeting screen. It is explanatory drawing which shows the variation of the layout of a meeting screen. It is explanatory drawing which shows the specific example of the meeting screen displayed on the display part of the terminal device which the user B uses. It is explanatory drawing which shows operation | movement of the remote conference system by 1st Embodiment. It is explanatory drawing which shows the structure of the terminal device by 2nd Embodiment. It is explanatory drawing which shows the structure of the server by 2nd Embodiment. The meeting screen displayed on the terminal device which user B uses before presence is adjusted is shown. The meeting screen displayed on the terminal device which user B uses after presence is adjusted is shown. It is explanatory drawing which shows the information space of the abstract interest vector corresponding to a meeting screen.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In the present specification and drawings, a plurality of components having substantially the same functional configuration may be distinguished by adding different alphabets after the same reference numeral. For example, a plurality of configurations having substantially the same functional configuration or logical significance are distinguished as necessary, such as the terminal devices 200A, 200B, and 200C. However, when it is not necessary to particularly distinguish each of a plurality of constituent elements having substantially the same functional configuration, only the same reference numeral is given to each of the plurality of constituent elements. For example, when it is not necessary to distinguish the terminal devices 200A, 200B, and 200C, each terminal device is simply referred to as the terminal device 200.

<0. Overview of the remote conference system>
Embodiments described herein relate generally to an information processing system in which a plurality of users share the same display object. For example, the information processing system can remotely display a display object corresponding to each of a plurality of conference materials while sharing the display object. It can be realized as a remote conference system for conducting a conference. Hereinafter, an overview of a remote conference system according to an embodiment of the present invention will be described with reference to FIG.

  FIG. 1 is an explanatory diagram showing a configuration of a remote conference system according to an embodiment of the present invention. As shown in FIG. 1, the remote conference system according to the embodiment of the present invention includes a server 100 and a plurality of terminal devices 200. The server 100 is connected to each terminal device 200 via a LAN (Local Area Network).

(Terminal device)
Each terminal device 200 is located at a different base and is used by different users. For example, in the example illustrated in FIG. 1, the terminal device 200A is located at the base bA and used by the user A (uA), the terminal device 200B is located at the base bB and used by the user B, and the terminal device 200C is located at the base b The terminal device 200D is located at the base bD and used by the user D. However, although FIG. 1 illustrates an example in which one terminal device 200 and one user exist at one site, a plurality of terminal devices 200 and a plurality of users may exist at one site.

  Here, the hardware configuration of the terminal device 200 will be described with reference to FIG. FIG. 2 is an explanatory diagram illustrating a hardware configuration of the terminal device 200.

  As shown in FIG. 2, the terminal device 200 according to the present embodiment includes a CPU (Central Processing Unit) 801, a ROM (Read Only Memory) 803, a RAM (Random Access Memory) 805, a bus 807, a storage device 809, and a communication interface. 811, a camera 813, a microphone 815, a speaker 817, and an input device 819.

  The CPU 801 executes various processes in the terminal device 200. The ROM 803 stores a program and data for causing the CPU 801 to execute processing in the terminal device 200. The RAM 805 temporarily stores programs and data when the CPU 801 executes processing.

  A bus 807 connects the CPU 801, the ROM 803, and the RAM to each other. A storage device 809, a communication interface 811, a camera 813, a microphone 815, a speaker 817, and a touch panel 820 are further connected to the bus 807. The bus 807 includes, for example, a plurality of types of buses. As an example, the bus 807 includes a high-speed bus that connects the CPU 801, the ROM 803, and the RAM 805, and one or more other buses that are slower than the high-speed bus.

  The storage device 809 stores data to be temporarily or permanently stored in the terminal device 200. The storage device 809 may be, for example, a magnetic storage device such as a hard disk, or an EEPROM (Electrically Erasable and Programmable Read Only Memory), a flash memory, an MRAM (Magnetoresistive Random Access Memory). Further, non-volatile memories such as FeRAM (Ferroelectric Random Access Memory) and PRAM (Phase change Random Access Memory) may be used.

  The communication interface 811 is a communication unit included in the terminal device 200, and communicates with an external device via a network (or directly). The communication interface 811 may be an interface for wireless communication. In this case, for example, a communication antenna, an RF circuit, and other communication processing circuits may be included. The communication interface 811 may be an interface for wired communication. In this case, for example, a LAN terminal, a transmission circuit, and other communication processing circuits may be included.

  The camera 813 images a subject. The camera 813 includes, for example, an optical system, an image sensor, and an image processing circuit. The microphone 815 collects ambient sounds. The microphone 815 converts ambient sound into an electric signal, and converts the electric signal into digital data. The speaker 817 outputs sound. The speaker 817 converts digital data into an electric signal, and converts the electric signal into sound.

  The input device 819 is a touch panel, a mouse, a line-of-sight detection device, or the like. The display device 823 displays an output image (that is, a display screen) from the terminal device 200. The display device 823 can be realized by using, for example, a liquid crystal, an organic EL (Organic Light-Emitting Diode: OLED), a CRT (Cathode Ray Tube), or the like.

(server)
The server 100 has a function of forming a remote conference virtual space and providing a user of each terminal device 200 with a remote conference place. Specifically, the server 100 collects user images and sounds from each terminal device 200 and provides the collected user images and sounds to the terminal devices 200 of all users participating in the remote conference. Thereby, all users participating in the remote conference can share the images and sounds of each user as in the face-to-face conference.

  Here, the hardware configuration of the server 100 will be described with reference to FIG. FIG. 3 is an explanatory diagram showing the hardware configuration of the server 100.

  As illustrated in FIG. 3, the server 100 according to the present embodiment includes a CPU 901, a ROM 903, a RAM 905, a bus 907, a storage device 909, and a communication interface 911.

  The CPU 901 executes various processes in the server 100. The ROM 903 stores a program and data for causing the CPU 901 to execute processing in the server 100. The RAM 905 temporarily stores a program and data when the CPU 901 executes the process.

  A bus 907 connects the CPU 901, the ROM 903, and the RAM to each other. A storage device 909 and a communication interface 911 are further connected to the bus 907. The bus 907 includes, for example, a plurality of types of buses. As an example, the bus 907 includes a high-speed bus that connects the CPU 901, the ROM 903, and the RAM 905, and one or more other buses that are slower than the high-speed bus.

  The storage device 909 stores data to be temporarily or permanently stored in the server 100. The storage device 909 may be a magnetic storage device such as a hard disk, or may be a non-volatile memory such as EEPROM, flash memory, MRAM, FeRAM, and PRAM.

  The communication interface 911 is a communication unit included in the server 100 and communicates with an external device via a network (or directly). The communication interface 911 may be an interface for wireless communication. In this case, for example, a communication antenna, an RF circuit, and other communication processing circuits may be included. The communication interface 911 may be an interface for wired communication. In this case, for example, the communication interface 911 may include a LAN terminal, a transmission circuit, and other communication processing circuits.

(Overview of operation)
In the present embodiment, by the cooperation of the server 100 and the terminal device 200 described above, a conference screen (display screen) including display objects corresponding to each of a plurality of materials used for the remote conference is provided to each user. In addition, there are many cases where multiple materials are used in face-to-face meetings in the real world, so it is considered that there is a demand for sharing multiple materials in remote conferences, and when multiple materials are used Since the interest of each user is dispersed, the necessity of supporting each user's interest status to be transmitted to each other is increased, and so on, in this embodiment, a device in a remote conference system using a plurality of materials is proposed.

  FIG. 4 is an explanatory diagram illustrating a specific example of a conference screen displayed according to the present embodiment. More specifically, FIG. 4 shows a conference screen 300A displayed on the terminal device 200A used by the user A.

  As shown in FIG. 4, the conference screen 300A includes display objects 320E, 320F, and 320G corresponding to the material E, the material F, and the material G used for the remote conference. Furthermore, the conference screen 300 </ b> A according to the present embodiment includes a display showing a transition of interest in each material of each user (for example, a time during which the user has been interested in each material). Specifically, as shown in FIG. 4, the conference screen 300 </ b> A includes a user display 330 corresponding to each user, and a connected display 350 (projected interest vector described later) that connects each user display 330 and one of the display objects 320. )including.

  The user display 330 corresponding to one user is arranged at a position corresponding to the transition of the material that the user is interested in among the plurality of materials (a plurality of display objects 320). As a result, as shown in FIG. 4, the user displays 330A, 330B and 330C corresponding to each user can be arranged at different positions.

  Further, the connection display 350 is arranged so as to connect a user display 330 corresponding to one user and a display object 320 corresponding to a material that the user is currently interested in. In the example shown in FIG. 4, a connection display that connects the user display 330B corresponding to the user B and the display object 320E corresponding to the material E is arranged, and the connection display indicates that the user B is currently interested in the material E. Indicates that

  Note that the user A is currently interested in the material E as indicated by the linked display connecting the user display 330A and the display object 320E corresponding to the material E. In this case, on the conference screen 300A displayed on the terminal device 200A used by the user A, as shown in FIG. 4, the display object 320E corresponding to the material E is displayed larger than the other display objects.

  Based on such a conference screen 300, each user can know the change of interest of other users among the plurality of display objects 320 and the object of current interest. Accordingly, each user can realize smooth communication even in the remote conference by taking into account the interests of other users in the same manner as the face-to-face conference. Hereinafter, the configuration and operation of each embodiment of the present invention will be sequentially described in detail.

<1. First Embodiment>
(Server configuration)
FIG. 5 is an explanatory diagram showing the configuration of the server 100 according to the first embodiment of the present invention. As illustrated in FIG. 5, the server 100 includes a communication unit 110, a storage unit 120, and a control unit 130.

  The communication unit 110 communicates with other devices. For example, the communication unit 110 is directly connected to the LAN 20 and communicates with the terminal device 200 at each base. In addition, when there is a sensor device such as a network camera or a microphone that captures the space of each site at the site where the terminal device 200 exists, the communication unit 110 communicates with the sensor device and acquires sensor data. The acquired sensor data may be transmitted to the terminal device 200 at another base. Note that the communication unit 110 can be implemented by the communication interface 911, for example.

  The storage unit 120 stores a program and data for the operation of the server 100. The storage unit 120 includes an interest transition history storage unit 121. The interest transition history storage unit 121 stores an interest transition history indicating a transition of each user's interest among a plurality of materials (display objects). Here, with reference to FIG. 6, the interest transition history memorize | stored in the interest transition history memory | storage part 121 is demonstrated more concretely.

  FIG. 6 is an explanatory diagram illustrating an example of the interest transition history stored in the interest transition history storage unit 121. As illustrated in FIG. 6, the interest transition history stores, for each user, the material selected by the user and the time of the selection in association with each other. For example, FIG. 6 shows an interest transition history when the user A selects the material G at 10:00:05 and the user A selects the material F at 10:12:32. It can be treated that the user A has selected the material G, that is, is interested in the material G, for 12 minutes and 27 seconds from the time when the user A selects the material G to the time when the material F is selected. Note that the interest transition history storage unit 121 may store the transition of the data of the abstract interest vector for each user calculated by the control unit 130 described later in association with the time information.

  The control unit 130 controls the overall operation of the server 100. For example, the control unit 130 receives multimedia data such as human video data, shared material data, audio data, and text data shared between users who are participants through the terminal device 200 in a remote conference. In response to a request from 200, the data is acquired from another terminal device 200, and data is provided to each terminal device 200. In addition, the control unit 130 acquires data related to the process of selecting an object of interest from the terminal device 200, obtains an abstract interest vector based on the data, and provides the abstract interest vector data to the terminal device 200. Further, the control unit 130 may cause the storage unit 120 to store the above-described various multimedia data, data related to the process of selecting an object of interest, and abstract interest vector data. Note that the control unit 130 appropriately updates various information used in the remote conference system. The control unit 130 may update the information automatically or in response to a manual instruction. Such a function of the control unit 130 can be realized by the CPU 901, the ROM 903, and the RAM 905, for example.

  As an element function for realizing the control unit 130 described above, the control unit 130 includes a person video control unit 131, a shared material control unit 133, another media information control unit 135, and an abstract interest vector calculation unit 137 as shown in FIG. Have

  The person video control unit 131 acquires data captured by the imaging unit 230 of the terminal device 200 of the user participating in the remote conference (for example, data of a human video of the user using the terminal device 200), Manage and provide to other terminal devices 200. When there is a network camera that captures the space of each base, the person video control unit 131 uses the data captured by the network camera as data of the user's human video instead of the data captured by the imaging unit 230. And may be provided to another terminal device 200.

  The shared material control unit 133 acquires, manages, and provides the data of the material shared in the remote conference to the terminal device 200. The material shared in the remote conference typically includes a slide material, but the material shared in the remote conference is not limited to the slide material. For example, information having a form that can be visually recognized by a human, such as text, an image, a video, and a multi-view video may be treated as the material.

  The other media information control unit 135 acquires and manages various data handled by the remote conference system other than the person video data handled by the person video control unit 131 and the data of the material handled by the shared material control unit 133. Provided to the terminal device 200. The data may be multimedia data such as voice data and text data shared between users who are participants through the terminal device 200 in a remote conference, for example.

  The abstract interest vector calculation unit 137 obtains data related to the process of selecting an object of interest from each terminal device 200 of each user participating in the remote conference, calculates an abstract interest vector from the data, and calculates the abstract interest vector data. Is provided to the terminal device 200. In the following, the abstract interest vector will be described.

/ Outline of abstract interest vector /
The abstract interest vector is data reflecting how much time each user is interested in each material and what material each user is currently interested in. The abstract interest vector is expressed in an information space having the number of dimensions of the number of materials shared in the remote conference.

  In the information space, a coordinate axis corresponding to each material shared in the teleconference is set. For example, when three materials E, F, and G are shared, each of the materials E, F, and G corresponds. A three-dimensional information space consisting of coordinate axes is defined.

  The abstract interest vector calculation unit 137 is a vector of a basic prescribed length buMV parallel to the positive direction of the coordinate axis corresponding to the material in the information space every time a certain time elapses while a certain user selects the same material. Is accumulated. Here, assuming that the maximum set value of the vector parallel to each coordinate axis is maxMV, the above-mentioned fixed time is buT, and the scheduled execution time of the remote conference is totalT, the above-mentioned basic prescribed length buMV is calculated as buMV = maxMV * buT / totalT. Is done.

  Note that buT and maxMV may be set in advance as predetermined values by the administrator of the remote conference system. For example, buT is 1 second (adding buMV once per second), and maxMV is 7200. It may be stored in the storage unit 120 as a specified value. Since totalT, which is the scheduled execution time of the remote conference, is considered to be different for each conference, the value of totalT can be set for each conference by the user of the remote conference system.

  Then, the abstract interest vector calculation unit 137 calculates the coordinate position obtained by averaging the buMV accumulated from the conference start time to a certain point in time, and the maximum setting on the coordinate axis corresponding to the material the user is currently interested in. A vector connecting the value maxMV is calculated for each user as an abstract interest vector.

/ Specific examples of abstract interest vectors /
The outline of the abstract interest vector has been described above. Next, the abstract interest vector will be described more specifically with reference to FIG.

  FIG. 7 is an explanatory diagram showing an information space 400 in which an abstract interest vector is expressed. The information space 400 has coordinate axes corresponding to each of the three materials E, F, and G. In this information space, buT is 1 second, user A is interested in material E for 200 seconds, user A is interested in material F for 400 seconds, and material G is user When A is interested in 100 seconds, the abstract interest vector calculation unit 137 accumulates vectors every second, so that the component on the coordinate axis corresponding to the material E corresponds to 200 and the material F corresponds. A vector whose component on the coordinate axis is 400 and whose component on the coordinate axis corresponding to the material G is 100 is obtained.

  When the user is interested in the material E at the time point 700 seconds after the start of the conference, the abstract interest vector calculation unit 137 adds the coordinate position 430A obtained by averaging each component of the vector, and the material E A vector 450A connecting the maximum set value maxMV410E on the coordinate axis corresponding to is calculated as an abstract interest vector 450A.

  Note that the tip of the abstract interest vector 450A changes according to the change of the user's interest, and the other end of the abstract interest vector 450A (the coordinate position 430A in the example shown in FIG. 7) changes with time. The direction and size of the interest vector 450A changes at least every time buT (1 second in this example) elapses.

  FIG. 7 shows the abstract interest vector 450A of the user A. The abstract interest vector calculation unit 137 calculates the abstract interest vector of the user B and the abstract interest vector of the user C. There may be an abstract interest vector for user B and an abstract interest vector for user C.

  Further, in the above description, an example in which the value of totalT is set by the user for each conference has been described. For example, the totalT may be acquired as data from the application in cooperation with the scheduler application to which the conference schedule is input, or statistical information (for example, a predetermined time until the time of use) from the usage history of the remote conference system. It may be automatically set based on a totalT setting parameter average value or mode value in the period. Further, when the remote conference is extended beyond the schedule and the conference time exceeds the totalT set before the conference starts, the control unit 130 may automatically increase the totalT value.

(Configuration of terminal device)
The function of the server 100 has been described above. Next, the function of the terminal device 200 will be described with reference to FIG.

  FIG. 8 is an explanatory diagram illustrating a configuration of the terminal device 200 according to the first embodiment. As illustrated in FIG. 8, the terminal device 200 according to the first embodiment includes a communication unit 210, an input unit 220, an imaging unit 230, a sound collection unit 240, a storage unit 250, a display unit 260, an audio output unit 270, and a control. Part 280. As the terminal device 200, for example, a notebook PC (Personal Computer) or a smartphone including a front camera is assumed.

  The communication unit 210 communicates with other devices. For example, the communication unit 210 is directly connected to the LAN 20 and communicates with the server 100. Further, the communication unit 210 communicates with other terminal devices 200 at other bases via the server 100. The communication unit 210 can be realized by the communication interface 811, for example.

  The input unit 220 receives input from the user of the terminal device 200. The input unit 220 provides the input result to the control unit 280. For example, the user can select one display object from the display objects corresponding to each material displayed on the display unit 260 by operating the input unit. Such an input unit 220 can be realized by, for example, a mouse, a touch panel, or a keyboard.

  Note that the input unit 220 may be a line-of-sight detection device. When the input unit 220 is realized by a line-of-sight detection device, a display object to which the user's line of sight is directed is displayed among the display objects corresponding to each material displayed on the display unit 260 by the function of the input unit 220. Can be identified. The method for detecting the line of sight is not particularly limited, and may be, for example, an optical method (corneal reflection method or the like), a physical method (search coil method or the like), or an electric method (EOG method or the like). Furthermore, in recent years, methods have been developed that perform line-of-sight detection and gaze point estimation from a general-purpose imaging device such as a Web camera without using a special measuring instrument, and these methods may be used. . In that case, the input unit 220 can be substituted by an imaging unit 230 described later.

  The imaging unit 230 images a user who is operating the terminal device 200. The imaging unit 230 images, for example, a spatial region in a range in which the display unit 260 of the terminal device 200 can be viewed (the front direction of the display unit 260). Then, the imaging unit 230 provides the imaging result (that is, the captured image) to the control unit 280. Such an imaging unit 230 can be realized by the camera 813, for example.

  The sound collection unit 240 collects sounds around the terminal device 200. For example, the sound collection unit 240 collects the voice of the user who is using the remote conference system by operating the terminal device 200. Then, the sound collection unit 240 provides the sound collection result (that is, sound data) to the control unit 280. Note that the sound collection unit 240 can be realized by a microphone 815, for example.

  The storage unit 250 stores a program and data for the operation of the terminal device 200. For example, the storage unit 250 temporarily stores interest target selection history data and projection interest vector data calculated by the control unit 280 described later, or in the display unit 260 set by the display environment setting unit 283. Store display environment setting data. The storage unit 250 can be realized by the storage device 809, for example.

  The display unit 260 displays various display screens. For example, the display unit 260 displays a conference screen, which is an application screen of the remote conference system according to the embodiment of the present invention, under the control of the display control unit 289 described later. The configuration of the conference screen will be described later in detail. The display unit 260 can be realized by the display device 823, for example.

  The audio output unit 270 outputs audio. The voice output unit 270 outputs voice (for example, voices of other users in the remote conference system and voice media information of shared materials) in accordance with control by the control unit 280. Note that the audio output unit 270 can be realized by the speaker 817, for example.

  The control unit 280 controls the overall operation of the terminal device 200. In particular, the control unit 280 of the terminal device 200 according to the present embodiment has functions of an interest target specifying unit 281, a display environment setting unit 283, a projected interest vector calculation unit 285, and a display control unit 289. Each function of the control unit 280 can be realized by, for example, the CPU 801, the ROM 803, and the RAM 805.

  The interest target specifying unit 281 uniquely specifies which material included in the conference screen displayed on the display unit 260 the user of the terminal device 200 is interested in. The interest target specifying unit 281 may specify, for example, a display object including an input point (one coordinate on the conference screen) provided from the input unit 220 as a display object of a material that the user is interested in. Good. Note that the input point may be a position where the user is turning his / her line of sight, a selection position with a mouse, or the like.

  By the way, according to this embodiment, not all the display objects corresponding to each material are displayed equally, but the display object corresponding to one material of interest to the user is enlarged and displayed on other materials. The corresponding display object is displayed in a reduced size. For this reason, it is highly likely that the user will naturally make an input to select one of the materials of interest, and the terminal device 200 uniquely identifies the interest of interest when gaze detection is used. It becomes easy to specify.

  Note that the terminal device 200 may perform enlargement or reduction processing of the display object corresponding to each material described above, or the server 100 may transmit the enlarged or reduced display object to the terminal device 200. In the latter case, for example, when the material is an image, the server 100 may transmit another display object at a lower resolution or lower frame rate than the display object corresponding to the material that the user is interested in. Therefore, the video reception data capacity and the video reproduction processing load in the terminal device 200 are reduced.

  The display environment setting unit 283 can perform setting to change the layout of the conference screen displayed on the display unit 260 of the terminal device 200 based on user operation data from the input unit 220. The display environment setting unit 283 may store the setting data in the storage unit 250. The layout of the conference screen that can be set by the display environment setting unit 283 will be described later with reference to the drawings.

  The projected interest vector calculation unit 285 acquires abstract interest vector data from the server 100 via the communication unit 210, and has a number of dimensions (for example, two dimensions) that can represent the abstract interest vector data on the display unit 260. Projectively transforms into an interest vector. For example, the abstract interest vector 450A shown in FIG. 7 is projectively transformed into the two-dimensional projected interest vector 350A (concatenated display) shown in FIG. That is, the abstract interest vector 450A is projectively transformed so that the tip of the abstract interest vector 450A corresponds to the barycentric point of the display object 320E corresponding to the material E, and the on-screen position, direction, and size of the projected interest vector 350A are obtained. . Also, the origin 440 in the information space 400 shown in FIG. 7 may be projectively transformed and displayed as the barycentric point 340 on the conference screen 300A shown in FIG. In the drawings attached to the present specification, an example in which the vector length is adjusted while maintaining the direction information so that both ends of the projected interest vector 350A do not overlap the display area of each display object 320 and the user display 330. Is shown.

  The display control unit 289 generates a conference screen for the remote conference according to the present embodiment, and causes the display unit 260 to display the conference screen. For example, the display control unit 289 arranges display objects corresponding to each material on the conference screen according to the layout set by the display environment setting unit 283. Further, the display control unit 289 arranges the user display and the projection interest vector corresponding to each user on the conference screen according to the projection interest vector of each user obtained by the projection interest vector calculation unit 285. Hereinafter, specific examples of the conference screen generated by the display control unit 289 will be described in more detail with reference to FIGS.

/ Specific examples of the meeting screen /
FIG. 9 is an explanatory diagram illustrating a specific example of the conference screen 300A displayed on the display unit 260 of the terminal device 200A used by the user A. As shown in FIG. 9, the conference screen 300A includes a display object 320E corresponding to the material E, a display object 320F corresponding to the material F, and a display object 320G corresponding to the material G. Here, when the interest target specifying unit 281 specifies that the user A is interested in the material E, the display control unit 289 displays the display object 320E corresponding to the material E as shown in FIG. Compared with the other display objects 320F and 320G, it is enlarged and displayed. The sharing of the display object 320 of each material may be realized by a user participating in the remote conference uploading the material to the server 100.

  In addition, the display control unit 289 uses the projection interest vector 350 of each user obtained by the projection interest vector calculation unit 285, and the center of gravity position of the display object 320 corresponding to the material in which the user is interested is the projection interest vector 350. Arrange it to be at the tip. As shown in FIG. 9, the projected interest vector 350 is expressed by, for example, an arrow. Then, the display control unit 289 places a user display 330 corresponding to the user at the other end of the projected interest vector 350. Since the projection interest vector is generated for each user who participates in the remote conference, when three users (user A, user B, and user C) participate in the remote conference, the three users as shown in FIG. Three projected interest vectors 350A, 350B and 350C corresponding to each of the three and three user displays 330A, 330B and 330C are arranged on the conference screen 300A.

  In FIG. 9, a person image communicated between the terminal devices 200 via the server 100 is shown as the user display 330, but the user display 330 can be used to specify a user such as a text display of the user's nickname. May be displayed.

  Also, the conference screen 300A shown in FIG. The barycentric point 340 is arranged at the barycentric position of a polygon formed by connecting the barycentric points of the display objects 320 corresponding to the respective materials. The user can grasp the time relationship in which each user is interested in each material from the positional relationship between the barycentric point 340 and the user display 330 corresponding to each user. However, even if the barycentric point 340 is not included in the conference screen 300A, the user can grasp to some extent the time relationship in which each user is interested in each material from the position of each user display 330. It may not be included in the screen 300A.

  The basic configuration of the conference screen 300 according to the present embodiment has been described above with reference to FIG. Next, with reference to FIG. 10 and FIG. 11, a change example of the conference screen 300 according to a change in the user's interest will be described.

  10 and 11 are explanatory diagrams showing another example of the conference screen 300A displayed on the display unit 260 of the terminal device 200A used by the user A. FIG. When the user A selects the material F using the input unit 220, for example, as shown in FIG. 10, the display object 320F corresponding to the material F is enlarged and displayed on the other display objects, and the projection of the user A The tip of the interest vector 350A indicates the display object 320F corresponding to the material F. On the other hand, when the user A selects the material G using the input unit 220, for example, the display object 320G corresponding to the material G is enlarged and displayed with respect to the other display objects as shown in FIG. The tip of the projected interest vector 350A indicates the display object 320G corresponding to the material G.

  Each user can freely select the materials described above, and therefore, it is of course assumed that different materials are selected among the users. For example, when the conference screen 300A shown in FIG. 9 is displayed on the display unit 260 of the terminal device 200A used by the user A, the display unit 260 of the terminal device 200B used by the user B displays the conference shown in FIG. Screen 300B may be displayed. On the conference screen 300B, as shown in FIG. 12, the display object 320F corresponding to the material F selected by the user B using, for example, the input unit 220 is enlarged and displayed on the other display objects.

  The display control unit 289 controls the layout of the conference screen according to the layout setting by the display environment setting unit 283 based on the user operation. Hereinafter, variations of the layout of the conference screen will be described with reference to FIGS. 13 and 14.

  13 and 14 are explanatory diagrams showing variations of the layout of the conference screen. As a result of the user A changing the position of the display object corresponding to each material using, for example, the input unit 220 of the terminal device 200A, as shown in FIG. 13, for example, the user A seems to be upside down from the conference screen 300A shown in FIG. A conference screen 300A having a simple layout may be displayed. Further, as shown in FIG. 14, the conference screen 300A having a layout in which the positions of the display object 320E corresponding to the material E and the display object 320G corresponding to the material G are different from the conference screen 300A shown in FIG. obtain.

  At this time, the arrangement of the user displays 330A, 330B, and 330C and the projection interest vectors 350A, 350B, and 350C is automatically calculated by the projection conversion calculation process of the projection interest vector calculation unit 285 described above, and the display of the conference screen 300A is displayed. Be changed. For example, when the user A performs an operation for changing the layout of the display object 320 corresponding to each material on the conference screen 300A shown in FIG. 9, in addition to the layout of the display object 320, as shown in FIGS. The arrangement of the user displays 330A, 330B and 330C and the projection interest vectors 350A, 350B and 350C are also changed.

  It should be noted that the user operates only the shared material among the many display elements on the conference screen 300, so that his / her preference of the display layout and which material he / she wants to view (for example, the material E is not displayed at the top of the screen). I want to browse at the bottom of the screen). Most remote conferencing systems equipped with a document sharing function share screens by sharing screens, and remote conference participants cannot use the remote conference application with their preferred display layout. In comparison, the layout changing function of the present embodiment is useful.

  The change in the conference screen 300 accompanying the change in the user's interest and the layout change have been described above. Next, a change with time change of the conference screen 300 will be described.

  FIG. 15 is an explanatory diagram illustrating a specific example of the conference screen 300B displayed on the display unit 260 of the terminal device 200B used by the user B after a lapse of time since the conference screen 300B illustrated in FIG. 12 is displayed. . However, in order to simplify the drawing, the user display 330 and the projection interest vector 350 regarding the user A and the user B are omitted in FIG.

  A user display 330Cpast indicated by a broken line in FIG. 15 is a position of the user display 330C on the conference screen 300B illustrated in FIG. When the user C continues to select the material E after the display of the conference screen 300B shown in FIG. 12, the user display 330C of the user C approaches the display object 320E corresponding to the material E, as shown in FIG. The projection interest vector 350C of the user C is also shortened.

(Operation)
The configuration of the remote conference system according to the first embodiment has been described above. Subsequently, the operation of the remote conference system according to the first embodiment is organized.

  FIG. 16 is an explanatory diagram illustrating the operation of the remote conference system according to the first embodiment. Steps S1301 to S1311 in FIG. 16 show processing until the remote conference is started between the terminal device 200A, the terminal device 200B, the terminal device 200C, and the server 100, and after S1313, the user A inputs at the terminal device 200A. The processing in the case of performing

  As shown in FIG. 16, when the terminal device 200A, the terminal device 200B, and the terminal device 200C accept input corresponding to the remote conference participation desires of the users A, B, and C in step S1301, respectively, in step S1303, the terminal device 200A, terminal device 200B, and terminal device 200C transmit a new participation request message to the remote conference to server 100.

  In step S1305, the server 100 generates, as data, a virtual space of a remote conference in which users A, B, and C participate. The data may include, for example, the above-described abstract interest vector information space 400. In step S1307, the server 100 transmits data related to the participation process in the remote conference to the terminal device 200A, the terminal device 200B, and the terminal device 200C. At this time, processing related to call control may be performed between the terminal device 200 and the server 100.

  Thereby, in step S1309, the remote conference session is started among the terminal device 200A, the terminal device 200B, the terminal device 200C, and the server 100. In step S1311, the display of the terminal device 200A, the terminal device 200B, and the terminal device 200C is displayed. In part 260, display of conference screen 300 of the remote conference system is started. The display process may include, for example, a projective conversion process performed by the projected interest vector calculation unit 285 of the abstract interest vector received from the server 100. Further, the conference screen 300 may have a different layout for each of the terminal device 200A, the terminal device 200B, and the terminal device 200C based on the layout setting by the display environment setting unit 283 described above.

  Thereafter, when terminal device 200A accepts input from user A in step S1313, terminal device 200A transmits data based on the input from user A to server 100 in step S1315. The input of the user A is, for example, a material selection operation, and data indicating the material specified by the interest target specifying unit 281 based on the selection operation is transmitted to the server 100.

  In step S1317, the server 100 changes the data in the virtual space based on the user A input data. For example, the abstract interest vector calculation unit 137 of the server 100 may perform recalculation processing on the position, direction, and size of each abstract interest vector in the information space 400.

  Subsequently, in step S1319, the server 100 transmits the changed virtual space data to the terminal device 200A, the terminal device 200B, and the terminal device 200C. Thereby, in step S1321, the display of the conference screen 300 of the remote conference system is changed in the display unit 260 of the terminal device 200A, the terminal device 200B, and the terminal device 200C in accordance with the received virtual space data.

  Note that although an example in which an input by the user A is described in FIG. 16, the same processing is performed even when an input by the user B or the user C is performed. Further, even if inputs are made simultaneously by a plurality of users, the server 100 can simultaneously accept inputs from a plurality of users.

  In the description of FIG. 16, the process of transmitting conference data from the server 100 to the terminal device 200 is explicitly shown as in steps S <b> 1311 and S <b> 1319, but the timing of data transmission is not limited. For example, since a constant-connection network environment is common in a remote conference, the terminal device 200 may continuously receive conference data from the server 100. In particular, the terminal device 200 may perform data communication with the server 100 in real time when human video data or audio data in a remote conference, or when the material is video streaming data.

(Supplement)
In the above, with reference to FIG. 7, the server 100 calculates the abstract interest vector 450 that connects the coordinate position 430 in the information space 400 and the maximum set value maxMV410 on the coordinate axis corresponding to any of the materials, and the terminal device 200 calculates the abstract interest. The example in which the projection interest vector 350 is obtained from the vector 450 and the projection interest vector 350 is arranged on the conference screen 300 has been described. As another method, the server 100 may provide the terminal device 200 with the coordinate position 430 in the information space 400. In this case, the terminal device 200 arranges the user display 330 at a position corresponding to the coordinate position 430 on the conference screen 300, and the display object 320 corresponding to the user display 330 and the material that the user is currently interested in. You may arrange | position an arrow between. Also by this method, it is possible to obtain a conference screen 300 equivalent to the example described with reference to FIG.

<2. Second Embodiment>
The first embodiment of the present invention has been described above. Subsequently, a second embodiment of the present invention will be described. In a meeting, it is also important that each user's degree of interest in the “place” of the meeting (the amount of will to be involved in the meeting) can be shared. The degree of interest tends to be difficult to communicate and share. The present inventor has created the second embodiment in view of the circumstances. According to the second embodiment, it is possible to share the degree of interest of each user in the remote conference among the users. Hereinafter, the second embodiment of the present invention will be described in detail.

(Configuration of terminal device)
FIG. 17 is an explanatory diagram illustrating a configuration of the terminal device 202 according to the second embodiment. As illustrated in FIG. 17, the terminal device 202 according to the second embodiment includes a communication unit 210, an input unit 220, an imaging unit 230, a sound collection unit 240, a storage unit 250, a display unit 260, an audio output unit 270, and a control. Part 280. The control unit 280 has functions of an interest target specifying unit 281, a display environment setting unit 283, a projected interest vector calculation unit 285, an interest level determination unit 287, and a display control unit 289. The interest level determination unit 287 is a main difference between the terminal device 202 according to the second embodiment and the terminal device 200 according to the first embodiment.

  The interest level determination unit 287 is based on the input data or sensing data acquired by the input unit 220, the imaging unit 230, the sound collection unit 240, or the like, to what extent the user of the terminal device 202 is interested in the remote conference. Whether the user has (interested degree) is obtained, and data indicating the interested degree is transmitted to the server 102 via the communication unit 210. The interest level determination unit 287 can determine the interest level of the user by various methods. Hereinafter, several methods for obtaining the degree of interest will be described.

/ First method /
The interest level determination unit 287 may calculate, as the interest level, the ratio of the time during which the user was interested in the conference screen 300 to the conference time. When the conference screen 300 occupies the entire area of the display unit 260, the above ratio is the ratio of the time the user was watching the display unit 260 to the conference time.

  For example, if the time during which the user has been interested in the conference screen 300 is 90 minutes and the other time is 30 minutes in the 120 minutes of the conference time, the interest level determination unit 287 sets the interest level as “90/120 = 0. .75 "is calculated. The degree of interest may not be calculated at the end of the conference, but may be calculated sequentially as the conference time elapses.

  Note that the fact that the user is looking at the display unit 260 is not necessarily synonymous with the fact that the user is looking at the conference screen 300. For example, when the terminal device 202 has a general-purpose computer function, the user does not always display the conference screen 300 on the display unit 260, and is different from the remote conference on the terminal device 202. There is also a possibility of temporarily starting up and using a work application (“in-house” in a remote conference). At this time, even if the user looks at the display unit 260, the user does not see the conference screen 300. For this reason, the interest level determination unit 287 obtains information as to whether or not the user is displaying the conference screen 300 on the display unit 260 of the terminal device 202, and sets the time when the conference screen 300 is not displayed as another space. It may be counted as the time of viewing the area.

/ Second method /
As described above, when the imaging unit 230 captures a spatial area in the front direction of the display unit 260, the face recognition succeeds while the user is viewing the conference screen 300 displayed on the display unit 260, and the user It is considered that face recognition is not successful while the face is turned in a direction different from 260 or when the user is away from the desk. For this reason, the degree-of-interest determination unit 287 may calculate, as the degree of interest, the ratio of the time when face recognition is successful from the video by the imaging unit 230 of the terminal device 202 to the meeting time.

/ Third method /
The interest level determination unit 287 may calculate the interest level of the user based on the amount of utterances by the user in the remote conference, for example, the total length of utterances, the number of utterances, the utterance volume, and the like. Specifically, the degree-of-interest determination unit 287 recognizes a user's utterance from the data obtained by the sound collection unit 240, and uses a value obtained by relativizing the amount of utterance with the reference value of the user as the degree of interest of the user. It may be calculated. The reference value of the user may be a statistical value (for example, an average value) of the amount of utterance of the user in the past remote conference.

  In addition, the tendency of the way of participating in a meeting changes with users. Some users tend to speak more at meetings, while others tend to speak less at meetings. For this reason, it may be difficult to appropriately calculate the degree of interest of each user in the meeting by comparing the amount of speech between users in a certain meeting. On the other hand, a user who speaks more than usual is considered highly interested in the conference, and a user who speaks less than usual is considered less interested in the conference. Therefore, by comparing the average utterance amount of a certain user as described above with the utterance amount of the user in the remote conference, it is possible to calculate a more appropriate degree of interest regarding the user.

  In addition, even if the same user participates in several remote conferences with the same interest level, the total length of the user's absolute utterances and the number of spoken words depends on the number of users participating in each remote conference. It is assumed that they are different. Therefore, the interest level determination unit 287 adjusts the user's reference value according to the number of users participating in the remote conference, for example, and relativizes the user's utterance amount by adjusting the user's utterance amount. It may be calculated.

/ Fourth method /
If the user's arousal level is low, that is, if the user is attacked by drowsiness, the user is likely not actively participating in the remote conference. Therefore, the interest level determination unit 287 may calculate the user's arousal level as the user's level of interest.

  The user's arousal level can be calculated by various methods. For example, when the input unit 220 is realized by a line-of-sight detection device, the interest level determination unit 287 may calculate the user's arousal level using the pupil diameter of the eyeball measured by the input unit 220. In addition, the interest level determination unit 287 may estimate a pulse from the skin area of the user's human video and calculate the user's arousal level using the fluctuation data of the pulse.

/ Fifth method /
The interest level determination unit 287 may generate a display change event on the display unit 260 or the conference screen 300, and calculate the interest level based on a user's reaction to the display change event. As the display change event, for example, a display shielding process such as a screen saver, or a process of making the display of the conference screen 300 difficult to visually recognize as a predetermined time elapses (the screen brightness gradually decreases and the background transparency gradually increases). Etc.

  The interest level determination unit 287 eliminates the display change event based on a user input such as a single tap on the display unit 260. A user who has a strong interest in the remote conference and is carefully watching the conference screen 300 thinks that when the conference screen 300 becomes invisible or the visibility of the conference screen decreases, an operation to cancel these display change events is performed. It is done. On the other hand, it is considered that the user who is not interested in the remote conference and does not look at the conference screen 300 does not perform the operation for canceling the display change event or delays the operation for canceling the display change. Therefore, the interest level determination unit 287 may calculate the interest level of the user based on the presence or absence of the user's response to the display change event or the speed of the response.

/combination/
The relationship between the first method to the fifth method described above is not an exclusive relationship, and the interest level determination unit 287 combines the two or more methods from the first method to the fifth method described above to determine the interest level. It may be calculated. For example, the interest level determination unit 287 may determine the average value or the mode value of the interest level calculated by each method as the interest level to be transmitted to the server 102.

(Server configuration)
The configuration of the terminal device 202 according to the second embodiment has been described above. Next, the configuration of the server 102 according to the second embodiment will be described with reference to FIG.

  FIG. 18 is an explanatory diagram showing the configuration of the server 102 according to the second embodiment. As illustrated in FIG. 18, the server 102 according to the second embodiment includes a communication unit 110, a storage unit 120, and a control unit 130. The control unit 130 has functions of a person video control unit 131, a shared material control unit 133, another media information control unit 135, an abstract interest vector calculation unit 137, an interest level management unit 139, and a presence adjustment unit 141. The interest level management unit 139 and the presence adjustment unit 141 are the main differences between the server 102 according to the second embodiment and the server 100 according to the first embodiment.

  The degree-of-interest management unit 139 acquires data indicating the degree of interest from the terminal device 202 of each user participating in the remote conference, and manages the data centrally. Also, the interest level management unit 139 provides the presence adjustment unit 141 with data indicating the acquired interest level. Note that the data indicating the degree of interest may be stored in the storage unit 120.

  The presence adjusting unit 141 adjusts the display mode of each user display 330 on the conference screen 300 according to the level of interest of each user based on the data indicating the interest level of each user provided from the interest level management unit 139. Processing, that is, adjustment processing of presence of each user.

  The presence adjustment may be, for example, adjustment of the amount of information included in the person video. In this case, the presence adjusting unit 141 instructs the information amount of the personal video data provided to the terminal device 202 of each user by the human video control unit 131 according to the degree of interest of each user. For example, the presence adjustment unit 141 adjusts the person video control unit 131 so that the resolution or display size of the person video of the user who is less interested in the remote conference is smaller than the person video of the user who is more interested in the user. May be indicated. Alternatively, the person video control unit 131 reduces the brightness of the person video of the user who is less interested in the remote conference or the person video of the user who is less interested in the remote conference based on the instruction from the presence adjusting unit 141. May be subjected to blur filter processing.

  As a result, a user having a relatively low interest level in the remote conference is represented on the conference screen 300 by the user display 330 having a low presence. For this reason, each user participating in the remote conference can intuitively grasp how much each user is interested in the remote conference.

/ Presence adjustment example /
Hereinafter, specific examples of presence adjustment will be described with reference to FIGS.

  FIG. 19 shows a conference screen 300B displayed on the terminal device 202B used by the user B before the presence is adjusted. FIG. 20 shows a conference screen 300B displayed on the terminal device 202B used by the user B after the presence is adjusted.

  For example, the presence adjusting unit 141 acquires the interest level “1.0” of the user A, the interest level “2.0” of the user B, and the interest level “0.5” of the user C from the interest level management unit 139. In this case, the presence adjustment unit 141 adjusts the person video control unit 131 so that the user display 330B corresponding to the user B has a size (resolution, number of pixels) four times that of the user display 330A corresponding to the user A. Instruct. In addition, the presence adjustment unit 141 instructs the person video control unit 131 to perform adjustment so that the user display 330C corresponding to the user C is ¼ times the size of the user display 330A corresponding to the user A.

  As a result, as shown in FIG. 20, on the conference screen 300B, the user display 330B corresponding to the user B is displayed in a size four times the user display 330A corresponding to the user A, and the user display 330C corresponding to the user C is displayed. It is displayed in a size that is 1/4 times the user display 330A corresponding to user A.

  In addition, although the example in which the person video control unit 131 of the server 102 adjusts the information amount of the user display 330 has been described above, the terminal device 202 can also adjust the information amount. For example, the server 102 provides each terminal device 202 with the display of each user whose information amount is not adjusted and the degree of interest of each user, and each terminal device 202 has the user of each user based on the degree of interest of each user. The amount of information displayed may be adjusted.

  Next, another example of presence adjustment will be described. The presence adjusting unit 141 may instruct the abstract interest vector computing unit 137 to change the end coordinate position 430 of the user's abstract interest vector 450 according to the magnitude of the user's interest level. For example, when the interest level of a certain user is lower than a predetermined value, it is considered that the user's interest in the display object is low in the remote conference. For this reason, when the interest level of a certain user is lower than a predetermined value, the presence adjustment unit 141 may stop the abstract interest vector calculation unit 137 from updating the coordinate position 430 related to the user, You may make it approach the origin 440 gradually.

  FIG. 21 is an explanatory diagram showing an information space 400 of an abstract interest vector corresponding to the conference screen 300. An abstract interest vector 450Apast indicated by a broken line in FIG. 21 is an abstract interest vector of the user A before presence adjustment is performed. On the other hand, an abstract interest vector 450A indicated by a solid line is an abstract interest vector of user A after presence adjustment.

  When the interest level of the user A in the remote conference is low, as shown in FIG. 21, the coordinate position 430Apast of the abstract interest vector 450Apast is moved to the coordinate position 430A close to the origin 440, and the abstract interest vector indicated by a solid line 450A is obtained. As a result, the positions of the projected interest vector 350A and the user display 330A on the conference screen displayed on the terminal device 202 are also changed. In this way, not only the amount of information on the user display 330 but also the position of the user display 330 can be changed according to the degree of interest of each user.

  In addition, when the interest level of a certain user is lower than a predetermined value, the presence adjustment unit 141 does not display the projected interest vector 350 corresponding to the user until the interest level of the user exceeds a predetermined value. An instruction may be issued to the display control unit 289 of the apparatus 202.

<3. Conclusion>
As described above, according to the embodiment of the present invention, a conference screen including a display object corresponding to each of a plurality of materials used for a remote conference is provided to each user. Furthermore, since the user display corresponding to each user is arranged at a position corresponding to the transition of each user's interest on the conference screen, each user should take into account the interests of other users as in the face-to-face meeting. Therefore, it is possible to realize smooth communication even in a remote conference.

  According to the embodiment of the present invention, the display environment setting unit 283 can change the layout setting of the conference screen based on a user operation. Most remote conferencing systems equipped with a document sharing function share screens by sharing screens, and remote conference participants cannot use the remote conference application with their preferred display layout. In comparison, the layout changing function of the present embodiment is useful.

  In addition, according to the second embodiment of the present invention, by adjusting the presence of the user display corresponding to each user according to the degree of interest of each user, the degree of interest in the remote conference of each user is changed between users. It is possible to share.

  Although the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

  For example, each step in the processing of the remote conference system of the present specification does not necessarily have to be processed in chronological order in the order described as a sequence diagram. For example, each step in the processing of the remote conference system may be processed in an order different from the order described as the sequence diagram or may be processed in parallel.

  In addition, it is possible to create a computer program for causing hardware such as a CPU, ROM, and RAM incorporated in the server 100 and the terminal device 200 to perform the same functions as the components of the server 100 and the terminal device 200 described above. . A storage medium storing the computer program is also provided.

100, 102 Server 110 Communication unit 120 Storage unit 121 Interest transition history storage unit 130 Control unit 131 Human image control unit 133 Shared material control unit 135 Other media information control unit 137 Interest vector calculation unit 139 Interest level management unit 141 Presence adjustment unit 200 202 Terminal device 210 Communication unit 220 Input unit 230 Imaging unit 240 Sound collection unit 250 Storage unit 260 Display unit 270 Audio output unit 280 Control unit 281 Interest target identification unit 283 Display environment setting unit 285 Projection interest vector calculation unit 287 Interest level determination Part 289 Display control part

Claims (16)

An information processing system having a plurality of terminal devices and displaying a plurality of display objects common to each of the plurality of terminal devices,
A display control unit that generates a display screen including the plurality of display objects;
The information processing system, wherein the display screen includes a display showing a transition of a display object in which each user of the plurality of terminal devices is interested in the plurality of display objects.   The information processing system according to claim 1, wherein the display includes a user display that is arranged in association with a position corresponding to a transition of a display object that the user is interested in for each user.   The information processing system according to claim 2, wherein the display includes a linked display that connects the user display and a display object in which the user is interested. The information processing system is configured to calculate a coordinate position expressed by coordinate values on a coordinate axis corresponding to each display object from a time when the user is interested in each display object among the plurality of display objects. Further comprising
The information processing system according to claim 2, wherein the display control unit arranges the user display at a position corresponding to the coordinate position on the display screen. The information processing system includes:
An abstract interest vector calculation unit that calculates an abstract interest vector that connects the coordinate position and a predetermined value on the coordinate axis corresponding to the display object that the user is currently interested in,
A projective transform unit for projectively transforming the abstract interest vector into a projective interest vector having a number of dimensions that can be expressed on the display screen;
The information processing system according to claim 4, wherein the display control unit arranges the user display at a position of the other end when the display object that the user is currently interested in is one end of the projection interest vector. . The display screen is used for a remote conference by users of the plurality of terminal devices,
The position calculation unit is configured to multiply a value obtained by dividing the time when the user is interested in a display object by the scheduled time of the remote conference by the predetermined value on the coordinate axis corresponding to the display object. The information processing system according to claim 5, wherein a coordinate value on a coordinate axis corresponding to the display object is calculated.   The information processing system according to claim 2, further comprising an adjustment unit that adjusts a display mode of the user display for each user. The display screen is used for a remote conference by users of the plurality of terminal devices,
The information processing system according to claim 7, wherein the adjustment unit performs the adjustment according to a degree of interest of each user in the remote conference.   The information processing system according to claim 8, wherein the adjustment unit performs the adjustment such that a user having a higher degree of interest in the remote conference has a higher visibility of the user display of the user.   The adjustment unit performs the adjustment according to a length of time that each user of the plurality of terminal devices has been interested in the display screen as a degree of interest of each user in the remote conference. 9. The information processing system according to 9.   The adjustment unit, as the degree of interest of each user in the remote conference, the time when face recognition succeeded for each user from a captured image in a range in which the display screen displayed on each of the plurality of terminal devices can be visually recognized The information processing system according to claim 8 or 9, wherein the adjustment is performed according to a length.   The information processing system according to claim 8 or 9, wherein the adjustment unit performs the adjustment according to an amount of each user's utterance as a degree of interest of each user in the remote conference. The display screen is used for a remote conference by users of the plurality of terminal devices,
The information processing system includes:
The information processing system according to claim 4, further comprising an adjustment unit that adjusts the coordinate position of each user calculated by the position calculation unit according to the degree of interest of each user in the remote conference.   The information processing system according to any one of claims 1 to 13, wherein the display control unit changes an arrangement or size of the plurality of display objects on the display screen based on an operation from the user. Each of the plurality of terminal devices includes displaying a display screen including a plurality of common display objects,
The information processing method, wherein the display screen includes a display indicating a transition of a display object in which each user of the plurality of terminal devices is interested in the plurality of display objects. A program for realizing an information processing system having a plurality of terminal devices and displaying a plurality of display objects common to each of the plurality of terminal devices,
Computer
A display control unit configured to generate a display screen including a display indicating a transition of a display object in which each user of the plurality of display objects and the plurality of terminal devices is interested in the plurality of display objects;
Program to function as

Images