JP6729193B2 - Information processing server, information processing system, terminal device, and program - Google Patents

Description

The present invention relates to an information processing server, an information processing system, a terminal device, and a program.

In recent years, softphones realized by application software have become popular in place of conventional telephones realized by hardware. Since the softphone is realized by software, addition of functions to the softphone, cooperation between the softphone and other application software, etc. can be realized relatively easily as compared with the telephone. Therefore, various applied technologies regarding softphones have been proposed. For example, in Patent Document 1 below, when a person shown in the captured image is touched on a display screen of the captured image obtained from a camera, the person is identified by face recognition, and the telephone number of the person is acquired. A technique for making a telephone call to the person is disclosed.

Further, Patent Document 2 below discloses an information processing method capable of handling position information in a real space on a captured image in the real space in a softphone having a video call function, for example.

The softphone as described above is also used in a unified communication (registered trademark) technology/system that integrates a fax (facsimile), an electronic mail, an instant messenger, VoIP (Voice over Internet Protocol), and the like. The unified communication technology/system is used in a remote distributed collaborative office (distributed environment) in which a plurality of persons collaborate at a plurality of different places.

JP, 2007-208863, A Japanese Patent No. 5692204

The act of "conversation" is not always performed by two people, but is often performed by a group of three or more people. As a conversation conducted in a group of three or more people, for example, in addition to the pattern in which all the members of the group are all in place at the beginning of the conversation, other people notice the existence of the conversation started (Awareness) and become aware. There is a pattern in which the number of members of the group having the conversation increases as the person who joins the conversation later on. In particular, it is considered that conversational actions based on informal communication (communication that occurs accidentally without being planned in advance such as chat) often occur in the latter pattern. In the latter case, the surrounding people before joining the conversation have information about the occurrence of the conversation and the rough contents of the conversation (a fragment having a granularity sufficient to determine whether or not the user wants to participate in the conversation later). In many cases, it is in a state where it is possible to know basic conversation information).

However, it is premised that the conventional softphone products are used in intentional, formal, and formal communication such as a discussion after the agenda is decided. That is, since the conventional softphone product assumes a pattern in which all members have the same conversation from the beginning, a video conference (multi-party call) function in which all members start conversation (communication) all at once, It has only the function of being able to know the binary information indicating whether or not the existing member is in the call state on the presence list (location list) of the display screen. Therefore, when a conventional softphone product is used in a remote distributed collaborative office (distributed environment), the presence of conversation can be felt by hearing the surrounding conversation naturally, as if the members were in the same room environment. However, it is difficult to realize the natural act of participating in a conversation and finally reaching a multi-person conversation because of the interest in the conversation from the general understanding of the conversation content. That is, the conventional softphone products do not support the above-mentioned informal communication.

Therefore, the present invention has been made in view of the above circumstances, and an object of the present invention is to allow a user to grasp the occurrence of a conversation at a remote place and the rough content of the conversation in a distributed environment. In addition, it is possible to provide a new and improved information processing server, information processing system, terminal device, and program that enable intuitive recognition of who is in a call.

In order to solve the above problems, according to an aspect of the present invention, a conversation event object relating to the call is generated by linking communication identification information of a plurality of speakers involved in the call, and from the voice data of the call. An information processing server is provided that includes a control unit that generates a language phrase object related to an extracted phrase and controls distribution of data related to the conversation event object and data related to the language phrase object .

The control unit may acquire a user's input to the conversation event object and associate the communication identification information of the plurality of speakers associated with the conversation event object with the communication identification information of the user.

It said information processing server further includes a weighting processing portion that performs weighting processing for the previous SL onset language phrase objects, the control unit compares the results with the predetermined value of the weighting processing, the comparison result The distribution of data relating to the language phrase object may be controlled based on

The weighting processing unit may perform the weighting processing based on the frequency of appearance of the phrase in the call.

The weighting processing unit may perform the weighting processing based on the degree of abstraction of the phrase.

The weighting processing unit may perform the weighting processing based on a part-of-speech category of the phrase.

The weighting processing unit may perform the weighting processing based on the data relating to the sound pressure of the utterance of the phrase included in the voice data of the call.

Wherein, in association cord to the remarks phrase object, it may control the distribution of data according to the result of the weighting processing.

The information processing server further determines a display position of the language phrase object based on a position of the speaker who has said a phrase related to the language phrase object, and further includes a statement status calculation unit that distributes the determined display position. You may prepare.

The control unit , based on the positional relationship between one of the plurality of speakers involved in the call and a user who is not participating in the call in the real space , Distribution may be controlled .

The control unit , based on the positional relationship between one of the plurality of speakers involved in the call and a user who is not participating in the call in the real space , Distribution may be controlled .

The information processing server acquires an input of the interest level of the user with respect to the call, and associates the acquired interest level with the communication identification information of the user with respect to the conversation event object. You may further prepare.

The information processing server further includes a display body control unit that generates a display body for the user, and determines a virtual positional relationship between the position of the conversation event object and the display body based on the acquired degree of interest. You may prepare.

It said information processing server further includes a weighting processing portion that performs weighting processing on the calling language phrase objects, the control unit compares the result of the weighting processing, and a predetermined value, the comparison result The distribution of the data related to the language phrase object is controlled based on, and the predetermined value may be changed based on the acquired degree of interest.

The degree-of-interest control unit acquires an input of the degree-of-interest ratio of the plurality of speakers involved in the call of the user, associates the input with the conversation event object, and obtains the data regarding the acquired degree-of-interest ratio. You may deliver.

It said information processing server further includes a weighting processing portion that performs weighting processing on the calling language phrase objects, the control unit compares the result of the weighting processing, and a predetermined value, the comparison result The distribution of the data related to the language phrase object is controlled based on, and the predetermined value may be changed based on the acquired interest level and the acquired ratio of the interest level.

The interest degree control unit may associate the communication identification information of the plurality of speakers associated with the conversation event object with the communication identification information of the user who has input the interest degree for the call. ..

In order to solve the above problems, according to another aspect of the present invention, a three-dimensional virtual space corresponding to a real space in which a plurality of speakers involved in a call exist is generated, and a plurality of talks related to the call are generated. A plurality of objects corresponding to each of the communication identification information of the person and a conversation event object relating to the call, which is associated with the plurality of objects, and relates to a phrase extracted from the voice data of the call. An information processing server is provided, which includes a control unit that generates a language phrase object and arranges data relating to the conversation event object and data relating to the language phrase object in the three-dimensional virtual space.

The information processing server acquires an input of a degree of interest of the user with respect to the call, arranges a user object corresponding to the user in the three-dimensional virtual space, and based on the obtained degree of interest, the three-dimensional virtual space. In, the display control unit may further include a display body control unit that determines a virtual distance between the conversation event object and the user object.

The information processing server acquires an input of a ratio of interest levels for the plurality of speakers involved in the call of the user, and based on the acquired ratio of interest levels, on the generated conversation event object, the An interest degree control unit may be further provided, which arranges a reference point indicating a ratio of interest degrees.

In order to solve the above problems, according to still another aspect of the present invention, there is provided an information processing system including an information processing server and a plurality of terminal devices, wherein the information processing server relates to a call. Data relating to the conversation event object, which is associated with communication identification information of a plurality of speakers, generates a conversation event object relating to the call, generates a language phrase object relating to a phrase extracted from the voice data of the call, and An information processing system is provided that distributes data relating to the language phrase object to the plurality of terminal devices.

Further, in order to solve the above-mentioned problems, according to still another aspect of the present invention, a conversation event object related to the call and a voice of the call are associated with communication identification information of a plurality of speakers involved in the call. A terminal device is provided that includes a display unit that displays a language phrase object related to a phrase extracted from data .

The terminal device acquires the result of the weighting process performed on the language phrase object, compares the result of the weighting process with a predetermined value, and based on the comparison result, displays the display unit. You may further provide the language phrase object control part which controls.

The terminal device obtains the result of the weighting process performed on the language phrase object, and based on the result of the weighting process, the size, color, contrast, and display position of the language phrase object. A language phrase object control unit that controls any one may be further provided.

The terminal device may further include a voice output control unit that acquires a degree of interest of the user in the call and controls output of a voice related to the call based on the degree of interest.

The terminal device may further include a conversation interest level setting unit that acquires an interest level of the user with respect to the call based on an operation performed by the user on a user object of the user displayed on the display unit.

The terminal device may further include an imaging unit that acquires a face image of the user in order to display the user object.

The terminal device may further include a conversation interest level notification unit that performs a notification indicating the presence of the user based on the acquisition of the interest level of the user for the call.

The conversation interest level notification unit displays a notification display indicating the presence of the user on the display unit, and based on the acquired interest level, any one of the size, color, movement, contrast, and display position of the notification display. You may control one.

The conversation interest level notification unit causes the audio output unit to perform audio output indicating the presence of the user,
The volume of the audio output may be controlled based on the acquired degree of interest.

Further, in order to solve the above-mentioned problems, according to still another aspect of the present invention, a computer generates a conversation event object relating to the call, which associates a computer with communication identification information of a plurality of speakers involved in the call. Then, a program for generating a language phrase object relating to a phrase extracted from the voice data of the call, and causing it to function as a control unit for controlling the distribution of the data relating to the conversation event object and the data relating to the language phrase object. Will be provided.

As described above, according to the present invention, a user can grasp the occurrence of a conversation at a remote place and a rough content of the conversation in a distributed environment, and further intuitively know who is in a call state. Can be recognized.

It is explanatory drawing which shows an example of a schematic structure of the information processing system which concerns on 1st Embodiment. It is a block diagram which shows an example of the hardware constitutions of the terminal device 100 which concerns on 1st Embodiment. It is a block diagram which shows an example of a functional structure of the terminal device 100 which concerns on 1st Embodiment. It is an explanatory view for explaining an example of a display screen displayed in a bird's-eye view mode according to the first embodiment. FIG. 7 is an explanatory diagram illustrating an example of a display screen displayed in the proximity mode according to the first embodiment. FIG. 8 is an explanatory diagram for explaining a first example of a position in the close-up captured image 71 designated by the user. FIG. 11 is an explanatory diagram for explaining a second example of the position in the close-up captured image 71 designated by the user. FIG. 3 is an explanatory diagram illustrating a first example of a three-dimensional virtual space 90 corresponding to the center office 10. FIG. 9 is an explanatory diagram for explaining an example of selection of an object 91 arranged in the three-dimensional virtual space 90 shown in FIG. 8. FIG. 9 is an explanatory diagram for explaining a second example of the three-dimensional virtual space 90 corresponding to the center office 10. FIG. 11 is an explanatory diagram for describing an example of selection of an object 91 arranged in the three-dimensional virtual space 90 shown in FIG. 10. It is an explanatory view for explaining an example of display screen 80 displayed in conversation mode. It is a transition diagram for explaining an example of transition of display modes. FIG. 6 is an explanatory diagram illustrating an example of a COMM link 86 in a three-dimensional virtual space 90 corresponding to the center office 10. FIG. 8 is an explanatory diagram illustrating an example of a COMM link 86 and a COMM word 87 displayed on the display screen 50 of the terminal device 100. FIG. 11 is an explanatory diagram for explaining another example of the COMM link 86 and the COMM word 87 displayed on the display screen 55 of the terminal device 100. It is a block diagram which shows an example of the software configuration of the terminal device 100 which concerns on 1st Embodiment. It is a block diagram which shows an example of the hardware constitutions of the information management server 200 which concerns on 1st Embodiment. It is a block diagram which shows an example of a functional structure of the information management server 200 which concerns on 1st Embodiment. It is a block diagram which shows an example of the hardware constitutions of the voice recognition server 201 which concerns on 1st Embodiment. It is a block diagram which shows an example of a functional structure of the voice recognition server 201 which concerns on 1st Embodiment. 3 is a flowchart showing an example of a schematic flow of information processing according to the first embodiment. 6 is a flowchart showing an example of a schematic flow of a startup process according to the first embodiment. It is a sequence diagram which shows an example of the schematic flow of the communication control process which concerns on 1st Embodiment. It is a block diagram showing an example of functional composition of information management server 200A concerning a 2nd embodiment. 9 is a flowchart showing an example of a schematic flow of information processing according to the second embodiment. It is a block diagram showing an example of functional composition of terminal unit 100A concerning a 3rd embodiment. It is an explanatory view for explaining an example of third party object 94 and virtual distance 97 in three-dimensional virtual space 90 corresponding to a distributed office. It is explanatory drawing for demonstrating an example of the third party object and virtual distance displayed on the display screen of the terminal device 100A which concerns on 3rd Embodiment. It is an explanatory view for explaining other examples of the third party object 94 and virtual distance 97 displayed on display screen 57 of terminal unit 100A concerning a 3rd embodiment. It is an explanatory view for explaining an example of setting of a speaker interest ratio reflection position 96 in a three-dimensional virtual space 90 corresponding to a distributed office. It is an explanatory view for explaining an example of setting of speaker interest ratio reflection position 96 displayed on display screen 58 of terminal unit 100A concerning a 3rd embodiment. FIG. 11 is an explanatory diagram illustrating an example of a display screen 62 when a speaker interest ratio reflection position 96 is set by the third party object 94. It is explanatory drawing for demonstrating an example of the display of the notification containing the information regarding the conversation interest degree displayed on the display screen of the terminal device 100A which concerns on 3rd Embodiment. It is a block diagram showing an example of functional composition of information management server 200B concerning a 3rd embodiment. FIG. 16 is a sequence diagram showing an example of a schematic flow of conversation interest level setting, communication information amount change of COMM word 87, and notification process according to the third embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification and the drawings, constituent elements having substantially the same functional configuration are designated by the same reference numerals, and a duplicate description will be omitted.

In addition, in the present specification and the drawings, a plurality of constituent elements having substantially the same functional configuration may be distinguished by attaching different alphabets after the same reference numeral. For example, a plurality of configurations having substantially the same functional configuration or logical significance are distinguished as needed, such as a button image 63A and a button image 63B. However, if it is not particularly necessary to distinguish each of the plurality of constituent elements having substantially the same functional configuration, only the same reference numeral is given. For example, when it is not necessary to distinguish between the button image 63A and the button image 63B, they are simply referred to as the button image 63.

The description will be given in the following order.
1. 1. First embodiment 1.1 Schematic configuration of information processing system 1.2 Configuration of terminal device 1.2.1 Hardware configuration 1.2.2 Functional configuration 1.2.3 Software configuration 1.3 Information management Server configuration 1.3.1 Hardware configuration 1.3.2 Functional configuration 1.4 Speech recognition server configuration 1.4.1 Hardware configuration 1.4.2 Functional configuration 1.5 Process flow 2. Second embodiment 2.1 Configuration of information management server 2.1.1 Functional configuration 2.2 Process flow 3. Third Embodiment 3.1 Configuration of Terminal Device 3.1.1 Functional Configuration 3.2 Configuration of Information Management Server 3.2.1 Functional Configuration 3.3 Process Flow 4. Supplement

<1. First Embodiment>
<1.1 Schematic configuration of information processing system>
The first embodiment of the present invention relates to an information processing system used in a conversation between a plurality of persons existing in a plurality of remote places in a distributed environment. First, with reference to FIG. 1, a schematic configuration of an information processing system according to the first embodiment will be described. FIG. 1 is an explanatory diagram showing an example of a schematic configuration of an information processing system according to the present embodiment.

As shown in FIG. 1, the information processing system according to the present embodiment is used, for example, over a plurality of bases. In the example of FIG. 1, the information processing system is used across a center office 10 and a satellite office 20 (or home offices 20). For example, the center office 10 is a relatively large office, and the satellite office 20 (or home office and 20) is a relatively small office.

The information processing system includes, in the center office 10, a camera 11, a microphone 13, a sensor 15, a media distribution server 17, an information management server (information processing server) 200, a voice recognition server 201, and a LAN (Local Area network) 19. Further, the information processing system includes a terminal device 100, a display 21, and a LAN 23 in the satellite office 20 (or home office and 20). Further, the information processing system further includes an external network 30 and a PBX (Private Branch eXchange) 40.

(Camera 11)
The camera 11 captures an image of the area in the direction in which the camera 11 is facing (that is, the capturing direction). One or more cameras 11 are installed in the center office 10. Then, each installed camera 11 can capture an image of a part or the whole of the center office 10 from each installation position. As can be seen from FIG. 1, in the information processing system according to the present embodiment, the plurality of cameras 11 installed in the center office 10 can image the center office 10 from various positions. In addition, in the present embodiment, the captured image generated by the camera 11 may be a still image or a moving image (that is, a video), and is not particularly limited. Further, the camera 11 can automatically change the imaging direction, for example. Further, the camera 11 has, for example, a zoom function. The zoom function may be an optical zoom function or a digital zoom function, and is not particularly limited.

Further, the camera 11 may be able to change the position of the camera 11. For example, the camera 11 may be configured to be movable by a dolly (not shown). In other words, the camera 11 may be configured to be movable along a rail (not shown). In this case, the camera 11 may move along the rail by controlling a motor (not shown) for moving the camera 11 along the rail. As a result, even if the number of cameras 11 installed in the center office 10 is one, it is possible for the cameras 11 to acquire captured images captured from different positions.

If the camera 11 can change its position, the zoom function may be a zoom function by changing the position of the camera 11. For example, the zoom function may be a dolly zoom function. Specifically, zooming in may be performed by moving the camera 11 toward the subject, and zooming out by moving the camera 11 in a direction in which the camera 11 moves away from the subject. Note that the dolly zoom function need not be a precisely adjusted zoom, such as an optical zoom or a digital zoom. In this case, for example, in zooming in, it is only necessary to obtain a captured image in which the subject is larger, and in zooming out, it is sufficient to obtain a captured image in which the subject is smaller.

(Microphone 13)
The microphone 13 collects sounds around the microphone 13. In the center office 10, for example, one or a plurality of microphones 13 are installed. For example, each of the installed microphones 13 collects a sound around the installation position in the center office 10. As described above, in the information processing system according to the present embodiment, the plurality of microphones 13 installed in the center office 10 collect sounds at various positions in the center office 10.

(Sensor 15)
The sensor 15 may include various types of sensors that detect various types. In the center office 10, for example, one or more sensors 15 are installed. The sensor 15 may be, for example, a seat sensor that determines whether a person is in the seat. The seat sensor is installed in each seat and detects whether or not the seat is pressed to determine whether or not a person is sitting in each seat. Further, the sensor 15 may be, for example, a vibration sensor installed in a seat or the like, and determines whether or not a person is seated in the seat by detecting the vibration caused by the person sitting in the seat. To do. Further, the sensor 15 may be, for example, a motion sensor installed under a desk or the like, and the motion sensor changes in infrared rays, ultrasonic waves, visible light, etc. due to a person sitting in the seat. Is detected, it is determined whether or not a person is sitting in the corresponding seat.

(Media distribution server 17)
The media delivery server 17 delivers media (for example, audio, video, etc.) to the terminal device 100 or the like in response to the request.

(Information management server 200)
The information management server 200 manages various information used in the information processing system according to this embodiment. That is, the information management server 200 stores the various information and updates the various information in a timely manner. For example, the information management server 200 manages the parameters related to the camera 11, the microphone 13, and the sensor 15 described above. Specifically, for example, the information management server 200 stores and updates information such as the installation position of the camera 11, the imaging direction (for example, the direction perpendicular to the lens of the camera 11), the zoom ratio, etc., as the parameters of the camera 11. To do.

Further, for example, the information management server 200 generates and manages data in the three-dimensional virtual space corresponding to the real space. The three-dimensional virtual space means, for example, a three-dimensional virtual space imitating the center office 10. Further, an object is placed in the three-dimensional virtual space. For example, the object corresponds to a person, and the object is arranged at the three-dimensional virtual position in the three-dimensional virtual space corresponding to the position of each seat of the center office 10. That is, when the person is sitting on the seat, the object is arranged at the three-dimensional virtual position corresponding to the position where the person is likely to exist. As an example, the object has a cylindrical shape. The three-dimensional virtual space and the object will be described later.

Furthermore, for example, the information management server 200 adds the data of the corresponding communication ID (identification) to the voice data acquired from the microphone 13 or the sound collection unit 140 of the terminal device 100 and transmits the voice data to the voice recognition server 201. .. Further, for example, the information management server 200 receives and manages word/phrase data (for example, data about words/phrases extracted from the above-mentioned voice data) as a recognition result from the voice recognition server 201. The voice data, the communication ID data, and the word/phrase data will be described later. Further, the processing performed by the information management server 200, such as processing related to acquisition and storage of voice data, or a part of the processing is performed by the media distribution server 17, other devices such as the terminal device 100, instead of the information management server 200. May be done.

(Voice recognition server 201)
The voice recognition server 201 has a large-scale word list data built therein, receives voice data acquired by the terminal device 100 or the microphone 13 via, for example, the information management server 200, and performs voice recognition processing. The voice recognition process is a process of extracting a phrase included in the received voice data using the phrase list described above. Then, the voice recognition server 201 transmits the data, which is the result of the voice recognition process, to the information management server 200 described above. The voice recognition server 201 may be realized as a virtual server or application software by the information management server 200, the media distribution server 17, the other server device such as the PBX 40 described later, or the like.

(LAN 19, external network 30)
The LAN 19 is a network that connects the respective devices in the center office 10. Further, the LAN 19 connects each device inside the center office 10 and a device outside the center office 10 via the external network 30. The LAN 19 and the external network 30 can be wired or wireless, and for example, the Internet, IP-VPN (Internet Protocol-Virtual Private Network), leased line, or WAN (Wide Area Network), any communication such as infrared communication. It consists of a network.

(Terminal device 100)
The terminal device 100 is used by a user. For example, the terminal device 100 provides the user with a function for performing communication such as telephone and email. The terminal device 100 is, for example, a tablet terminal. Note that the terminal device 100 may be another device having a display function and a communication function, such as a smartphone, a PC (Personal Computer), or a telephone with a display, instead of the tablet terminal.

(Display 21)
The display 21 displays various screens. For example, the display 21 displays a screen including a captured image acquired through the camera 11. As a result, many persons including the user of the terminal device 100 can see the state of the center office 10 via the display 21. Further, the display 21 may output any sound, for example. Specifically, the display 21 may output the sound collected by the microphone 13. As a result, a large number of people including the user of the terminal device 100 can hear the sound in the center office 10 via the display 21.

(LAN23)
The LAN 23 is a network that connects the devices in the satellite office 20 (or the home office 20). In addition, the LAN 23 connects each device inside the satellite office 20 and a device outside the satellite office 20 via the external network 30. The LAN 23 can also be wired or wireless, and is composed of an arbitrary communication network such as a dedicated line and infrared communication.

(PBX40)
The PBX 40 enables communication between devices via the external network 30. The PBX 40 is, for example, H.264. 323 or SIP (Session Initiation Protocol). Specifically, for example, the PBX 40 stores communication identification information (for example, a telephone number) and an IP (Internet Protocol) address in association with each other. Then, the PBX 40 converts the identification information for communication into an IP address in response to the request, and provides the IP address to the request source. The PBX 40 may be connected to the above-mentioned LAN 19 or LAN 23.

<1.2 Configuration of terminal device>
Subsequently, an example of the configuration of the terminal device 100 according to the present embodiment will be described with reference to FIGS. 2 to 17. As described above, the terminal device 100 is a device that provides a user with a function for performing communication.

<1.2.1 Hardware configuration>
First, an example of the hardware configuration of the terminal device 100 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram showing an example of the hardware configuration of the terminal device 100 according to this embodiment. 2, the terminal device 100 includes a CPU (central processing unit) 801, a ROM (Read Only Memory) 803, a RAM (Random Access Memory) 805, a bus 807, a storage device 809, a communication interface 811, a camera 813, and a microphone. 815, a speaker 817, and a touch panel 820.

(CPU 801, ROM 803, RAM 805)
The CPU 801 executes various processes in the terminal device 100. Further, the ROM 803 stores a program and data for causing the CPU 801 to execute the process in the terminal device 100. Further, the RAM 805 temporarily stores programs and data when the processing of the CPU 801 is executed.

(Bus 807)
The bus 807 connects the CPU 801, the ROM 803, and the RAM 805 to each other. A storage device 809, a communication interface 811, a camera 813, a microphone 815, a speaker 817, and a touch panel 820, which will be described later, are further connected to the bus 807. The bus 807 includes, for example, a plurality of types of buses. Specifically, the bus 807 may include a high-speed bus that connects the CPU 801, the ROM 803, and the RAM 805, and one or more other buses that are slower than the high-speed bus.

(Memory device 809)
The storage device 809 stores data to be temporarily or permanently stored in the terminal device 100. The storage device 809 may be, for example, a magnetic storage device such as a hard disk, or an EEPROM (Electrically Erasable and Programmable Read Only Memory), a flash memory (FLASH Memory Memory), or a MRAM (MRAM). Non-volatile memory such as FeRAM (Ferroelectric Random Access Memory) and PRAM (Phase change Random Access Memory) may be used.

(Communication interface 811)
The communication interface 811 is a communication unit of the terminal device 100, and communicates with an external device via the network (or directly). The communication interface 811 may be an interface for wireless communication, and in this case, may include, for example, a communication antenna, an RF (Radio Frequency) circuit, and other circuits for communication processing. Further, the communication interface 811 may be an interface for wired communication, and in this case, for example, may include a LAN terminal, a transmission circuit, and other circuits for communication processing.

(Camera 813)
The camera 813 images a subject. The camera 813 includes, for example, an optical system, an image sensor, and an image processing circuit.

(Microphone 815)
The microphone 815 collects ambient sounds. The microphone 815 converts ambient sound into an electric signal and converts the electric signal into digital data.

(Speaker 817)
The speaker 817 outputs sound. The speaker 817 converts digital data into an electric signal and converts the electric signal into voice.

(Touch panel 820)
The touch panel 820 includes a touch detection surface 821 and a display surface 823.

The touch detection surface 821 detects a touch position on the touch panel 820. More specifically, for example, when the user touches the touch panel 820, the touch detection surface 821 senses the touch, generates an electrical signal according to the position of the touch, and then applies the electrical signal to the touch position. Convert to information. The touch detection surface 821 can correspond to an arbitrary touch detection method such as a capacitance method, a resistance film method, or an optical method.

The display surface 823 displays an output image (that is, a display screen) from the terminal device 100. The display surface 823 can be realized using, for example, liquid crystal, organic EL (Organic Light Emitting Diode: OLED), CRT (Cathode Ray Tube), or the like.

<1.2.2 Functional configuration>
Next, an example of the functional configuration of the terminal device 100 according to the present embodiment will be described. FIG. 3 is a block diagram showing an example of the functional configuration of the terminal device 100 according to this embodiment. Referring to FIG. 3, the terminal device 100 includes a communication unit 110, an input unit 120, an imaging unit 130, a sound collection unit 140, a display unit 150, a sound output unit 160, a storage unit 170, and a control unit 180.

(Communication unit 110)
The communication unit 110 communicates with other devices. For example, the communication unit 110 is connected to the above-described LAN 23 and communicates with each device in the satellite office 20. Further, the communication unit 110 communicates with each device in the center office 10 via the external network 30 and the LAN 19 described above. Specifically, for example, the communication unit 110 communicates with the camera 11, the microphone 13, the sensor 15, the media distribution server 17, the information management server 200, and the voice recognition server 201. The communication unit 110 can be realized by, for example, the communication interface 811.

(Input unit 120)
The input unit 120 receives an input from the user of the terminal device 100. Then, the input unit 120 provides the input result to the control unit 180 described later. For example, the input unit 120 detects the position designated by the user on the display screen. More specifically, the input unit 120 is realized by the touch detection surface 821 and detects the touch position on the touch panel 820. Then, the input unit 120 provides the detected touch position to the control unit 180.

(Imaging unit 130)
The image capturing unit 130 captures an image of a subject. For example, the image capturing unit 130 captures an image of the area in the front direction of the terminal device 100. In this case, the image capturing unit 130 can capture the user of the terminal device 100. The imaging unit 130 provides the imaging result (that is, the captured image) to the control unit 180. The image capturing unit 130 may be realized by the camera 813, for example.

(Sound collection unit 140)
The sound collection unit 140 collects sounds around the terminal device 100. For example, the sound collection unit 140 can collect the voice of the user of the terminal device 100. The sound collection unit 140 provides the sound collection result (sound data) to the control unit 180. The sound collection unit 140 can be realized by, for example, a microphone 815.

(Display unit 150)
The display unit 150 displays an output image (display screen). The display unit 150 displays a display screen under the control of the control unit 180. The display unit 150 may be realized by the display surface 823, for example.

(Voice output unit 160)
The voice output unit 160 outputs a voice. The voice output unit 160 outputs a voice under the control of the control unit 180. The audio output unit 160 can be realized by, for example, the speaker 817.

(Storage unit 170)
The storage unit 170 stores programs and data for the operation of the terminal device 100. For example, the storage unit 170 stores the data of the three-dimensional virtual space corresponding to the real space. Specifically, for example, the information management server 200 stores the data of the three-dimensional virtual space corresponding to the center office 10, and the control unit 180 transmits the data of the three-dimensional virtual space via the communication unit 110. To get Then, the storage unit 170 stores the acquired data of the three-dimensional virtual space. The storage unit 170 can be realized by, for example, the storage device 809.

(Control unit 180)
The control unit 180 provides various functions of the terminal device 100. The control unit 180 includes a real space information providing unit 181, a voice output control unit 182, a position acquisition unit 183, an object selection unit 185, an ID acquisition unit 187, a telephone unit 189, a conversation object selection unit 191, a COMM link control unit 193, and It includes a COMM word control unit (language phrase object control unit) 195. The control unit 180 can be realized by, for example, the CPU 801, the ROM 803, and the RAM 805. The functional units of the control unit 180 will be described below.

(Real space information providing unit 181)
The real space information providing unit 181 provides real space information to the user of the terminal device 100. Specifically, the real space information providing unit 181 causes the display unit 150 to display the display screen of the captured image of the real space. More specifically, for example, the captured image is a captured image acquired through the camera 11 in the real space (center office 10). The captured image may be a captured image acquired by the camera 11 or may be a captured image generated by processing the captured image acquired by the camera 11. The display screen is a screen that includes the captured image in part or in whole. That is, the real space information providing unit 181 acquires the captured image of the camera 11 via the communication unit 110. Then, the real space information providing unit 181 generates a display screen including the captured image and causes the display unit 150 to display the display screen.

Further, for example, the captured image may be a captured image acquired through a selected one of the plurality of imaging devices in the real space. More specifically, for example, the captured image may be a captured image acquired through one selected camera 11 of the plurality of cameras 11 arranged in the center office 10. Note that a specific method of how the user selects the camera 11 will be described later. In this way, since the user can select the camera 11, the user can see the captured image from the desired position.

Further, for example, the display screen includes a captured image according to the display mode. More specifically, for example, in the first display mode, the display screen includes a first captured image obtained by capturing a first region of the real space (for example, in the center office 10), and the second display The mode includes the second captured image in which the second region in the real space, which is narrower than the first region, is captured. That is, the real space information providing unit 181 displays the first captured image on the display unit 150 in the first display mode, and displays the second captured image on the display unit 150 in the second display mode. Let

More specifically, for example, the first captured image is a captured image corresponding to the first zoom ratio. Then, the second captured image is a captured image corresponding to a second zoom rate larger than the first zoom rate. For example, the real space information providing unit 181 makes a request regarding the zoom (optical zoom, digital zoom, or zoom by changing the position of the imaging device (for example, dolly zoom)) to the camera 11 via the communication unit 110. Thus, the captured image corresponding to the first zoom ratio or the captured image corresponding to the second zoom ratio is acquired. Alternatively, the real space information providing unit 181 may acquire the captured image corresponding to the first zoom ratio or the captured image corresponding to the second zoom ratio by digital zooming the captured image of the camera 11.

The zoom ratio here does not have to be a delicate value such as 1.5 times or 2 times, and may be any value that directly or indirectly indicates the size of the size of the subject in the captured image. .. For example, when zooming by changing the position of the camera 11 (for example, zooming in and out by dolly) is used, the zoom rate is not a precise value such as 1.5 times or 2 times, but the size of the subject. Directly indicating the degree of the subject (for example, a parameter indicating the approximate size of the subject) or indirectly indicating the degree of the subject (eg, the position of the camera 11 on the rail). May be The captured image corresponding to the first zoom ratio is a captured image in which the subject is smaller, and the captured image corresponding to the second zoom ratio that is larger than the first zoom ratio is the subject in a larger size. Any captured image may be used.

As an example, the display screen includes a bird's-eye view captured image generated by the camera 11 in imaging at a zoom rate of X times (for example, X=1) in the bird's-eye view mode, and a zoom rate of Y times in the close-up mode ( (Y>X) includes a close-up captured image generated by the camera 11. That is, the bird's-eye view captured image is a captured image in which a wider area in the center office 10 is captured, and the close-up captured image is a captured image in which a smaller area in the center office 10 is captured. Hereinafter, specific examples of these captured images will be described with reference to FIGS. 4 and 5.

-Display screen displayed in overhead view mode-
FIG. 4 is an explanatory diagram illustrating an example of a display screen displayed in the overhead view mode according to the present embodiment. More specifically, FIG. 4 shows a display screen 60 in the overhead view mode. The display screen 60 includes a bird's-eye view captured image 61, a button image 63, a presence icon 65, a balloon image 67, and a map image 69.

The bird's-eye view captured image 61 is, for example, a captured image acquired by the camera 11 at a zoom ratio of X times. As an example, X=1. That is, the bird's-eye view captured image 61 is a captured image of the camera 11 without zooming. Further, for example, when the user specifies the position of the bird's-eye view captured image 61, the real space information providing unit 181 switches the display mode from the bird's-eye view mode to the proximity mode. More specifically, for example, when the user touches a desired position in the bird's-eye view captured image 61 and a touch position corresponding to the bird's-eye view captured image 61 is detected, the real space information providing unit 181 looks down the display mode. Switch from mode to proximity mode.

The button image 63 is an image for selecting another camera 11. For example, when the user specifies the position of the button image 63, the real space information providing unit 181 acquires the bird's-eye view captured image of another camera 11, and causes the display unit 150 to display the bird's-eye view captured image. More specifically, for example, when the user touches the position of the button image 63 and the touch position corresponding to the button image 63 is detected, the real space information providing unit 181 displays the bird's eye captured image of another camera 11. The acquired captured bird's-eye view image is displayed on the display unit 150. Specifically, in the example of FIG. 4, when the position of the button image 63A is designated by the user, the camera 11 located on the left side of the current camera 11 is selected. When the position of the button image 63B is designated by the user, the camera 11 located on the right side of the current camera 11 is selected. Then, the real space information providing unit 181 acquires the bird's-eye view captured image of the selected camera 11, and causes the display unit 150 to display the bird's-eye view captured image.

Further, the presence icon 65 is, for example, an icon indicating the busy degree of a person shown in the bird's-eye view captured image 61. More specifically, the presence icon 65 changes its color according to the busyness of a person. As an example, the presence icon indicates that the busyness degree is high when the color is red, indicates that the busyness degree is normal when the color is yellow, and indicates that the busyness degree is low when the color is blue. As will be described later, it is possible to display such an icon because it is possible to know where in the bird's-eye view captured image 61 the person should appear. Regarding the busyness of a person, the operating status of the terminal device 100 such as a PC corresponding to the person (the person is busy if high frequency keystrokes per certain time or long-term continuous use of business applications are performed). It may be determined according to (determination, etc.). Further, in the present embodiment, the presence icon 65 is not limited to the above-described form. For example, when the presence icon 65 is a white circle icon, it means that the corresponding person is present. And a black circle icon may indicate that the corresponding person is absent.

As described above, the display screen 60 includes, for example, information related to a person shown in the bird's-eye view captured image 61 (hereinafter, referred to as “person-related information”). Then, the person-related information includes, for example, state information indicating the state of the person. As described above, the presence icon 65 can be cited as an example of the state information. In addition, the person-related information may include a state history indicating the state of the person at two or more time points. Further, as an example, the state history information may include a history of the busyness of the person shown in the bird's-eye view captured image 61. That is, the history of the busyness of a person may be displayed on the display screen 60. The real space information providing unit 181 acquires the person-related information or information necessary for displaying the person-related information from the information management server 200 via the communication unit 110, for example.

In the present embodiment, such person-related information allows the user to more accurately understand the situation in which the person is placed. In addition, the above-mentioned status information allows the user to more accurately determine whether or not to contact the person corresponding to the status information. Further, the state history allows the user to grasp not only the instantaneous state of the person corresponding to the state history but also the state of the person in a certain period, so the user may contact the person. It can be judged more accurately.

Further, the balloon image 67 is an image including character information presented by the person shown in the bird's-eye view captured image 61. The balloon image 67 is also an example of the person-related information.

The map image 69 is an image showing a map of the center office 10. The map image 69 further shows the camera 11 being used by the icon 31. If only one or a small number of cameras 11 are installed in the center office 10, the map image 69 may be omitted.

-Display screen displayed in proximity mode-
FIG. 5 is an explanatory diagram for explaining an example of a display screen displayed in the proximity mode according to the present embodiment. Specifically, FIG. 5 shows a display screen 70 displayed in the proximity mode. The display screen 70 includes a close-up captured image 71, a button image 73, and a map image 75.

The close-up captured image 71 is, for example, a captured image of the camera 11 at a zoom ratio of Y (Y>X). As an example, Y=1.5. That is, the bird's-eye view captured image 61 is a captured image of the camera 11 in the 1.5× zoom imaging.

The button image 73 is an image for switching the display mode from the close-up mode to the bird's eye view mode. For example, when the user specifies the position of the button image 73, the real space information providing unit 181 switches the display mode from the proximity mode to the bird's eye view mode. More specifically, for example, when the user touches the button image 73 and the touch position corresponding to the button image 73 is detected, the real space information providing unit 181 switches the display mode from the proximity mode to the bird's eye view mode.

Further, the map image 75 is an image showing a map of the center office 10, like the map image 69 in the overhead view mode. The map image 75 further shows the camera 11 being used. For example, in the proximity mode, the icon 31 of the camera being used in the map image 75 is displayed at a position closer to the shooting target in order to symbolically indicate zooming. Similar to the map image 69 in the overhead view mode, the map image 75 may be omitted when only one or a small number of cameras 11 are installed in the center office 10.

The display screen 70 displayed in the proximity mode may also include the person-related information such as the presence icon 65 and the balloon image 67.

By switching the display mode as described above, the captured image in which a wider area is captured is displayed, so that the user can see the overall situation in the real space and easily find a specific person. be able to. Then, by displaying the captured image in which a narrower area is captured, the user can more easily specify the position of a specific person. In addition, in this embodiment, the user can switch the display mode by an easy operation. In addition, since the captured image in which the wider area is captured and the captured image in which the narrower area is captured are captured images having different zoom rates, the user can intuitively easily understand the relationship between these captured images. Can be grasped. Therefore, even if the display mode is switched, the user can easily find a specific person and specify the position of the specific person.

-Other real space information-
As described above, the real space information providing unit 181 provides the visual information of the real space by displaying the display screen of the captured image of the real space on the display unit 150. Furthermore, the real space information providing unit 181 may also provide auditory information in the real space. That is, the real space information providing unit 181 may cause the sound output unit 160 to output the sound of the sound data obtained by the sound collection in the real space. For example, the real space information providing unit 181 selects the microphone 13 close to the camera 11 selected by the user. Then, the real space information providing unit 181 acquires the voice data obtained by the sound collection in the center office 10 from the microphone 13 via the communication unit 110. Then, the real space information providing unit 181 may cause the audio output unit 160 to output the audio of the acquired audio data.

(Voice output control unit 182)
The audio output control unit 182 controls the audio output by the audio output unit 160 based on the data acquired by the control unit 180 via the communication unit 110. Specifically, the audio output control unit 182 controls the volume of the audio output from the audio output unit 160 based on the above data.

(Position acquisition unit 183)
The position acquisition unit 183 acquires the position in the captured image specified by the user on the display screen of the captured image in the real space. More specifically, for example, when the input unit 120 detects a position in the captured image specified by the user on the display screen of the captured image in the real space, the position acquisition unit 183 acquires the position. Then, the position acquisition unit 183 provides the position in the captured image to the object selection unit 185.

For example, when the position acquisition unit 183 acquires any position in the close-up captured image 71 of FIG. 5, the position acquisition unit 183 provides the position to the object selection unit 185. Hereinafter, the acquisition and provision of the position by the position acquisition unit 183 will be specifically described with reference to FIGS. 6 and 7.

FIG. 6 is an explanatory diagram for describing a first example of the position in the close-up captured image 71 designated by the user. FIG. 6 shows the close-up captured image 71 and the user's hand 3 included in the display screen in the close-up mode. Further, FIG. 6 shows a person image 77 of a person shown in the close-up captured image 71. Then, the user touches the position of the person image 77 with the hand 3 to specify the position of the person image 77 of the close-up captured image 71. In this case, the input unit 120 detects the position of the person image 77 of the close-up captured image 71, and the position acquisition unit 183 acquires the position. Then, the input unit 120 provides the acquired position to the object selection unit 185.

FIG. 7 is an explanatory diagram for explaining a second example of the position in the close-up captured image 71 designated by the user. FIG. 7 shows person images 77A and 77B shown in the close-up captured image 71. Then, the user touches the position of the person image 77A with the hand 3 to specify the position of the person image 77A of the close-up captured image 71. In this case, the input unit 120 detects the position of the person image 77A of the close-up captured image 71, and the position acquisition unit 183 acquires the position. Then, the input unit 120 provides the acquired position to the object selection unit 185.

(Object selection unit 185)
The object selection unit 185 selects an object arranged in the three-dimensional virtual space corresponding to the real space based on the position in the captured image that is acquired. For example, when the position acquisition unit 183 acquires the position in the captured image specified by the user on the display screen of the captured image in the real space, the object selection unit 185 corresponds to the real space based on the position. Select an object placed in a three-dimensional virtual space. Specifically, the object is an object arranged at a three-dimensional virtual position in the three-dimensional virtual space corresponding to the position in the captured image. Further, for example, the captured image is a captured image in the second mode (for example, the proximity mode) included in the display screen. For example, when the position acquisition unit 183 acquires the position in the close-up captured image 71 generated by the camera 11, the object selection unit 185 causes the object selection unit 185 to select one of the objects arranged in the three-dimensional virtual space corresponding to the center office 10. Select the object corresponding to the above position. The object selection unit 185 acquires, for example, the data of the three-dimensional virtual space corresponding to the center office 10 from the storage unit 170.

-Example when one object is placed-
Hereinafter, a specific example of the three-dimensional virtual space 90 in the case where one object 91 is arranged in the three-dimensional virtual space 90 will be described with reference to FIGS. 8 and 9. FIG. 8 is an explanatory diagram illustrating a first example of the three-dimensional virtual space 90 corresponding to the center office 10. In FIG. 8, a three-dimensional virtual space 90 corresponding to the center office 10 is shown. An object 91 is arranged in the three-dimensional virtual space 90. The object 91 corresponds to a person (for example, Mr. A). Then, the object 91 is arranged at a three-dimensional virtual position corresponding to the position of the seat of the person (for example, Mr. A) in the center office 10. That is, when the person is sitting on the seat, the object 91 is arranged at the three-dimensional virtual position corresponding to the position where the person is likely to exist. In this example, the object 91 is a columnar object. The cylindrical object 91 is, for example, a cylindrical object having a radius R and a height H. The radius R and the height H are predetermined, for example. The data of the three-dimensional virtual space 90 also includes information related to each camera 11. For example, the information related to each camera 11 includes a three-dimensional virtual position corresponding to the installation position of each camera 11, an imaging direction (for example, a direction perpendicular to the lens of the camera), an angle of view, and the like.

FIG. 9 is an explanatory diagram for explaining an example of selection of the object 91 arranged in the three-dimensional virtual space 90 shown in FIG. In FIG. 9, the positional relationship in the horizontal plane in the three-dimensional virtual space 90 is shown for easy understanding. Specifically, in FIG. 9, an object 91 arranged in a three-dimensional virtual space 90, a three-dimensional virtual position (hereinafter, referred to as “virtual camera position”) O corresponding to the installation position of the camera 11 used for imaging, An axis y corresponding to the imaging direction of the camera 11 (for example, a direction perpendicular to the camera lens) and an axis x orthogonal to the axis y are shown. In the example of FIG. 9, for easy understanding, the camera 11 is assumed to be installed such that the imaging direction of the camera 11 is parallel to the horizontal plane.

Further, in FIG. 9, the angle of view θ of the camera 11 is also shown. Further, in FIG. 9, an imaginary plane 93 which is perpendicular to the axis y corresponding to the imaging direction of the camera 11 and has a width corresponding to the angle of view θ is shown. The virtual surface 93 is separated from the virtual camera position O by the distance I. The virtual surface 93 is a quadrangular surface and has the same aspect ratio as the captured image. That is, the virtual surface 93 is a surface corresponding to the captured image.

The object selection unit 185 converts the position in the captured image designated by the user into the three-dimensional virtual position A, as shown in FIG. 6, for example. Then, the object selecting unit 185 identifies an object that intersects with a straight line passing through the virtual camera position O and the three-dimensional virtual position A. That is, the object selection unit 185 identifies the object 91. Then, the object selection unit 185 selects the object 91. In the example of FIG. 9, for example, the position in the captured image that is converted into the three-dimensional virtual position between the three-dimensional virtual position B and the three-dimensional virtual position D on the virtual surface 93 is the user in the captured image. Object 91, the object 91 is selected. Note that such a position is generally a position where a person corresponding to the object 91 is shown in the captured image. The distance I is determined so that the virtual plane 93 is located between the virtual camera position O and the object 91. As an example, the distance I is the focal length of the camera 11, but naturally it is not limited to this in the present embodiment.

Further, in the above, the method of selecting the object 91 by paying attention to the horizontal plane of the three-dimensional virtual space 90 has been described, but according to the method, of course, processing in the vertical direction (for example, the z axis) is added. Even if it is, it is possible to convert the position in the captured image into a three-dimensional virtual position and specify the object 91 from the three-dimensional virtual position. Further, in the example described above, the object 91 corresponding to the position in the captured image is specified by converting the position in the captured image into a three-dimensional virtual position, but in the present embodiment, the position in the captured image is determined. The method of identifying the object 91 corresponding to is not limited to this.

As an example, the object selection unit 185 may project the object 91 on the virtual surface 93 with the virtual camera position O as the origin, and convert the projection range of the object 91 into a range within the captured image. Then, when the position in the captured image designated by the user is included in the range, the object selection unit 185 may select the object 91.

As another example, a rendering image is generated by rendering the three-dimensional virtual space 90 using the virtual camera position O, the axis y, and the angle of view θ, and the object 91 in the rendering image is captured. The range in the captured image corresponding to the object 91 may be obtained from the range. Then, when the position in the captured image designated by the user is included in the range, the object selection unit 185 may select the object 91.

-Selection of Object Considering State Also, for example, the object 91 arranged in the three-dimensional virtual space 90 corresponds to state information indicating any one of a plurality of states. For example, the object 91 corresponds to a person. Then, the object 91 corresponds to state information (hereinafter, referred to as “presence/absence information”) indicating whether or not the person is sitting on the seat. The presence/absence information indicates a state where a person is sitting on the seat or a state where the person is not sitting on the seat. The object selection unit 185 acquires, for example, the determination result of whether or not the person corresponding to the object 91 is sitting on the seat from the sensor 15 via the communication unit 110. Then, the determination result becomes the presence/absence information.

Then, for example, the object selecting unit 185 selects the object 91 arranged in the three-dimensional virtual space 90 and corresponding to the status information indicating a predetermined status of the plurality of statuses. .. More specifically, the object selection unit 185 selects the object 91 arranged in the three-dimensional virtual space 90 and corresponding to the state information indicating the state in which a person is sitting on the seat. That is, when the sensor 15 determines that the person is sitting on the seat, the object selecting unit 185 can select the object 91 corresponding to the person, but if the sensor 15 does not place the person on the seat. If determined, the object 91 corresponding to the person is not selected.

As described above, in the present embodiment, the object 91 is selected according to the state of the person, so that it is possible to avoid selecting the object 91 that should not be selected. For example, in the present embodiment, it can be avoided that the object 91 corresponding to the person is selected when there is no person.

-Example in which two objects are arranged Further, two or more objects 91 may be arranged in the three-dimensional virtual space 90. Hereinafter, a specific example of the three-dimensional virtual space 90 when two objects 91 are arranged will be described with reference to FIG. FIG. 10 is an explanatory diagram illustrating a second example of the three-dimensional virtual space 90 corresponding to the center office 10. FIG. 10 shows a three-dimensional virtual space 90 corresponding to the center office 10. In addition, an object 91A and an object 91B are arranged in the three-dimensional virtual space 90. The object 91A corresponds to a certain person (for example, Mr. A), and is arranged at a three-dimensional virtual position corresponding to the position of the seat of the certain person in the center office 10. Further, the object 91B corresponds to a certain person (for example, Mr. B), and is arranged at a three-dimensional virtual position corresponding to the position of the seat of the certain person in the center office 10. Similar to the example of FIG. 8, the object 91 is a cylindrical object with a radius R and a height H.

When two or more objects 91 are arranged in the three-dimensional virtual space 90 in this way, the plurality of objects 91 may correspond to positions in the captured image designated by the user. For example, in the example of FIG. 10, both the object 91A and the object 91B may correspond to the position in the captured image designated by the user. As an example, when two people can appear in the captured image as shown in FIG. 7, when the user specifies a position where the two persons overlap in the captured image, the two persons are selected. Two corresponding objects 91 may correspond to the above positions.

Therefore, in such a case (that is, in the case where the plurality of objects 91 correspond to the positions of the captured image designated by the user), the object selecting unit 185 causes the object selecting unit 185 to select one of the plurality of objects 91. Select the object 91. That is, when there are a plurality of objects 91 respectively arranged at the three-dimensional virtual positions in the three-dimensional virtual space 90 corresponding to the positions in the captured image designated by the user, the object selection unit 185 determines that the plurality of objects 91. Any one of the objects 91 is selected.

For example, the captured image is generated through an image capturing device in the real space. Then, the object selecting unit 185 selects, from the plurality of objects 91, the object 91 closer to the three-dimensional virtual position in the three-dimensional virtual space 90 corresponding to the imaging device. More specifically, for example, the captured image is acquired by the camera 11 in the center office 10. Then, when there are a plurality of objects 91 corresponding to the positions in the captured image designated by the user, the object selection unit 185, the three-dimensional virtual position corresponding to the installation position of the camera 11 (that is, virtual camera position O). The object 91 closer to is selected. Hereinafter, a specific example of this point will be described with reference to FIGS. 10 and 11.

FIG. 11 is an explanatory diagram for explaining an example of selection of the object 91 arranged in the three-dimensional virtual space 90 shown in FIG. In FIG. 11, for easy understanding, the positional relationship in the horizontal plane in the three-dimensional virtual space 90 is shown. Specifically, in FIG. 11, an object 91A and an object 91B arranged in the three-dimensional virtual space 90 are shown. Further, in FIG. 11, as in FIG. 9, the virtual camera position O, the axis y, the axis x, the angle of view θ, and the virtual surface 93 are shown. In the example of FIG. 11 as well, in order to facilitate understanding, it is assumed that the camera 11 is installed such that the imaging direction of the camera 11 is parallel to the horizontal plane, as in FIG. 9.

For example, as shown in FIG. 7, the user specifies the position in the captured image. In this case, according to the method described with reference to FIG. 9, the position in the captured image is converted into a three-dimensional virtual position between the three-dimensional virtual position B′ and the three-dimensional virtual position D. In this case, both the object 91A and the object 91B are specified as objects corresponding to the above positions in the captured image. Then, the object selecting unit 185 selects the object 91A that is closer to the virtual camera position O from the objects 91A and 91B.

Note that when the position in the captured image designated by the user is converted into a three-dimensional virtual position between the three-dimensional virtual position B and the three-dimensional virtual position B′, the object 91A is the above-mentioned captured image. Identified and selected as the object corresponding to the position within. Further, when the position in the captured image designated by the user is converted into the three-dimensional virtual position between the three-dimensional virtual position D and the three-dimensional virtual position D′, the object 91B becomes the captured image. Identified and selected as the object corresponding to the position within.

Thus, in the present embodiment, when one object 91 is selected from the plurality of objects 91, an error (a plurality of communication IDs) occurs in the subsequent processing due to the selection of the plurality of objects 91. Error can be avoided. In addition, by selecting the object 91 close to the virtual camera position corresponding to the camera 11 among the plurality of objects 91, for example, even when people are overlapped in the captured image, the person shown in the foreground is displayed. The object 91 corresponding to is selected. Therefore, the user can select the object 91 corresponding to the person intended by the user.

As described above, the object selection unit 185 selects the object 91. Then, the object selection unit 185 provides the ID acquisition unit 187 with the identification information of the selected object 91 (hereinafter referred to as “object ID”). The object ID may be identification information of a person corresponding to the selected object 91, or may be a simple number attached to the selected object 91 (for example, composed of numbers and letters). May be.

(Conversation object selection unit 191)
The conversation object selection unit 191 selects a “COMM link” object described later. When the position acquisition unit 183 acquires the position in the captured image designated by the user on the display screen of the captured image in the real space, the conversation object selection unit 191 corresponds to the real space 3 based on the position. Select a COMM link (ie, a conversation event object) located in the dimensional virtual space. The COMM link is, for example, an object corresponding to a call, has a line segment shape, and has an object 91 or the like corresponding to a person involved in the call at both ends thereof. Details of the COMM link will be described later.

(ID acquisition unit 187)
The ID acquisition unit 187 acquires the identification information corresponding to the selected object 91. For example, the identification information is identification information for communication (hereinafter, referred to as “communication ID”) corresponding to the selected object 91. An example of the communication ID is a telephone number. Specifically, for example, when the object selection unit 185 selects the object 91, the ID acquisition unit 187 acquires the object ID of the selected object 91. Then, the object selection unit 185 transmits the object ID to the information management server 200 via the communication unit 110, and acquires the communication ID corresponding to the object ID. The communication ID is the communication ID of the person corresponding to the selected object 91. The communication ID is the communication ID of the communication device of the person corresponding to the object 91 (that is, the person shown in the position of the captured image designated by the user). Then, the ID acquisition unit 187 provides the acquired communication ID to the telephone unit 189 described later.

As described above, in the present embodiment, when the position in the captured image is designated by the user, the object 91 corresponding to the position is selected and the communication ID corresponding to the object 91 is acquired. This allows the user to contact the target person with an intuitive operation. Further, the object 91 corresponding to the person is selected and the communication ID is acquired regardless of how the person appears in the captured image, so that the person can be contacted more reliably. ..

Further, the ID acquisition unit 187 communicates a plurality of pieces of identification information corresponding to the selected conversation object, specifically, the communication of two or more speakers corresponding to the objects 91 located at both ends of the line segment of the COMM link described later. It is also possible to acquire a business ID. Specifically, for example, when the object selection unit 185 selects a conversation object (COMM link), the ID acquisition unit 187 causes two or more speakers corresponding to the objects 91 located at both ends of the selected conversation object. Get the object ID. Then, the object selection unit 185 transmits the two or more object IDs acquired by the ID acquisition unit 187 to the information management server 200 via the communication unit 110, and acquires the communication ID corresponding to the object ID. The communication ID is the communication ID of the person corresponding to the selected object 91. Further, the ID acquisition unit 187 acquires the plurality of communication IDs acquired by the object selection unit 185. Then, the ID acquisition unit 187 provides the acquired communication ID to the telephone unit 189.

As described above, when the position in the captured image is designated by the user, the conversation object corresponding to the position is selected, and the plurality of communication IDs corresponding to the conversation object are acquired. As a result, the user can specify the invisible presence of remote conversation by an intuitive operation, and can easily perform the operation of transitioning to the existing three-way call by joining the existing two-way call, for example. Become.

(Telephone section 189)
The telephone unit 189 provides a function for making a telephone call. For example, the telephone unit 189 provides a softphone function. For example, when the telephone unit 189 acquires the communication ID provided by the ID acquisition unit 187, it makes a telephone call using the communication ID. More specifically, for example, when the telephone unit 189 acquires the communication ID, the telephone unit 189 provides the communication ID to the PBX 40 via the communication unit 110 and acquires the IP address from the PBX 40. Then, the telephone unit 189 executes a series of sequences for establishing a session with the communication device having the IP address (that is, the communication device at the destination). In this way, the telephone unit 189 makes a telephone call for making a telephone call to the person shown in the position in the captured image designated by the user on the display screen. That is, the telephone unit 189 makes a telephone call to the communication device of the person.

When voice data from the communication device of the other party of the call is received by the communication unit 110, the telephone unit 189 causes the voice output unit 160 to output the voice of the voice data. The telephone unit 189 also causes the communication unit 110 to transmit the voice data provided by the sound collection unit 140 to the communication device of the other party of the telephone. The telephone unit 189 also causes the communication unit 110 to transmit, for example, the captured image provided by the image capturing unit 130 (for example, the captured image showing the user of the terminal device 100) to the communication device of the other party of the telephone.

In addition, the telephone unit 189 causes the display unit 150 to display a display screen during a telephone call. For example, a display mode for displaying a display screen during a call is called a conversation mode. In this case, when the telephone unit 189 acquires the communication ID provided by the ID acquisition unit 187, the display unit switches the display mode from the proximity mode to the conversation mode. Hereinafter, a specific example of the conversation mode display screen will be described with reference to FIG.

FIG. 12 is an explanatory diagram for explaining an example of the display screen 80 displayed in the conversation mode. In FIG. 12, a display screen 80 displayed in the conversation mode is shown. The display screen 80 includes a partner side captured image 81, a button image 83, and a self side captured image 85.

The other party's captured image 81 is, for example, a captured image acquired from the communication device of the other party of the telephone. For example, when the communication unit 110 receives a captured image from the communication device of the other party of the telephone, the telephone unit 189 uses the captured image as the captured image 81 of the other party.

The button image 63 is an image for ending the call. For example, when the user specifies the position of the button image 83, the telephone unit 189 ends the call. More specifically, for example, when the user touches the button image 83 and the touch position corresponding to the button image 83 is detected, the telephone unit 189 executes a sequence for terminating the call including disconnecting the session. Further, for example, the telephone unit 189 switches the display mode from the conversation mode to the bird's eye view mode.

The own-side captured image 85 is a captured image provided by the image capturing unit 130.

− Transition of display mode −
Here, a specific example of the transition of the display mode of the display screen, that is, the transition of the overhead view mode, the proximity mode, and the conversation mode will be described with reference to FIG. 13. FIG. 13 is a transition diagram for explaining an example of transition of display modes. Referring to FIG. 13, for example, when the connection process with the camera 11, the microphone 13, the sensor 15, the information management server 200, and the like is performed at the time of starting the software, the display mode becomes the overhead view mode 301 (ENTRY).

In the overhead view mode 301, the overhead view mode process is executed (DO). Then, when the user specifies the position of the bird's-eye view captured image 61, a mode changing process is performed (EXIT), and the display mode is switched from the bird's-eye view mode 301 to the proximity mode 303. The mode changing process includes the zoom process of the camera 11 (ENTRY).

In the proximity mode 303, proximity mode processing is executed (DO). Then, when the user specifies the position of the person in the close-up captured image 71, the mode changing process is performed (EXIT), and the display mode is switched from the close-up mode 303 to the conversation mode 305. The mode changing process in this case includes a process for a call (ENTRY). When the user specifies the position of the button image 73, the mode changing process is performed (EXIT), and the display mode is switched from the proximity mode 303 to the bird's eye view mode 301. The mode changing process in this case includes the connection process described above (ENTRY).

In the conversation mode 305, conversation mode processing is executed (DO). Further, when the user touches the button image 83, a mode changing process is performed (EXIT), and the display mode is switched from the conversation mode 305 to the bird's eye view mode 301. The mode changing process in this case includes the connection process described above (ENTRY).

-Captured image according to display mode-
As described above, for example, in the first display mode (for example, bird's-eye view mode), the real space information providing unit 181 uses the first captured image (for example, the bird's-eye view captured image) in which the first region of the real space is captured. 61) is displayed on the display unit 150. Further, in the second display mode (for example, the proximity mode), the real space information providing unit 181 uses the second captured image (for example, the proximity captured image) in which the second region narrower than the first region is captured. 71) is displayed on the display unit 150. Then, in the example described above, the first captured image in which the first region of the real space is captured is the captured image corresponding to the first zoom ratio, and the second region narrower than the first region. The second captured image of is captured is a captured image corresponding to a second zoom rate that is higher than the first zoom rate. However, in the present embodiment, the first captured image and the second captured image are not limited to this.

For example, in the overhead view mode, the real space information providing unit 181 selects the camera 11 that captures a wide area of the center office 10 and acquires the captured image of the camera 11 as the overhead captured image 61. Then, in the proximity mode, the real space information providing unit 181 selects the camera 11 (for example, the camera in front) that captures a narrower area of the center office 10, and captures the image captured by the selected camera 11. Is acquired as the close-up captured image 71.

Thereby, depending on the arrangement of the camera 11, it becomes easier to specify the position of the person by the captured image acquired by the camera 11. In such a case, it may not be necessary to request the camera 11 to zoom. As a result, in this case, for example, as in the case of requesting optical zoom or zoom by a dolly, requests from the plurality of individual terminal devices 100 for the same camera 11 compete with each other, and one of the terminal devices 100 is in a waiting state. Will not occur. Further, in this case, the processing amount does not increase unlike the case of using the digital zoom.

-Captured image generated by imaging under more free conditions-
Moreover, although the example in which the display mode is switched has been described in the above-described example, the display screen is not limited to this in the present embodiment. For example, instead of switching the display mode, a captured image may be acquired by capturing under more free conditions, and a display screen including the captured image may be displayed. For example, the captured image in the real space may be a captured image corresponding to the selected zoom ratio of the plurality of zoom ratios. In this case, for example, the real space information providing unit 181 requests the camera 11 via the communication unit 110 for the zoom ratio specified by the user via the input unit 120. As a result, the camera 11 changes the zoom rate according to the request, and provides the terminal device 100 with the captured image generated by the image capturing with the changed zoom rate. Then, the real space information providing unit 181 causes the display unit 150 to display a display screen including the provided captured image. Then, when the user specifies a position in the captured image, the position acquisition unit 183 acquires the position. Further, the position acquisition unit 183 provides the position in the captured image to the object selection unit 185. This allows the user to specify a fine zoom ratio and display a desired captured image. Therefore, the user can more easily specify the position of the specific person by using the captured image.

Note that, as described above, the zoom ratio here does not have to be a delicate value such as 1.5 times or 2 times, and directly or indirectly indicates the degree of the size of the subject captured in the captured image. It is a thing. For example, especially when zooming by changing the position of the camera 11 (for example, zooming in and out by dolly) is used, the zoom ratio is not a precise value such as 1.5 times or 2 times, but the size of the subject. That directly indicates the size of the subject (for example, a parameter that indicates the size of the subject, or the like) or indirectly indicates the size of the subject (eg, the position of the camera 11 on the rail). Etc.).

(COMM link control unit 193)
The COMM link control unit 193 receives information regarding the “COMM link” object from the information management server 200, and controls display of the COMM link.

Here, in the present specification, the “COMM link” object may be a display element or an input element of a user interface presented on the screen of the display unit 150 of the terminal device 100. More specifically, the COMM link is a position in the captured image corresponding to the object 91 corresponding to the person C with respect to, for example, the person C and the person D who are talking via the sound collection unit 140 of the terminal device 100. And a position in the captured image corresponding to the object 91 corresponding to the person D is a “line segment” or a curved “string”. The COMM link can exist as a three-dimensional line segment or curved chord in the three-dimensional virtual space 90 corresponding to the center office 10 (described later with reference to FIG. 14).

The person C and the person D may exist at different bases in the real space, and in that case, the COMM link may be displayed across a plurality of captured images of the different bases. I do not care. Further, two or more COMM links may exist for one person at the same time. For example, in the case of a three-way call, three positions corresponding to the person L, the person M, and the person N are formed into, for example, three sides of a triangle. There may be three COMM links to be connected and displayed. Further, the user designates a part of the internal area of the COMM link that forms a triangle or a polygon having vertices or more, so that a plurality of objects located at the ends of the plurality of COMM links that form the polygon or the like. All of 91 may be designated (FIG. 16 to be described later is an example of such a display). In addition, in the polygonal COMM link, it is preferable that a plurality of sides forming the same polygon are configured not to intersect with each other.

The conversation behavior (calling operation) that is the basis of the COMM link does not have to be via the sound collecting unit 140 of the terminal device 100. For example, the information management server 200, which will be described later, may record and update information about the installation position of each microphone 13 and the sound collection direction as parameters of the plurality of microphones 13 included in the information processing system according to the present embodiment. In this case, the information management server 200 uses the voice data collected from the microphone 13 installed at the position closest to the position information of the object 91 of the person who is the speaker of the conversation behavior (call operation), and The COMM link and the processing relating to the generation and management of the COMM word (language phrase object) described later may be performed.

The COMM link is projected and superposed on the screen of the display unit 150 of the terminal device 100, for example, on the bird's-eye view captured image 61 or the map image 69 in FIG. 4 by controlling the display unit 150 by the COMM link control unit 193. It is displayed (described later with reference to FIG. 15). The user who sees the display screen of the terminal device 100 visually recognizes the COMM link displayed on the mapped image of the real space, and thereby identifies who and who are in a call in a plurality of remote places in the distributed environment. You can understand intuitively. Furthermore, by visualizing a voice information-based action called a conversation (call) by a COMM link, even when media information sharing is restricted to remote communication of only image data, for example, voice data at a remote location cannot be acquired and output. The user can know the occurrence of the conversation act at the remote place by looking at the display screen of the terminal device 100 on which the COMM link is displayed.

Hereinafter, an example of the COMM link 86 will be described with reference to FIG. FIG. 14 is an explanatory diagram illustrating an example of the COMM link 86 in the three-dimensional virtual space 90 corresponding to the center office 10. The data regarding the three-dimensional virtual space 90 is managed by the information management server 200 as described above. In FIG. 14, a three-dimensional virtual space 90 corresponding to the center office 10 is shown. In addition, an object 91C and an object 91D are arranged in the three-dimensional virtual space 90. The object 91C corresponds to the person C, for example, and is arranged at a three-dimensional virtual position corresponding to the position of the seat of the person C in the center office 10. Further, the object 91D corresponds to, for example, the person D, and is arranged at a three-dimensional virtual position corresponding to the position of the seat of the person D in the center office 10. Similar to the examples of FIGS. 8 and 10, these objects 91 are columnar objects with a radius R and a height H, for example. Further, in FIG. 14, three-dimensional barycentric positions 92C and 92D of the object 91C and the object 91D are also shown. In the present embodiment, the three-dimensional barycentric positions 92C and 92D do not have to be geometrical three-dimensional barycentric positions, and are positions (points) included in the three-dimensional shape of the object 91, respectively. It may be any position, and may be the center position of the upper surface of the object 91, for example. Further, FIG. 14 shows a line-segment-shaped COMM link 86 having both ends at the three-dimensional center-of-gravity positions 92C and 92D. As described above, the COMM link 86 can have information such as position and shape in the three-dimensional virtual space 90 as a kind of object. The generation of the COMM link 86 will be described later.

Then, as described above, since the COMM link 86 is an object that connects the objects 91 of the person who is the user of the terminal device 100 in the talking state, the COMM link 86 is located at both ends of the line segment, It is associated with at least two or more objects 91 and corresponding communication ID data. For example, when the user specifies the COMM link 86 in FIG. 14 by the input method described below, the terminal device 100 of the user has information on the communication IDs of the objects 91C and 91D at both ends of the COMM link 86. Can be obtained. As a result, the terminal device 100 of the user simultaneously selects two objects 91C and 91D at different positions based on one operation of designating one object of the COMM link 86 of the user, for example. be able to. Further, the terminal device 100 of the user accesses (the terminal device 100 corresponding to) the person C and the person D by using the communication IDs of the person C and the person D corresponding to the above-described two objects 91. be able to. That is, the user can participate in the two-party call by performing one operation for the person C and the person D who are in the two-party call to specify the COMM link 86 that connects them to each other. A three-way call with C and person D can be made.

Also, when two calls, a three-way call and a two-way call, exist in the real space, when one of the two-way calls specifies a triangular COMM link indicating the three-way call. , Only the speaker who has designated the COMM link for the three-way call may participate in the three-way call, or both speakers of the two-way call may participate in the three-way call. Furthermore, if only the speaker who specified the COMM link for the three-way call participates in the three-way call, and if more than one speaker in the three-way call permits, the remaining two-way caller becomes the three-way call. You may participate.

(COMM word control unit 195)
The COMM word control unit 195 receives information regarding a “COMM word” object described later from the information management server 200, and controls the display of the COMM word. Further, the COMM word control unit 195 causes the communication unit 110 to transmit the voice data provided by the sound collection unit 140 to the information management server 200. At this time, the COMM word control unit 195 may add the data of the communication ID corresponding to the person who uttered the voice to the voice data to be transmitted, and transmit the voice data. In the process of identifying the communication ID, the terminal device 100 selects a communication ID corresponding to a person registered in advance as a user of the terminal device 100, or the voice recognition server 201 performs a speaker recognition process to perform voice recognition. You may implement by specifying a speaker from data.

Here, in the present specification, the “COMM word” object may be a display element of a user interface presented on the screen of the display unit 150 of the terminal device 100. More specifically, the COMM word is an object that is displayed in the vicinity of the COMM link 86 described above, in which the main part of the call content via the terminal device 100 is extracted by the voice recognition process, visualized, and displayed. Good. For example, a person C and a person D are making a two-party call via the terminal device 100C and the terminal device 100D, respectively. At this time, the voice data of the person C and the person D respectively input from the terminal devices 100C and 100D are added to the communication ID data of the terminal devices 100C and 100D and transmitted to the information management server 200. The information management server 200 transmits the received voice data to the voice recognition server 201, receives the word/phrase data as the recognition result from the voice recognition server 201, and transmits the word/phrase data corresponding to the recognition result to the terminal devices 100C and 100D. To do. The details of the processing in the information management server 200 and the voice recognition server 201 will be described later. Further, the COMM word is displayed on the screen of the display unit 150 of the terminal device 100, for example, on the bird's-eye view captured image 61 or the map image 69 in FIG. 4 by controlling the display unit 150 by the COMM word control unit 195. It is superposed and displayed at a position near the corresponding COMM link 86 (see FIG. 15). The “COMM word” object may be data having a position and a shape (volume) in the three-dimensional virtual space 90 like the above-mentioned “COMM link” object, and the data is managed by the information management server 200. May be. Further, at that time, the “COMM word” object may be set in a position spatially close to or in the vicinity of the corresponding “COMM link” object in the three-dimensional virtual space 90, and the setting is performed by the information management server. It may be performed by 200.

An example of the COMM word 87 will be described below with reference to FIG. FIG. 15 is an explanatory diagram illustrating an example of the COMM link 86 and the COMM word 87 displayed on the display screen 50 of the terminal device 100. In FIG. 15, a display screen 50 displayed in the overhead view mode or the proximity mode is shown. The display screen 50 shows the presence of the captured images 51A, 51B, 51Z, the map images 69A, 69B, 69Z, the person images 77C, 77D, 77E, 77F, the persons C, D, E, F of the persons C, D, E, F. It includes icons 79C, 79D, 79E, 79F, COMM links 86G2, 86G3, 86H2, 86H3, and COMM words 87I, 87J, 87K, 87L projected and superimposed. The captured images 51A, 51B, and 51Z are images obtained by bird's-eye view of the bases A, B, and Z (for example, Tokyo, Osaka, and Okinawa) in the distributed environment, and the map images 69A, 69B, and 69Z are captured images 51A and 51Z. It is the top view which expressed two-dimensionally the positional relationship of various objects in 51B and 51C. Only one of the captured image 51 and the map image 69 may be displayed on the display screen 50.

For example, when the person C at the location A and the person D at the location A start a two-party call between the terminal devices 100C and 100D at their own seats, the COMM link 86G3, between the person images 77C and 77D on the captured image 51A, COMM links 86G2 are displayed on the presence icons 79C and 79D on the map image 69A, and COMM words 87I, 87J, and 87K are displayed near the COMM links 86G2. The person images 77C and 77D on the captured image 51A correspond to the person C and the person D, respectively, and the presence icons 79C and 79D on the map image 69A correspond to the person C and the person D, respectively. Further, when the person E at the location A and the person F at the location B start a two-party call between the terminal devices 100E and 100F in their own seats, the person image 77E on the captured image 51A and the person image 77F on the captured image 51B are straddled. The COMM link 86H2 is displayed across the COMM link 86H3, the presence icon 79E on the map image 69A, and the presence icon 79F on the map image 69B. The person images 77E and 77F on the captured image 51A correspond to the person E and the person F, respectively, and the presence icons 79E and 79F on the map image 69A correspond to the person E and the person F, respectively.

Then, when the user of the terminal device 100U touch-inputs and specifies the COMM link 86G3 or the COMM link 86G2 on the display screen 50, the user can participate in the two-party conversation between the person C and the person D. When the user of the terminal device 100U touch-inputs and specifies the COMM link 86H3 or the COMM link 86H2 on the display screen 50, the user can participate in the two-party conversation between the person E and the person F.

In FIG. 15, COMM words 87I, 87J, and 87K are displayed near the COMM link 86G3 or the COMM link 86G2. Here, as described above, the COMM word 87 is an object in which the main part of the content of the call via the terminal device 100 is extracted by the voice recognition process, visualized, and displayed. For example, in FIG. 15, the COMM word 87 is displayed so that the weight by the statistical weighting process described later is reflected (the weighting process is performed by the information management server 200 described later, and the result of the weighting process is performed. Is distributed to the terminal device 100, etc. The details will be described later). For example, in FIG. 15, the phrase “meeting” of the COMM word 87I is displayed in a larger size than the phrase “cancel” of the COMM word 87J and the “holding” of the phrase of the COMM word 87K. The display of the COMM word in FIG. 15 is a display in which the weight by the statistical weighting process is reflected. For example, when the index that is the basis of the weight is the number of times the speaker speaks, the word “meeting” is not displayed. It is shown that the speaker speaks more times than the phrase “stop” and appears frequently in the call. In this way, the COMM word 87 is displayed so that the weight by the statistical weighting process is reflected, so that the user looking at the display screen 50 can intuitively understand the main part of the conversation.

Further, explaining the position of the COMM word 87, the phrase "meeting" of the COMM word 87I is located near the center of the corresponding COMM link 86G. On the other hand, the phrase “stop” of the COMM word 87J is linked to the position of the person image 77C of the person C linked to the COMM link 86G, and the phrase “holding” of the COMM word 87K is linked to the COMM link 86G. It is displayed at a position near the person image 77D of the attached person D. This is based on the position (the object 91 at the end of the COMM link 86) of the speech state of which person C or person D making a two-way call corresponding to the COMM link 86G frequently speaks. It is the result of the visualization process after conversion into a parameter (synonymous with the distance from). For example, the phrase “meeting” of the COMM word 87I is located near the center of the COMM link 86, and thus indicates that both the person C and the person D are speaking the same number of times. On the other hand, since the phrase “stop” of the COMM word 87J is located near the person image 77C of the person C, the person C is speaking a lot, and the phrase “holding” of the COMM word 87K is the person D. Since it is located in the vicinity of the person image 77D, it indicates that the person D speaks a lot. The positional relationship between each COMM word 87 and the object 91 (that is, the object of the speaker of the corresponding word/phrase) is not limited to the display on the display unit 150 and the same position or position in the three-dimensional virtual space 90. The same distance relationship may be used (for example, the COMM word 87J may be located near the object 91C corresponding to the person image 77C of the person C in the three-dimensional virtual space 90).

As described above, in the present embodiment, the COMM word control unit 195 performs the visualization processing that reflects the above statistical weighting and the utterance status, and displays the main part of the call content as the COMM word 87. Therefore, according to the present embodiment, even a user who is in a remote place and cannot hear the voice of conversation can roughly understand the conversation content by looking at the screen on which the COMM word 87 is displayed. .. For example, in the example of FIG. 15, the user at the remote place sees the screen on which the COMM word 87 is displayed, and the person C and the person D are having a conversation with the theme of whether to hold a meeting tomorrow, It is possible to grasp the content of the conversation such that the person C insists on canceling or postponing the conference, while the person D insists that the conference should be held. Then, if the user becomes interested in the conversation content after having roughly grasped the conversation content, the user can smoothly participate in the conversation by a simple designation operation on the COMM link 86 described above. That is, the user can smoothly participate in the two-party conversation, as in the case of being in the same environment.

Note that the COMM word control unit 195 does not have to display all the phrase data included in the call when displaying the COMM word 87. For example, the COMM word control unit 195 compares the weight value in the above weighting with a predetermined threshold value (predetermined value), and only the words and phrases whose weight value is equal to or greater than the predetermined threshold value are displayed on the display screen of the terminal device 100. You may control so that it may be displayed in. By doing so, only the words and phrases having a high degree of importance in the conversation are “selected” and displayed as the COMM word 87. At this time, the display control may also be performed on the COMM link 86 corresponding to the COMM word 87 of the weighted phrase data. Further, the COMM word control unit 195 may control the display of the COMM word 87 based on the weight value. Specifically, the COMM word control unit 195 controls the display size and color of the COMM word 87, the contrast with the screen on which the display is superimposed, the display position, and the like based on the weight value. Good.

FIG. 16 is an explanatory diagram for explaining another example of the COMM link 86 and the COMM word 87 displayed on the display screen 55 of the terminal device 100. In FIG. 16, a display screen 55 displayed in the overhead view mode or the proximity mode is shown. First, when the person C at the location A and the person F at the location B start a two-way call between the terminal devices 100C and 100F at their own seats, a COMM link 86 and a COMM word 87 are displayed between the person images of the person C and the person F. To be done. Further, when the person D at the site A designates the COMM link 86, participates in the conversation corresponding to the COMM link 86, and shifts to the three-way call, the person C and the person D, and the person F and the person D A COMM link 86 and a COMM word 87 are displayed between the images. As a result, for example, triangular COMM links 86M3 and COMM links 86M2 that connect the person C, the person D, and the person F are displayed. In this case, in order for another person E to enter the three-way call and shift to the four-way call, for example, the person E is at a position corresponding to one side of the triangular COMM link 86M2, or An operation for designating a position corresponding to a part of the inner area of the triangle may be performed.

<1.2.3 Software configuration>
Next, an example of the software configuration of the terminal device 100 according to the present embodiment will be described. FIG. 17 is a block diagram showing an example of the software configuration of the terminal device 100 according to this embodiment. Referring to FIG. 17, the terminal device 100 includes an OS (Operating System) 840 and a plurality of application software. Then, the terminal device 100 includes a softphone 851, a super-realistic sensation client 853, and a telephone call control function 855 as application software.

(OS840)
The OS 840 is software that provides basic functions for operating the terminal device 100. The OS 840 executes each application software.

(Softphone 851)
The softphone 851 is application software for making a call using the terminal device 100. The telephone unit 189 can be realized by the softphone 851, for example.

(Ultra-realistic client 853)
The ultra-realistic client 853 is application software for providing information in the real space to the terminal device 100. The ultra-realism client 853 may acquire state information indicating the state of a person in the real space (for example, the center office 10) and provide the state information to the softphone 851 via the OS. Then, the softphone 851 may control the call origination based on the state information. The real space information providing unit 181 can be realized by, for example, the ultra-realistic presence client 853.

(Telephone call control function 855)
The telephone call control function 855 is application software that acquires the communication ID of the communication device of the person shown in the captured image on the display screen. Upon acquiring the communication ID, the telephone call control function 855 provides the softphone 851 via the OS 840. Then, the softphone 851 makes a telephone call using the communication ID. The position acquisition unit 183, the object selection unit 185, and the ID acquisition unit 187 can be realized by the telephone call control function 855.

<1.3 Information management server configuration>
Subsequently, an example of the configuration of the information management server 200 according to the present embodiment will be described with reference to FIGS. 18 and 19. As described above, the information management server 200 manages various information used in the information processing system according to this embodiment.

<1.3.1 Hardware configuration>
First, an example of the hardware configuration of the information management server 200 according to the present embodiment will be described with reference to FIG. FIG. 18 is a block diagram showing an example of the hardware configuration of the information management server 200 according to this embodiment. Referring to FIG. 18, the information management server 200 includes a CPU 901, a ROM 903, a RAM 905, a bus 907, a storage device 909, and a communication interface 911.

(CPU901, ROM903, RAM905)
The CPU 901 executes various processes in the information management server 200. Further, the ROM 903 stores a program and data for causing the CPU 901 to execute the process in the information management server 200. Further, the RAM 905 temporarily stores programs and data when the processing of the CPU 901 is executed.

(Bus 907)
The bus 907 connects the CPU 901, the ROM 903, and the RAM to each other. A storage device 909 and a communication interface 911 are further connected to the bus 907. The bus 907 includes, for example, a plurality of types of buses. As an example, the bus 907 may include a high-speed bus that connects the CPU 901, the ROM 903, and the RAM 905, and one or more other buses that are slower than the high-speed bus.

(Memory device 909)
The storage device 909 stores data to be temporarily or permanently stored in the information management server 200. The storage device 909 may be, for example, a magnetic storage device such as a hard disk, or a non-volatile memory such as EEPROM, flash memory, MRAM, FeRAM and PRAM.

(Communication interface 911)
The communication interface 911 is a communication unit included in the information management server 200, and communicates with an external device via a network (or directly). The communication interface 911 may be an interface for wireless communication, and in this case, may include, for example, a communication antenna, an RF circuit, and other circuits for communication processing. Further, the communication interface 911 may be an interface for wired communication, and in this case, for example, may include a LAN terminal, a transmission circuit, and other circuits for communication processing.

<1.3.2 Functional configuration>
Next, an example of a functional configuration of the information management server 200 according to the present embodiment will be described. FIG. 19 is a block diagram showing an example of the functional configuration of the information management server 200 according to this embodiment. Referring to FIG. 19, the information management server 200 includes a communication unit 210, a storage unit 220, and a control unit 230.

(Communication unit 210)
The communication unit 210 communicates with another device. For example, the communication unit 210 is directly connected to the LAN 19 and communicates with each device in the center office 10. Specifically, for example, the communication unit 210 communicates with the camera 11, the microphone 13, the sensor 15, the media distribution server 17, and the voice recognition server 201. In addition, the communication unit 210 communicates with each device in the satellite office 20 via the external network 30 and the LAN 23. Specifically, for example, the communication unit 210 communicates with the terminal device 100 and the display 21. The communication unit 210 can be realized by the communication interface 911, for example.

(Storage unit 220)
The storage unit 220 stores programs and data for the operation of the information management server 200. In particular, in this embodiment, the storage unit 220 stores various information used in the information processing system.

As a first example, the storage unit 220 stores parameters regarding the camera 11, the microphone 13, and the sensor 15. The specific content of the parameter is as described above. As a second example, the storage unit 220 stores the data of the three-dimensional virtual space 90 corresponding to the real space. The three-dimensional virtual space 90 is, for example, a three-dimensional virtual space imitating the center office 10. The specific content of the three-dimensional virtual space 90 is as described above. As a third example, the storage unit 220 stores person-related information. The person-related information is, for example, person-related information about a person in the center office 10. The specific content of the person-related information is as described above. As a fourth example, the storage unit 220 stores the object ID of the object 91 arranged in the three-dimensional virtual space and the communication ID in association with each other. The specific contents of the object ID and the communication ID are as described above.

(Control unit 230)
The control unit 230 provides various functions of the information management server 200. The control unit 230 includes a COMM link distribution unit (conversation event object distribution unit) 231, an extracted word data management unit 232, a weighting calculation unit (weighting processing unit) 233, a COMM word distribution unit (language phrase object distribution unit) 234, and a statement. The situation calculation unit 235 is included. The control unit 230 provides various information used in the information processing system in response to a request, for example. Specifically, the control unit 230, in response to a request from the terminal device 100, the data of the three-dimensional virtual space 90, the person-related information, the communication ID corresponding to the object ID, the camera 11, the microphone 13, and the sensor 15. It provides parameters related to. In addition, for example, the control unit 230 may appropriately update various information used in the information processing system according to the present embodiment. The control unit 230 updates the information automatically or according to a manual instruction.

(COMM link distribution unit 231)
The COMM link distribution unit 231 receives the information for generating the COMM link 86 from the terminal device 100 or the like, calculates the position and size of the COMM link 86, distributes the COMM link 86, and inputs the user to the COMM link 86. Acquire data. Specifically, the COMM link delivery unit 231 receives a conversation action (call operation) that is the basis of the COMM link 86 from the terminal device 100 via the communication unit 210. Then, the COMM link distribution unit 231 acquires the object 91 relating to the speaker who uses the terminal device 100 using the three-dimensional virtual space 90 described above, based on the information from the terminal device 100, and Data related to the COMM link 86 connecting the objects 91 is generated and distributed. Specifically, as described above, the COMM link distribution unit 231 refers to the plurality of person objects 91 in the three-dimensional virtual space 90 corresponding to the real space stored in the storage unit 220 in advance and refers to the terminal concerned. Based on the conversation behavior information obtained from the device 100, the speaker object 91 relating to the conversation behavior is selected. Then, a COMM link that connects the selected objects 91 is generated in the three-dimensional virtual space 90.

Further, the COMM link distribution unit 231 causes the storage unit 220 to store the data related to the generated COMM link 86. At this time, the COMM link delivery unit 231 may give an ID for identification to the COMM link 86, store it in the storage unit 220, and manage information. Further, the COMM link delivery unit 231 stores the ID given to the COMM link 86 and the object IDs (communication ID (communication ID information)) of two or more speakers corresponding to the COMM link 86 in association with each other. By storing in the unit 220, the CONN link 86 may be associated with the object IDs of two or more speakers and managed. Note that the COMM link distribution unit 231 does not use the data of the object 91 in the three-dimensional virtual space 90 for the position information of the person who is the speaker of the conversation action (call operation), and the information management server 200 stores a plurality of information items. The position may be estimated and obtained by performing sound source estimation processing from the sound collection data of the microphone 13.

Further, when the COMM link delivery unit receives an input operation from the user for the generated COMM link 86 via the communication unit 210, the COMM link distribution unit associates the object ID of the user with the COMM link 86. By doing so, the object IDs of two or more speakers corresponding to the COMM link 86 are associated with the object IDs of the user. Then, in the present embodiment, by referring to the object ID associated with one COMM link 86 and performing control, it is possible to start a call newly added by the user.

(Extracted word data management unit 232)
The extracted word/phrase data management unit 232 acquires the voice data received from the communication unit 110 of the terminal device 100 and transmits the voice data to the voice recognition server 201. Then, the extracted word/phrase data management unit 232 receives the word/phrase data of the recognition result from the voice recognition server 201, and manages it in association with the identification information of the speaker of the word/phrase. Note that, for example, the extracted word data management unit 232 may cause the storage unit 220 to store the data regarding the word and the identification information of the speaker of the word.

(Weighting calculation unit 233)
The weighting calculation unit 233 analyzes the word/phrase data managed by the extracted word/phrase data management unit 232, and performs statistical weighting processing. As an index for the weighting, for example, the appearance frequency (number of times) of a phrase in conversation (for example, the weight increases as the appearance frequency increases) and the abstraction degree of the phrase (for example, the weight increases as the specificity increases). Specifically, the phrase "French cuisine" is more specific than "Cooking," and "Buch Bourquinon (burgundy-style beef cooked in red wine)" is better than "French cuisine." The value indicating the degree of abstraction is supplied from the voice recognition server 201 together with the extracted word, for example, and the part of speech category of the word (for example, the weight of the noun rather than the verb). Increase) is used. Further, the terminal device 100 uses the sound collection unit 140 and the telephone unit 189 to acquire the sound pressure data of the call voice (word utterance) together with the voice data at the time of acquiring the voice data to the information management server 200. Then, the weighting calculation unit 233 may use the sound pressure data as an index for weighting (for example, a word or phrase emitted at a higher sound pressure level has a higher weight).

(COMM word delivery unit 234)
The COMM word distribution unit 234 generates a COMM word 87 including the phrase using the phrase data including the phrase acquired from the extracted phrase data management unit 232, the weighting calculation unit 233, and the statement situation calculation unit 235 described later, The data related to the COMM word 87 is distributed. At this time, the COMM word delivery unit 234 associates the generated COMM word 87 with the identification information of the speaker of the phrase included in the COMM word. Further, the COMM word distribution unit 234 may store the generated COMM word 87 in the storage unit 220. Further, the COMM word delivery unit 234 may deliver according to the result of the weighting processing by the weighting calculation unit 233. More specifically, the COMM word distribution unit 234 compares the weighting processing result (weight value) with a predetermined threshold value (predetermined value), and distributes the generated COMM word 87 based on the comparison result. You may. In this case, the COMM word delivery unit 234 may deliver the data of the result of the weighting process in association with the COMM word 87. Furthermore, the COMM word distribution unit 234 may also distribute the position information of the COMM word 87 calculated by the statement status calculation unit 235 described later.

(Speaking status calculation unit 235)
The utterance status calculation unit 235 analyzes the word/phrase data managed by the extracted word/phrase data management unit 232 and the identification information data corresponding to the word/phrase data, and determines whether the word/phrase data is a virtual place in a two-way call. It is calculated whether or not there is such a position (virtual distance from each speaker). The speech state calculation unit 235 calculates the position of the COMM link 86 in the three-dimensional virtual space 90 and the position of the COMM word 87 corresponding to the COMM link 86 with respect to the COMM link 86. For example, in the example of FIG. 15, the word “meeting” of the COMM word 87I is spoken by the person C and the person D at the same number of times (for example, the person C is 10 times and the person D is 10 times). In this case, the position of the COMM word 87I "meeting" is, for example, the "midpoint" on the COMM link 86G. On the other hand, in the phrase “stop” of the COMM word 87J, since the person C is often speaking (for example, the person C is 5 times and the person D is 0 times), the position of the COMM word 87J “stop” is the person. It is near C. Further, since the person D often speaks the phrase "holding" in the COMM word 87K (for example, the person C once and the person D six times), the position of the COMM word 87K "holding" is the person D. It becomes the neighborhood of. Then, the utterance status calculation unit associates the calculated position with the COMM word and supplies the COMM word to the COMM word distribution unit 234. Since the COMM word distribution unit 234 distributes the position information together with the COMM word 87 to the terminal device 100, the COMM word 87 is displayed on the terminal device 100 based on the position information.

That is, the control unit 230 processes the voice data received from the terminal device 100 in the call state by the information processing by the extraction word/phrase data management unit 232, the weighting calculation unit 233, the statement status calculation unit 235, etc. It is converted into weighted word/phrase data that reflects the situation, and is transmitted to the terminal device 100 at the relevant base. The control unit 230 (COMM word delivery unit 234) may deliver only the weighted word/phrase data whose weight value in the statistical weighting is equal to or greater than a predetermined threshold value to the terminal device 100.

<1.4 Structure of voice recognition server>
Subsequently, an example of the configuration of the voice recognition server 201 according to the present embodiment will be described with reference to FIGS. 20 and 21. As described above, the voice recognition server 201 has a large-scale word list data built therein, receives voice data acquired by the terminal device 100 or the microphone 13 via the information management server 200, and outputs the voice data. Voice recognition processing is performed on the received data and the data of the recognition result is transmitted to the information management server 200.

<1.4.1 Hardware configuration>
An example of the hardware configuration of the voice recognition server 201 according to the present embodiment will be described with reference to FIG. FIG. 20 is a block diagram showing an example of the hardware configuration of the voice recognition server 201 according to this embodiment. Referring to FIG. 20, the voice recognition server 201 includes a CPU 701, a ROM 703, a RAM 705, a bus 707, a storage device 709, and a communication interface 711.

(CPU 701, ROM 703, RAM 705)
The CPU 701 executes various processes in the voice recognition server 201. Further, the ROM 703 stores a program and data for causing the CPU 701 to execute the process in the voice recognition server 201. Further, the RAM 705 temporarily stores a program and data when the processing of the CPU 701 is executed.

(Bus 707)
The bus 707 connects the CPU 701, the ROM 703, and the RAM 705 to each other. A storage device 709 and a communication interface 711 are further connected to the bus 707. The bus 707 includes, for example, a plurality of types of buses.

(Memory device 709)
The storage device 709 stores data to be temporarily or permanently stored in the voice recognition server 201, for example, word/phrase data. The storage device 709 may be, for example, a magnetic storage device such as a hard disk, or a non-volatile memory such as EEPROM, flash memory, MRAM, FeRAM, and PRAM.

(Communication interface 711)
The communication interface 711 is a communication unit included in the voice recognition server 201, and communicates with an external device via a network (or directly). The communication interface 711 may be a wireless communication interface or a wired communication interface.

<1.4.2 Functional configuration>
Next, an example of the functional configuration of the voice recognition server 201 according to the present embodiment will be described. FIG. 21 is a block diagram showing an example of the functional configuration of the voice recognition server 201 according to this embodiment. Referring to FIG. 21, the voice recognition server 201 includes a communication unit 510, a storage unit 520, and a control unit 530.

(Communication unit 510)
The communication unit 510 communicates with other devices. For example, the communication unit 510 is directly connected to the LAN 19 and communicates with each device in the center office 10. Specifically, for example, the communication unit 510 communicates with the microphone 13 and the information management server 200. Further, the communication unit 510 communicates with each device in the satellite office 20 via the external network 30 and the LAN 23.

(Storage unit 520)
The storage unit 520 stores programs and data for the operation of the voice recognition server 201. Specifically, in the present embodiment, the storage unit 520 stores data of a large-scale word list.

(Control unit 530)
The control unit 530 provides various functions of the voice recognition server 201. The control unit 530 includes a phrase extraction unit 531 and a phrase data generation unit 533.

(Word extraction unit 531)
The word/phrase extraction unit 531 refers to the word/phrase list stored in the storage unit 520, and extracts a word/phrase from the voice data acquired from the terminal device 100 or the microphone 13 via the information management server 200. The word/phrase extraction unit 531 may sequentially extract words/phrases from the voice data upon receiving the voice data, or when the received voice data reaches a predetermined amount (for example, conversation for 5 minutes). May be extracted from the received voice data). Further, the phrase extracting unit 531 may extract the phrase using a phrase list stored in advance in the storage unit 520 and associated with the speaker who uttered the phrase. By doing so, information about the tendency of words and phrases spoken by the speaker is accumulated, and the phrase extraction unit 531 uses the accumulated information to give priority to words and phrases with a high frequency of speech by the speaker. Can be extracted. In addition, the phrase extraction unit 531 extracts a phrase so that a specific phrase is not extracted (a phrase that is not related to work (eg, a phrase such as “game”) is not extracted). At this time, filtering may be performed.

(Word data generation unit 533)
The word/phrase data generation unit 533 generates data of the result of extraction (recognition) of the word/phrase extracted by the word/phrase extraction unit 531 and transmits the data to the information management server 200 via the communication unit 510.

<1.5 Process flow>
Next, an example of information processing according to this embodiment will be described with reference to FIG. FIG. 22 is a flowchart showing an example of a schematic flow of information processing according to this embodiment. 22 includes steps S401 to S413.

(Step S401)
The position acquisition unit 183 determines whether the position in the captured image on the display screen of the captured image in the real space is designated by the user. More specifically, for example, the position acquisition unit 183 determines whether the position in the close-up captured image 71 is designated by the user. If the position is specified, the process proceeds to step S403. On the other hand, if the position is not designated, step S401 is repeated.

(Step S403)
The position acquisition unit 183 acquires the position in the captured image specified by the user.

(Step S405)
The object selection unit 185 acquires the data of the three-dimensional virtual space 90 corresponding to the real space from the storage unit 170.

(Step S407)
The object 91 arranged in the three-dimensional virtual space 90 is selected based on the acquired position in the captured image.

(Step S409)
The ID acquisition unit 187 acquires the communication ID corresponding to the selected object 91 as the communication ID on the receiving side.

(Step S411)
The ID acquisition unit 187 acquires the communication ID of the calling side (that is, the communication ID of the terminal device 100).

(Step S413)
The telephone unit 189 makes a telephone call using the communication ID on the receiving side. Thereafter, the telephone unit 189 executes various processes for making a call, and the process ends.

As described above, the information processing according to this embodiment is executed. Furthermore, an example of the activation process performed before the start of the information processing will be described with reference to FIG. FIG. 23 is a flowchart showing an example of a schematic flow of a startup process according to this embodiment. FIG. 23 includes steps S501 to S507.

(Step S501)
The activation process of the softphone 851 is executed. This activates the softphone 851.

(Step S503)
The registration process for the softphone 851 is executed. For example, as one of the registration processes of the softphone 851, registration with the PBX 40 (for example, SIP REGISTRATION) is performed.

(Step S505)
The activation process of the ultra-realistic client 853 is executed. For example, the camera 11, the microphone 13, the sensor 15, the media distribution server 17, the information management server 200, and the like used in the ultra-realistic client 853 are specified.

(Step S507)
The bird's-eye view mode process of the ultra-realism client 853 is executed. Then, the series of start-up processing ends.

Next, an example of the communication control process according to the present embodiment will be described with reference to FIG. FIG. 24 is a sequence diagram showing an example of a schematic flow of the communication control process according to the present embodiment. In the communication control process of FIG. 24, the information management server 200 uses the information regarding the conversation between the terminal device 100C of the person C and the terminal device 100D of the person D who started the two-party call as the COMM link 86 and the COMM word 87. It includes processing until distribution to the terminal device 100U of the user who is a third party. Further, the communication control process of FIG. 24 includes a process until the user enters the two-way call by designating and inputting the COMM link 86 from the terminal device 100U. Specifically, FIG. 24 includes steps S601 to S613.

(Step S601)
A two-party call is started between the terminal device 100C and the terminal device 100D via the information management server 200. The series of information processing described in steps S401 to S413 may be used to start the two-party call.

(Step S603)
The information management server 200 acquires the phrase data extracted by the voice recognition server 201 from the voice data in the two-way call between the terminal device 100C and the terminal device 100D. Furthermore, the information management server 200 uses the acquired word/phrase data to generate weighted word/phrase data that reflects position information, statistical weighting, and the state of speech.

(Step S605)
The information management server 200 transmits the weighted word/phrase data regarding the two-way call between the terminal device 100C and the terminal device 100D to the terminal device 100U.

(Step S607)
The terminal device 100U receives the weighted word/phrase data and displays the COMM link 86 and the COMM word 87 on the screen of the display unit 150.

(Step S609)
The terminal device 100U acquires a user input that specifies the COMM link 86 that connects the objects 91 corresponding to the displayed terminal devices 100C and 100D.

(Step S611)
The terminal device 100U transmits, to the information management server 200, data relating to a user input designating the COMM link 86 (for example, object IDs of two or more speakers corresponding to the conversation object described above).

(Step S613)
The information management server 200 receives the data related to the user input designating the COMM link 86, and establishes a call with the terminal device 100U in a two-party call session between the terminal devices 100C and 100D corresponding to the COMM link 86. to add. Then, a three-party call session between the terminal device 100C, the terminal device 100D, and the terminal device 100U is started.

As described above, in the present embodiment, since the COMM link 86 connecting the callers is displayed, the user visually recognizes the COMM link 86 to identify who is in a plurality of remote locations in the distributed environment. It is possible to intuitively understand with whom the conversation occurred. Further, in the present embodiment, since the main part of the call content is displayed as the COMM word 87, even a user who is in a remote place and cannot hear the voice can see the screen on which the COMM word 87 is displayed, It becomes possible to roughly understand the contents. Then, the user can participate in the conversation smoothly by performing a simple designation operation on the COMM link 86 when the user becomes interested in the conversation content after roughly understanding the conversation content. That is, according to the present embodiment, the user can grasp the occurrence of a conversation at a remote place and the rough content of the conversation in a distributed environment, and further intuitively recognize who is in a call state. It is possible to

<2. Second Embodiment>
Next, in the second embodiment of the present invention described below, in a mixed environment of a distributed environment and a common room environment, such as a distributed office, a user can talk to a plurality of speakers who are already talking. This processing is premised on that the speaker exists in the same room environment as any speaker.

Specifically, in the first embodiment described above, a communication event, which is a conversation that is difficult to notice in a distributed environment, is visualized as a COMM link 86 and a COMM word 87 so that the user can communicate in a remote area. I was trying to notice. However, for example, in an environment in which a distributed environment and a room environment coexist, such as a distributed office, it is unlikely that a user is located in an environment different from all speakers of a plurality of speakers who are already talking. Without limitation, it may be located in the same room environment as one of the callers. In this case, since the user is located in the same room environment, the user can naturally hear the content of the call as a surrounding conversation, and therefore the conversation is also presented as the COMM word 87 on the user's terminal device 100U. Then, the same information is provided to the user in duplicate based on the viewed information and the displayed information. Further, such double provision of information would rather confuse the user's thinking. Therefore, in the present embodiment, by avoiding the generation and distribution processing of the COMM word 87 related to the utterance of the person in the same room environment, it is possible to prevent the double provision of the same information and avoid the confusion of the user's orientation. I do. Furthermore, in the present embodiment, by avoiding the generation and distribution processing of the COMM word 87, a part of the processing in the system of the present embodiment can be reduced and the processing can be speeded up.

<2.1 Information management server configuration>
<2.1.1 Functional configuration>
An example of the functional configuration of the information management server 200A according to the present embodiment will be described with reference to FIG. FIG. 25 is a block diagram showing an example of the functional configuration of the information management server 200A according to this embodiment. The information management server 200A includes a communication unit 210, a storage unit 220, and a control unit 230, similar to the information management server 200 of the first embodiment described above. Further, the control unit 230 includes the COMM link distribution unit 231, the extracted word/phrase data management unit 232, the weighting calculation unit 233, the COMM word distribution unit 234, and the statement status calculation unit 235, as in the first embodiment. In addition, the control unit 230 further includes a position-linked distribution control unit 236. Therefore, here, the description of the functional units similar to those of the first embodiment will be omitted, and only the position-linked distribution control unit 236 will be described.

(Position linked delivery control unit 236)
The position-linked delivery control unit 236 compares the position of the person (speaker) corresponding to the call already made with the position of the third party user who is not participating in the call, and the third party user concerned. It controls whether or not to deliver the COMM word 87 to the terminal device 100U. More specifically, the position-linked delivery control unit 236 compares the position information of the person (speaker) related to the voice of the call with the position information of the terminal device 100U that is not involved in the call, and When the position information is within a predetermined distance or both exist in the same room environment of the same base, the control process related to the generation or distribution of the COMM word 87 is stopped.

<2.2 Process Flow>
Subsequently, an example of information processing according to the present embodiment will be described with reference to FIG. FIG. 26 is a flowchart showing an example of a schematic flow of information processing according to the present embodiment. FIG. 26 includes steps S701 to S707, and step S701 is started after step S601 of FIG. 24 of the first embodiment.

(Step S701)
When the information management server 200 detects that the two-party call between the terminal device 100C and the terminal device 100D is started, the information management server 200 proceeds to the next step S703. If the information management server 200 does not detect that the two-party call has started, step S701 is repeated.

(Step S703)
The position-linked delivery control unit 236 detects the position of the person who uttered the phrase related to the phrase data extracted from the call, and the position of the terminal device 100U that is not involved in the call (the position of the user who is not participating in the call. ) Is calculated.

(Step S705)
The position-interlocked delivery control unit 236 compares the distance calculated in step S703 with a predetermined distance. If the calculated distance is shorter than the predetermined distance, the process ends. On the other hand, when the calculated distance is longer than the predetermined distance, the process proceeds to step S707. The predetermined distance is, for example, a distance between the speaker and the user, which is assumed when the user is in the same room, or when one speaker speaks in a real space. , The distance between the position in the real space where the user can hear the utterance and the speaker.

(Step S707)
It progresses to step S603 of FIG. That is, words and phrases related to conversation are extracted, and the extracted words and phrases are weighted.

As described above, in the present embodiment, the position-interlocked delivery control unit 236, when positioned in the same room environment as one of the talkers or in the vicinity of the talker, the COMM regarding the utterance of the person in the same room environment. Processing for avoiding the generation and distribution processing of the word 87 is performed. Specifically, the position-interlocked delivery control unit 236 performs a process that skips the process of step S603 in FIG.

For example, when the second embodiment is described with reference to the example of FIG. 15, when the third party user is at the base B: Osaka office, the person F at the base B is in the same room as the third party user, and the conversation naturally occurs. Is heard. In such a case, in the present embodiment, for the utterance of the person F in the two-party call between the person E and the person F, the information management server 200A generates the COMM word 87 to generate the terminal of the third party user. The process of transmitting to the device 100U is avoided. Specifically, the COMM word 87L in FIG. 15 is not generated by the information management server 200A or is not delivered to the terminal device 100U, and as a result, is not presented on the display screen of the terminal device 100U.

It should be noted that the present embodiment is not limited to performing the process of avoiding the distribution of the COMM word 87, and for example, the process of avoiding the distribution of the COMM link 86 may be performed. Further, the present embodiment is not limited to the process of avoiding the distribution of the COMM word 87, and for example, the process of reducing the amount of the COMM word 87 to be distributed may be executed.

That is, according to the present embodiment, by avoiding the generation and distribution processing of the COMM word 87 regarding the utterance of the person in the same room environment, it is possible to prevent double provision of the same information and to confuse the thought of the third party user. Can be avoided. Specifically, since the COMM word 87 and the COMM link 86 are not displayed on the display screen of the terminal device 100 of the third party user, the COMM link 86, the COMM word 87, and the captured image 51 that is the background of the superimposed drawing are displayed. Visibility is improved. Furthermore, in the present embodiment, by avoiding the generation and distribution processing of the COMM word 87, a part of the processing in the system of the present embodiment can be reduced and the processing can be speeded up.

<3. Third Embodiment>
A third embodiment of the invention described below, for a conversation that has already started, shows to the parties involved in the conversation which third-party user is interested in the conversation. By doing so, the speaker becomes aware of the presence of the interested third-party user and can attract the third-party user to the conversation, as in a conversation conducted in the same environment.

Specifically, as described above, the act of "conversation" is not always performed by two people, but is often performed by a group of three or more people. In such a case, in addition to the pattern in which all the members of the group are all available at the beginning of the conversation, the surrounding person notices the existence of the started conversation (Awareness), and the person comes to the place of the conversation. There is a pattern in which the number of members of the group having the conversation increases as a result of joining the group.

For example, in such a situation, in the same room environment, the two parties in the conversation naturally notice the presence of a third party who stands near the two parties in the conversation and is interested in the content of the conversation. Let's do it. Further, the third party does not always stand in the same place, and if the interest in the conversation content becomes stronger, the third party will be closer to the two parties in the conversation, and conversely if the interest in the conversation content is lost. I leave the two in conversation and go away. In this way, the third party actively changes its position according to the degree of interest in the conversation content. That is, the degree of interest in the conversation content of the third party is manifested as a spatial “distance” between the two parties in the conversation and the third party. Further, if the distance is short, the two parties in the conversation are more likely to notice the existence of the third party, and conversely, if the distance is longer, the second party is less likely to notice the third party. Then, two people in the conversation notice the third party approaching and draw in the conversation, or the third party gradually approaches while attending the conversation and participates in the conversation as it is. Person conversation will begin.

However, since the conventional softphone products that have been used in the distributed environment conventionally assume a pattern in which all the members are aligned from the beginning of the conversation, as described above, the softphone product is not available to the third party. The processing is not performed in consideration of the degree of interest and the change in the mutual positional relationship between the speaker of the conversation and the third party according to the degree of interest. Therefore, when a conventional softphone product is used, compared to a conversation in the same room environment, a third party is less likely to join in the middle of the conversation, and the two parties in the conversation can suddenly speak to the third party. Therefore, the two parties may be surprised and the call between the third party and the second party may not proceed smoothly.

Therefore, in the present embodiment, the degree of interest in a conversation of a third party at a plurality of remote locations can be intuitively set and input by a third party as, for example, a virtual "distance", and the two parties who are engaged in the conversation. Is provided with a function to notify that there is a third party who is interested in the conversation according to the distance (degree of interest). Furthermore, the present embodiment provides a function that allows a third party to acquire information on the conversation content related to the conversation according to the distance (degree of interest).

<3.1 Configuration of Terminal Device>
<3.1.1 Functional configuration>
An example of the functional configuration of the terminal device 100A according to the present embodiment will be described with reference to FIG. FIG. 27 is a block diagram showing an example of the functional configuration of the terminal device 100A according to this embodiment. Referring to FIG. 27, the terminal device 100A, like the terminal device 100 according to the first embodiment, has a communication unit 110, an input unit 120, an imaging unit 130, a sound collecting unit 140, a display unit 150, and an audio output unit 160. The storage unit 170 and the control unit 180 are included. Further, as in the first embodiment, the control unit 180 further includes the real space information providing unit 181, the voice output control unit 182, the position acquisition unit 183, the object selection unit 185, the ID acquisition unit 187, the telephone unit 189, and the conversation object. It includes a selection unit 191, a COMM link control unit 193, and a COMM word control unit 195. In addition, the control unit 180 further includes a conversation interest level setting unit 196, a speaker interest ratio setting unit 197, and a conversation interest level notification unit 198. Therefore, the description of the functional units similar to those in the first embodiment is omitted here, and only the conversation interest level setting unit 196, the speaker interest ratio setting unit 197, and the conversation interest level notification unit 198 will be described.

(Conversation interest level setting unit 196)
The conversation interest level setting unit 196 accepts a setting input of a “conversation interest level (interest level)” to be described later from the user's input to the terminal device 100A, displays the result on the display unit 150, and also the conversation interest level. The data regarding to is transmitted to the information management server 200. Here, in the present specification, the “conversation interest degree” indicates the degree of interest felt by the user with respect to a certain call (conversation) that occurs on the information processing system according to the present embodiment in multiple stages. It is a thing. The multi-stage may be 10 stages of 0: no interest to 9: very interested, 100 stages of 0 to 99, or the Visual Analogue. It may be a stepless continuous scale such as Scale. Note that one conversation interest level can be associated with one call (conversation), that is, one COMM link 86, and further associated with one or a plurality of COMM words 87 related to the COMM link 86.

The operation of setting the conversation interest level by the user of the terminal device 100A may be, for example, designating and inputting a numerical value from the input unit 120. However, as a more intuitive input method, the user arranges his/her virtualized body (hereinafter, “third party object (user object)”) on the COMM link 86, and the third party object An operation method for inputting the conversation interest level based on the virtual distance between the position of the position and the position of the COMM link 86 will be described below with reference to FIG. FIG. 28 is an explanatory diagram for explaining an example of the third party object 94 and the virtual distance 97 in the three-dimensional virtual space 90 corresponding to the distributed office.

In FIG. 28, a three-dimensional virtual space 90 corresponding to the distributed office is shown. An object 91E corresponding to the person E and an object 91F corresponding to the person F are arranged in the three-dimensional virtual space 90. Further, a COMM link 86 corresponding to a two-way call between the persons E and F is generated and arranged between the objects 91E and 91F. It should be noted that the objects 91 at both ends, which are the bases of the COMM link 86, may exist at different bases in the real space. In this case, the objects 91 at both ends have absolute position coordinates in the same three-dimensional virtual space 90, and the relative positional relationship between the objects 91 may be calculated from the overall position coordinates. Alternatively, the positional relationship between the bases is predetermined, and the information management server 200 uses the positional relationship to calculate the relative positional relationship of the objects 91 corresponding to the persons existing at the bases when the bases are crossed. You may. When the user installs the third party object 94U on the COMM link 86, the third party object 94U is newly placed in the three-dimensional virtual space 90. The position of the third party object 94U can be freely set only by the user's horizontal position designation.

When the third-party object 94U is placed in the three-dimensional virtual space 90, the three-dimensional center-of-gravity position 92U of the third-party object 94U and the speaker interest ratio reflection position 96U of the COMM link 86 (speaker interest ratio reflection) The initial position of the position 96U is, for example, the middle point of the COMM link 86). Further, the virtual distance 97U between the position of the three-dimensional barycentric position 92U and the position of the three-dimensional barycentric position 96U is calculated by the information management server 200B and the terminal device 100A described later. Therefore, when the user (third party) changes the placement position of the third party object 94U, the size of the virtual distance 97U is correspondingly changed.

As described above, the concept that the degree of interest in the conversation is high or low is compatible with the concept that the third party in the conversation is close or far away from the position where the conversation is occurring. Therefore, when the degree of interest in conversation is displayed according to the distance, the degree of interest can be intuitively grasped by the display. Further, as described above, the terminal device 100A can display the captured image 51 and the map image 69 on the display unit 150. Since the captured image 51 and the map image 69 are information obtained by projecting the structure of the real space, they are suitable for superimposing information such as position and distance for graphical display or touch designation input. Therefore, in the information processing system according to the present embodiment, the third party object 94 described above is used rather than inputting a numerical value with a keyboard or the like and displaying the value of the virtual distance set as a result as a numerical value. Is projected and displayed on the captured image 51 and the map image 69, and the displayed third party object 94 (specifically, the screen display body) is operated, and the position is changed, so that the conversation is started. It is preferable to adopt the method of inputting the conversation interest level of. By adopting such a method, the user can intuitively input the conversation interest level, and the convenience of the user is improved.

In the present embodiment, a user can set one third-party object 94 and virtual distance 97 for one COMM link 86. However, the present embodiment is not limited to this. For example, when one third party object 94 is set for a plurality of COMM links 86 and the position of the third party object 94 is changed, the third party object 94 is changed according to the positional relationship with the plurality of COMM links 86. The plurality of virtual distances 97 to the plurality of COMM links 86 may be changed in conjunction with each other.

Next, an example of a third-party object and a virtual distance displayed on the display screen of the terminal device 100A according to the present embodiment will be described with reference to FIGS. 29 and 30. FIG. 29 is an explanatory diagram illustrating an example of a third-party object and virtual distance displayed on the display screen of the terminal device 100A according to the present embodiment. FIG. 30 is an explanatory diagram for explaining another example of the third party object 94 and the virtual distance 97 displayed on the display screen 57 of the terminal device 100A according to the present embodiment.

In FIG. 29, the display screen 56 is shown. In the real space, the persons E and F are making a two-way call, and the COMM link 86H3 corresponding to the two-way call is on the captured images 51A and 51B on the display screen 56, or the COMM link 86H2 is the map image 69A. It is displayed on 69B. The user (third party) can designate and input the position of the third party object 94U at the two-dimensional display position on the captured images 51A, 51B, 51Z with respect to the COMM link 86H3. Corresponding to the input, the screen display body (icon) 98U3 of the third party object 94U, the screen display body 101U3 of the speaker interest ratio reflecting position 96U, and the screen display body of the virtual distance 97U are displayed on the captured image 51. 103U3 is displayed.

Similarly, the user can specify and input the position of the third-party object 94U at the two-dimensional display position on the map images 69A, 69B, and 69Z for the COMM link 86H2. Then, in response to the input, the screen display body 98U2 of the third party object 94U, the screen display body 101U2 of the speaker interest ratio reflecting position 96U, and the screen display body 103U2 of the virtual distance 97U are displayed on the map image 69. To be done. The screen display 98U3 or 98U2 of the third party object 94U may be, for example, an icon in which the name of the user is drawn, or a face image of the user as illustrated in FIG. The face image of the user may be, for example, an image captured by the image capturing unit 130 of the terminal device 100A of the user and having the face area trimmed by the face detection processing.

The user can drag and operate the screen display body 98U3 or 98U2 of the third party object 94U on the screen, for example, to specify and input the position in the three-dimensional virtual space 90 described above. The two-dimensional display position on the screen of the screen display body 98U3 or 98U2 is also changed in response to the user's designation input of the position, and the length of the virtual distance 97U and the display of the screen display body 103U3 or 103U2 are correspondingly changed. The length and end point position are changed. Then, the changed virtual distance 97U corresponds to the degree of conversation interest in the conversation regarding the COMM link 86H2 of the user. In the present embodiment, the correspondence relationship between the virtual distance 97 and the conversation interest level appropriately corresponds to the maximum/minimum value of the interest level and the maximum/minimum value of the virtual distance 97. If so, it may be set arbitrarily. Then, the conversation interest level setting unit 196 can acquire the conversation interest level of the user based on the operation of the third party object 94 by the user as described above.

A display screen 57 is shown in FIG. 29, the COMM link 86H3 or 86H2, the screen display body 98U3 or 98U2 of the third party object 94U, and the screen display body 103U3 or 103U2 of the virtual distance 97U are displayed on the display screen 57. However, in FIG. 30, the screen display body 98U3 or 98U2 of the third party object 94U is set at a position closer to the COMM link 86H3 or 86H2 as compared with the example of FIG. Further, correspondingly, the display length and the end point position of the screen display body 103U3 or 103U2 with the virtual distance of 97U are changed.

Further, in FIG. 30, the number of COMM words 87 displayed near the COMM link 86H3 or 86H2 is larger than that in FIG. This reproduces in the system according to the present embodiment the phenomenon that in a shared room environment, the closer a third party is to the place of a two-party conversation, the more the content of the conversation is heard. That is, when the user is interested in conversation and sets the screen display body 98U3 or 98U2 of the third party object 94U closer to the COMM link 86H3 or 86H2, the screen display body 103U3 or 103U2 with a virtual distance of 97U is displayed. Indicates that the length of the is shorter and the degree of interest in conversation is higher. As a result, the number of COMM words 87 displayed increases. The processing for changing the number of display of the COMM words 87 according to the size of the virtual distance 97 (that is, the degree of conversation interest) uses the weight threshold value in the display of the weighted word/phrase data by the COMM word control unit 195 described above. In the processing, the COMM word control unit 195 may be realized by changing the size of the threshold value in accordance with the size of the virtual distance 97.

It should be noted that the present embodiment is not limited to changing the number of displayed COMM words 87 in correspondence with the size of the virtual distance 97 based on the acquired conversation interest level. For example, in the present embodiment, the voice output control unit 182 controls the volume of the voice of the conversation, which is acquired from the microphone 13 or the sound collection unit 140 and presented from the user's terminal device 100A via the information management server 200B, by the conversation interest. It may be changed according to the degree, that is, the size of the virtual distance 97.

(Speaker interest ratio setting unit 197)
The speaker interest ratio setting unit 197 performs a process of setting a “speaker interest ratio (ratio of interest rate)”, which will be described later, from the user's input to the terminal device 100A, and displays the result on the display unit 150. The data regarding the speaker interest ratio is transmitted to the information management server 200B described later. Further, the speaker interest ratio setting unit 197 changes the speaker ratio of the COMM word 87 displayed by the COMM word control unit 195. Here, the “speaker interest ratio” means, in the present specification, a conversation felt by the user with respect to each speaker (and the utterance content thereof) in a certain call generated on the information processing system according to the present embodiment. It is a ratio of high interest level. For example, the speaker interest ratio takes a value between 0.0 and 1.0 for each speaker of one conversation, and the total value of the speaker interest ratios of a plurality of speakers is 1.0. More specifically, a case where the user is interested in the conversation between the person E and the person F will be described as an example. The user sees the display of the COMM word 87 related to the conversation and has a higher interest in the remark of the person E in the conversation than in the remark of the person F in the conversation (specifically, for the person E. Is assumed to have about four times the interest of the person F). In such a case, for example, the speaker interest ratio is expressed as 0.8 for the person E and 0.2 for the person F when the overall conversation interest degree for the conversation is 1.0. be able to. In this way, by the user setting the speaker interest ratio, the display unit 150 displays the COMM word 87 (that is, the phrase spoken by the speaker) for each speaker to the user according to the set ratio. Can be displayed on the screen.

An example of setting the speaker interest ratio reflection position 96 indicating the speaker interest ratio according to the present embodiment will be described below with reference to FIGS. 31 and 32. FIG. 31 is an explanatory diagram for explaining an example of setting the speaker interest ratio reflection position (reference point) 96 in the three-dimensional virtual space 90 corresponding to the distributed office. FIG. 32 is an explanatory diagram for explaining an example of setting the speaker interest ratio reflection position 96 displayed on the display screen 58 of the terminal device 100A according to the present embodiment.

In FIG. 31, a three-dimensional virtual space 90 corresponding to the distributed office is shown. As compared with FIG. 28, the position where the speaker interest ratio reflecting position 96U is set is different. A method for the user to set the speaker interest ratio reflection position 96U will be described later. In FIG. 28, the speaker interest ratio reflecting position 96U is at the midpoint of the line segment of the COMM link 86 as the initial position, that is, the user's speaker interest ratio to the person E is 0.5, and the talk to the person F is 0.5. The speaker interest ratio is also 0.5, and the speaker interest ratio is the same for both persons E and F. On the other hand, in FIG. 31, the speaker interest ratio reflecting position 96U is at a position where the distance length is 7:3 from the object 91E and the object 91F on the line segment of the COMM link 86, that is, in inverse proportion to the distance length, It is shown that the user's speaker interest ratio to the person E is 0.3, and the speaker interest ratio to the person F is 0.7.

A display screen 58 is shown in FIG. As compared with FIG. 30, the position where the speaker interest ratio reflecting position 96U is set is different, and accordingly, the display length and the end point position of the screen display bodies 103U3 and 103U2 at the virtual distance 97U of FIG. 32 are received. Is different from FIG. The speaker interest ratio reflecting position 96U is the midpoint of the line segment of the COMM link 86 in FIG. 30, but in FIG. 32 the distance is 7:3 from the object 91E and the object 91F in accordance with the position setting state of FIG. It is set to a position that makes it long. That is, the speaker interest ratio reflection position 96U in FIG. 32 is inversely proportional to the distance length, and the user has a speaker interest ratio of 0.7 for the person E and a speaker interest ratio of 0.3 for the person F. It is shown that.

Further, in FIG. 32, the distribution of the number of COMM words 87 displayed near the COMM link 86H3 or 86H2 is different from that in FIG. That is, the number of COMM words related to the person E whose speaker interest ratio has decreased (0.5→0.3) (the number of COMM words displayed in the vicinity of the person E) decreases, and the speaker interest ratio increases. (0.5→0.7) The number of COMM words related to the person F (the number of COMM words displayed in the vicinity of the person F) is increasing. The position of the speaker interest ratio reflecting position 96U may be set when the user inputs, for example, one point on the COMM link 86H3 or 86H2 by touch designation.

Further, in the present embodiment, the position of the speaker interest ratio reflection position 96U may be set by another method. Another example will be described below with reference to FIG. 33. FIG. 33 is an explanatory diagram for explaining an example of the display screen 62 when the speaker interest ratio reflection position 96 is set by the third party object 94. In detail, FIG. 33 shows a partial area on the display screen 50 of the terminal device 100A, and in the left and right views of FIG. 33, the orientation of the third party object 94 and the speaker interest ratio reflection set by it are reflected. Two examples in which the position 96 is different are illustrated.

In detail, the screen display body 98U2 of the third party object 94U of FIG. 33 has one pointed portion 105 around the face image of the third party user. Then, the third party user who specifies and inputs the third party object 94U can change the orientation of the tip portion 105 by performing a touch rotation operation using, for example, two or more fingers. Furthermore, when the line segment of the COMM link 86H2 formed between the object of the person E and the object of the person F exists on the extension line 99 in the direction of the tip portion 105 and intersects the extension line 99 and the COMM link 86H3, The speaker interest ratio setting unit 197 acquires the position of the intersection as the position of the speaker interest ratio reflection position 96U. Since the position of the speaker interest ratio reflection position 96U is set in this manner, the speaker interest ratio setting unit 197 can calculate the speaker interest ratio for the speaker of the user's conversation. For example, looking at FIG. 33, the screen display body 98U2 of the third party object 94U in the right diagram is rotated rightward as compared with the screen display body 98U2 of the third party object 94U in the left diagram of FIG. Also, the position of the speaker interest ratio reflection position 96U (screen display body 101U2) corresponding to that direction is located more rightward in the right figure than in the left figure.

In addition, the present embodiment is not limited to the user operating the COMM link 86 and the third party object 94 to set the speaker interest ratio. For example, in the present embodiment, the user's line of sight with respect to each COMM word 87 may be recognized, and the speaker interest ratio may be calculated using the recognized frequency (for example, the person E and the person F talk to each other). An example will be described in which the user sees the COMM word 87 related to the phrase spoken by the person E and the COMM word 87 related to the phrase spoken by the person F displayed on the screen. By using the line-of-sight detection device that detects the line-of-sight, it is possible to detect and count the line-of-sight of the user directed to each COMM word 87. Therefore, the user can detect the COMM word 87 related to the phrase spoken by the person E. On the other hand, when the line of sight is dropped seven times and the line of sight is dropped three times with respect to the COMM word 87 related to the phrase uttered by the person F, the speaker interest ratio to the person E is 0.7, and the talk to the person F is Person interest ratio can be 0.3).

(Conversation interest degree notification unit 198)
The conversation interest degree notification unit 198 receives the information regarding the conversation interest degree from the information management server 200, and displays the information regarding the conversation interest degree on the screen of the display unit 150 of the terminal device 100 of the person who has the conversation in a push notification display. To do. For example, when the user's terminal device 100U sets the conversation interest level for a two-party call between the person devices C and D between the terminal devices 100C and 100D, the conversation interest levels of the terminal devices 100C and 100D. The notification unit 198 receives the conversation interest level data set in the terminal device 100U from the information management server 200B described later. Furthermore, the conversation interest level notification unit 198 of each of the terminal devices 100C and 100D has an attention intensity corresponding to the magnitude of the value of the conversation interest level, and is interested in the conversation on the screen of the display unit 150 of each of the terminal devices 100C and 100D. Display a notification informing you of the existence of a user with. Here, the notification is, for example, a pop-up window (display). Further, the alerting intensity is set according to the conversation interest level described above, and the greater the conversation interest level, the greater the alertness intensity. The display form of the notification is changed according to the alert strength. Specifically, the larger the alerting strength, the larger the display size of the popup window. Further, the present embodiment is not limited to changing the display size of the window according to the alert strength, but the display position of the window is changed (the closer to the center of the screen the alert strength is However, the contrast between the window and the background may be changed, the window color, the type of window display animation (motion), and the like may be changed. Further, the notification is not limited to the screen display, and may be, for example, a sign sound or the like indicating the presence of a user who is interested in conversation. In this case, the sound volume, frequency distribution, rhythm, etc. of the sign sound are called up. It may be changed according to the strength.

Hereinafter, with reference to FIG. 34, an example of the display of the notification including the information regarding the conversation interest degree displayed on the display screen of the terminal device 100A according to the present embodiment will be described. FIG. 34 is an explanatory diagram for describing an example of a display of a notification including information regarding the conversation interest degree displayed on the display screen of the terminal device 100A according to the present embodiment. The display screen 59 on the upper side of FIG. 34 is an example of the display of the notification with a high alerting intensity, and the display screen 66 on the lower side of FIG. 34 is an example of the display of the notification with a low alerting intensity. On the display screen 59, the notification window 107 is displayed in a larger size and closer to the center of the screen than the display screen 66, so that the user can easily notice the appearance of the notification. Further, the notification window 107 may display the size of the virtual distance in the three-dimensional virtual space 90 (for example, “1.5 m” is displayed in the window 107 of the display screen 59).

Although FIG. 34 shows an example in which one third party is interested in the corresponding conversation, a plurality of third parties are interested in the same conversation. In that case, the information to that effect may be shown in one window, or the window may be displayed for each third party. Further, in FIG. 34, the window is superimposed on the display screen displayed in the overhead view mode, but the present embodiment is not limited to such a display form. For example, the window may be superimposed on the display screen displayed in the conversation mode.

<3.2 Information management server configuration>
<3.2.1 Functional configuration>
Next, an example of the functional configuration of the information management server 200B according to the present embodiment will be described with reference to FIG. FIG. 35 is a block diagram showing an example of the functional configuration of the information management server 200B according to this embodiment. Referring to FIG. 35, the information management server 200B includes a communication unit 210, a storage unit 220, and a control unit 230, similar to the information management server 200 of the first and second embodiments. Further, as in the second embodiment, the control unit 230 further includes the COMM link distribution unit 231, the extracted word/phrase data management unit 232, the weighting calculation unit 233, the COMM word distribution unit 234, the statement status calculation unit 235, and the position-linked distribution. The control unit 236 is included. In addition, the control unit 230 further includes a conversation interest level control unit 241. Therefore, the description of the functional units similar to those in the first and second embodiments will be omitted here, and only the conversation interest level control unit 241 will be described.

(Conversation interest degree control unit 241)
The conversation interest level control unit 241 performs processing related to the amount of information provided in the COMM word 87 corresponding to the conversation interest level and the delivery of the notification corresponding to the conversation interest level. Specifically, the conversation interest degree control unit 241 includes an interest degree control unit 242, a display body control unit 243, an interest conversation information amount control unit 244, and a conversation interest degree notification transmission unit 245.

(Interest control unit 242)
The interest level control unit 242 acquires the input of the conversation interest level for the call of the user, and the acquired conversation interest level and the user object ID (communication ID (communication identification information)) for the COMM link 86. ) With. Further, the associated conversation interest degree is distributed to the terminal device 100 via the COMM link distribution unit 231 and the communication unit 210 together with the COMM link 86. By doing so, the object IDs of two or more speakers corresponding to the COMM link 86 are associated with the object IDs of the user. Then, in the present embodiment, the presence of the user can be notified to the terminal device 100 of the speaker by performing control by referring to the object ID associated with one COMM link 86. .. Further, the interest degree control unit 242 acquires the speaker interest ratio of the user, and the acquired speaker interest ratio and the object ID (communication ID (communication identification information)) of the user for the COMM link 86. ) And may be associated. Also in this case, the associated speaker interest ratio is distributed to the terminal device 100 through the COMM link distribution unit 231 and the communication unit 210 together with the COMM link 86.

(Display control unit 243)
The display body control unit 243 generates the screen display body 98 related to the user when the input operation of the conversation interest level in the conversation is performed by the user. Further, the display body control unit 243 determines the virtual positional relationship between the COMM link 86 and the screen display body 98 corresponding to the conversation based on the acquired conversation interest level. Specifically, when the user performs an operation of inputting the conversation interest level in the conversation, the display body control unit 243 displays the third-party object relating to the user on the three-dimensional virtual space 90 described above. 94 is arranged. At this time, the virtual distance between the COMM link 86 in the three-dimensional virtual space 90 and the third party object 94 is determined based on the conversation interest level. Furthermore, the display body control unit 243, in the case where the input operation of the speaker interest ratio is performed by the user, the speaker interest at the position corresponding to the speaker interest ratio on the COMM link 86 in the three-dimensional virtual space 90. The ratio reflection position 96 is arranged.

(Interested conversation information amount control unit 244)
The interest conversation information amount control unit 244 controls the COMM word distribution unit 234 based on the conversation interest level and the speaker interest ratio. More specifically, the interest conversation information amount control unit 244 sets a predetermined threshold value, which is used when the COMM word 87 is distributed by the COMM word distribution unit 234 and is compared with a numerical value of the weight related to the COMM word 87, as a conversation interest. Change based on degree and speaker interest rate.

(Conversation interest degree notification transmission unit 245)
The conversation interest level notification transmitting unit 245 stores the data of the notification information corresponding to the conversation interest level (or the speaker interest ratio) set by the user in the object 91 located at both ends of the COMM link 86 in which the conversation interest level is set. It is transmitted to the terminal device 100 corresponding to the communication ID. The data of the notification information corresponding to the conversation interest level includes the conversation interest level, the size of the virtual distance 97, the communication ID of the third-party user who has set the conversation interest level, and the like.

<3.3 Process flow>
Subsequently, an example of information processing according to the present embodiment will be described with reference to FIG. 36. FIG. 36 is a sequence diagram showing an example of a schematic flow of conversation interest level setting, communication information amount change of the COMM word 87, and notification processing according to the present embodiment. FIG. 36 includes steps S801 to S843. Note that steps S801 to S807 in FIG. 36 are the same processes as steps S601 to S607 in FIG. 24, respectively, and description thereof will be omitted here. Therefore, after step S801, step S803, step S805, and step S807 are sequentially performed, the following step S821 is performed.

(Step S821)
The terminal device 100U acquires the user input for setting the conversation interest level or the speaker interest ratio for the COMM link 86 that connects the displayed terminal devices 100C and the objects corresponding to the terminal devices 100D.

(Step S823)
The terminal device 100U transmits the data regarding the conversation interest level and the speaker interest ratio set for the COMM link 86 to the information management server 200B.

(Step S825)
The information management server 200B receives the data regarding the conversation interest level and the speaker interest ratio set for the COMM link 86, and manages the data in association with the identification information of the COMM link 86.

(Step S827)
The information management server 200B transmits, to the terminal device 100U, data regarding the COMM word 87 having an information amount corresponding to the degree of conversation interest or the degree of speaker interest.

(Step S829)
The terminal device 100U changes the state of the COMM word 87 displayed on the display unit 150 by using the received data regarding the COMM word 87 having the information amount corresponding to the magnitude of the conversation interest level or the speaker interest ratio.

(Step S831, Step S833)
The information management server 200B transmits the data of the notification information corresponding to the conversation interest level to the terminal devices 100C and 100D.

(Step S835, Step S837)
The terminal devices 100C and 100D use the received notification information data corresponding to the conversation interest level to cause the display unit 150 to display the notification at the alerting intensity corresponding to the conversation interest level.

(Step S839)
From the notification, the terminal device 100C (or 100D) acquires the input of the lead-in to the call by the person C (or person D) who notices the presence of the third-party user.

(Step S841)
The terminal device 100C transmits to the information management server 200B the data (which may include the identification information of the third party user) regarding the input of the pull-in to the call. Then, the process proceeds to step S809.

(Steps S809 and S811)
Step S809 and subsequent step S811 are the same processes as step S609 and step S611 of FIG. 24, respectively, and description thereof will be omitted here. If either one of step S811 and step S841 is executed, the process may proceed to step S843.

(Step S843)
The information management server 200B receives the data regarding the input of the call pull-in or the data regarding the user input for designating the COMM link 86. Then, the call of the terminal device 100U is newly added to the two-party call session between the terminal device 100C and the terminal device 100D corresponding to the COMM link 86, and the three-party call between the terminal device 100C, the terminal device 100D, and the terminal device 100U. Session will start.

As described above, in the present embodiment, the degree of interest in conversation with a third party at a plurality of remote locations can be intuitively set and input by the third party as, for example, a virtual “distance”. A function is provided to the two parties who are conducting a notification according to the distance (conversation interest level) to notify that there is a third party who is interested in the conversation. Furthermore, in the present embodiment, a function is provided that enables a third party to acquire information on the conversation content related to the conversation in accordance with the distance (degree of conversation interest). Therefore, according to the present embodiment, it is possible to notice the presence of a third party who stands close to two people in a conversation and is interested in the content of the conversation, as in a conversation in the same room environment. It is possible to invite a third party to the conversation by using the awareness. As a result, according to the present embodiment, it is possible to smoothly proceed with a conversation between the third party and the two persons, as in a conversation in the same room environment.

Furthermore, in the present embodiment, a function is provided that enables a third party to acquire information on the conversation content related to the conversation in accordance with the distance (degree of conversation interest). Therefore, according to the present embodiment, it is possible to provide an event similar to a conversation in a room environment, in which the closer the third party is to the place of the two-party conversation, the more the content of the conversation can be heard. As a result, the third party can more naturally acquire information about the conversation and can smoothly participate in the conversation based on the acquired information.

<Supplement>
The preferred embodiments of the present invention have been described above in detail with reference to the accompanying drawings, but the present invention is not limited to these examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

For example, the example in which the three-dimensional virtual space 90 of the center office 10 is prepared as the three-dimensional virtual space 90 corresponding to the real space has been described, but the embodiment of the present invention is not limited to this. For example, a plurality of three-dimensional virtual spaces 90 may be prepared. As an example, the three-dimensional virtual space 90 corresponding to the real space may be prepared for each of the plurality of offices. For example, a three-dimensional virtual space 90 for the satellite office 20, home office 20, other center offices 10, etc. may be prepared. In this case, the three-dimensional virtual space 90 of each office may be a three-dimensional virtual space having a size corresponding to the size of each office. Further, the object selection unit 185 of the terminal device 100 may acquire data of a desired three-dimensional virtual space 90 among the plurality of three-dimensional virtual spaces 90. A three-dimensional virtual space 90 other than the office may be prepared.

Further, the example in which the object 91 corresponding to the person is selected only when the person is seated is explained, but the embodiment of the present invention is not limited to this. For example, the object 91 may be selected even when the person is not sitting on the seat. As an example, when a person is sitting in the seat, the communication ID of the communication device installed in the seat is acquired, and when the person is not sitting in the seat, the communication ID of the mobile terminal of the person is acquired. May be obtained.

Further, although the example in which the communication ID is a telephone number has been described, the embodiment of the present invention is not limited to this. The communication ID may be an ID other than a telephone number. As an example, the communication ID may be a softphone ID other than a telephone number. Further, as another example, the communication ID may be an ID for communication other than a telephone call. For example, the communication ID may be a mail address or an ID for a short message. In this case, an email or a short message may be transmitted using the communication ID.

Further, although an example in which the communication ID corresponding to the object 91 is acquired when the object 91 is selected has been described, the embodiment of the present invention is not limited to this. For example, when the object 91 is selected, any identification information corresponding to the object 91 may be acquired. As an example, when the object 91 is selected, the identification information of any of the persons corresponding to the object 91 may be acquired. Then, for example, the communication ID may be acquired from this identification information.

Further, an example in which the object (object selected by the object selection unit 185) 91 arranged in the three-dimensional virtual space 90 corresponding to the real space is a columnar object corresponding to a person has been described. The embodiment is not limited to this. For example, the object may be an object having another shape instead of the cylindrical object. Further, for example, the object 91 may correspond to something other than a person. As an example, the object 91 may correspond to a real space area. Specifically, for example, the object 91 may correspond to a seat and may be arranged at a three-dimensional virtual position corresponding to the position of the seat. Then, when the communication ID of the communication device installed in the seat corresponds to the object 91 and the object 91 is selected, the communication ID may be acquired. Further, the object 91 may correspond to a region wider than the seat and may exist over a three-dimensional virtual range corresponding to the range of the region. Then, when the communication ID of the communication device installed in the area corresponds to the object 91 and the object 91 is selected, the communication ID may be acquired.

Further, although the example in which the position of the captured image is designated by the touch of the user on the display screen has been described, the embodiment of the present invention is not limited to this. For example, the position of the captured image may be designated by the user using an input unit other than the touch panel 820. For example, the position of the captured image may be designated by clicking with a mouse, or may be designated using another input means such as a button or a keyboard.

Further, an example in which the captured image generated by the camera 11, the audio data generated by the microphone 13, and the determination result by the sensor 15 are directly provided to the terminal device 100 by the camera 11, the microphone 13, and the sensor 15, respectively. However, the embodiment of the present invention is not limited to this. For example, another device may provide these data. As an example, any server (for example, a media distribution server) may acquire these data and provide these data to the terminal device 100.

In addition, although an example in which the terminal device 100 has functions such as the position acquisition unit 183, the object selection unit 185, and the ID acquisition unit 187 has been described, the embodiment of the present invention is not limited to this. For example, these functions may be provided by a device other than the terminal device 100. As an example, these functions may be provided by any server. Also, an example has been described in which the information management server 200 has functions such as the COMM link distribution unit 231 and the phrase extraction data management unit 232, but the embodiment of the present invention is not limited to this. For example, these functions may be provided by a device other than the information management server 200. As an example, these functions may be provided by the terminal device 100.

Further, although the example in which the display screen of the captured image of the real space is displayed by the terminal device 100 has been described, the embodiment of the present invention is not limited to this. For example, the display screen may be displayed by another device. As an example, the display screen may be displayed on the display 21 installed in the satellite office 20. Then, the user may specify the position of the captured image included in the display image on the display 21.

Further, although the speaker and the user are explained as being seated in the seat, the embodiment of the present invention is not limited to this, and the speaker and the user may be moving in the real space. For example, the position of each moving speaker and user is specified by communication performed by the terminal device 100 such as a tablet or the image held by the camera 11 that is held by each speaker and user, and based on the specified position. The position of the object in the above-mentioned three-dimensional virtual space 90 may be determined.

Further, the processing steps in the information processing of the present specification do not necessarily have to be executed in time series in the order described in the flowchart. For example, the processing steps in the information processing may be executed in an order different from the order described as the flowchart or may be executed in parallel.

Further, it is possible to create a computer program for causing hardware such as a CPU, a ROM, and a RAM built in an information processing device (for example, a terminal device) to exhibit the same function as each component of the information processing device. A storage medium storing the computer program is also provided.

3 Hands 10 Center office 11, 813 Camera 13, 815 Microphone 15 Sensor 17 Media distribution server 19, 23 LAN
20 Satellite Office 21 Display 30 External Network 31 Icon 40 PBX
50, 55, 56, 57, 58, 59, 60, 62, 66, 70, 80 Display screen 51, 51A, 51B, 51Z Captured image 61 Bird's eye captured image 63, 63A, 63B, 73, 83 Button image 65, 79C , 79D, 79E, 79F Presence icon 67 Balloon image 69, 69A, 69B, 69Z, 75 Map image 71 Close-up image 77, 77A, 77B, 77C, 77D, 77E, 77F Portrait image 81 Opposite image 85 Self-side image Image 86, 86G2, 86G3, 86H2, 86H3, 86M2, 86M3 COMM link 87, 87I, 87J, 87K, 87L COMM word 90 3D virtual space 91, 91A, 91B, 91C, 91D, 91E, 91F object 92, 92C, 92D, 92E, 92F, 92U Three-dimensional barycentric position 93 Virtual plane 94, 94U Third party object 96 96U Speaker interest ratio reflection position 97 97U Virtual distance 98, 98U2, 98U3, 101, 101U2, 101U3, 103, 103U2, 103U3 screen display 99 extension line 100, 100A, 100C, 100D, 100U terminal device 105 tip 107 notification window 110, 210, 510 communication unit 120 input unit 130 image pickup unit 140 sound collection unit 150 display unit 160 voice output unit 170, 220, 520 Storage section 180, 230, 530 Control section 181 Real space information provision section 182 Voice output control section 183 Position acquisition section 185 Object selection section 187 ID acquisition section 189 Telephone section 191 Conversation object selection section 193 COMM link control section 195 COMM Word control unit 196 Conversation interest level setting unit 197 Speaker interest ratio setting unit 198 Conversation interest level notification unit 200, 200A, 200B Information management server 201 Speech recognition server 231 COMM link distribution unit 232 Extracted word data management unit 233 Weighting calculation unit 234 COMM word distribution unit 235 utterance status calculation unit 236 position-linked distribution control unit 241 conversation interest degree control unit 242 interest degree control unit 243 display body control unit 244 interest conversation information amount control unit 245 conversation interest degree notice transmission unit 531 word extraction unit 533 Word/Data Generator 701, 801, 901 CPU
703, 803, 903 ROM
705, 805, 905 RAM
707, 807, 907 Bus 709, 809, 909 Storage device 711, 811, 911 Communication interface 817 Speaker 820 Touch panel 821 Touch detection surface 823 Display surface 840 OS
851 Softphone 853 Ultra-realistic client 855 Telephone call control function

Claims (31)

A conversation event object relating to the call, which associates communication identification information of a plurality of speakers involved in the call, generates a language phrase object relating to the phrase extracted from the voice data of the call, and the conversation event object An information processing server, comprising: a control unit that controls distribution of data related to the language and data related to the language phrase object . The control unit acquires a user's input to the conversation event object, and associates the communication identification information of the user with the communication identification information of the plurality of speakers linked to the conversation event object,
The information processing server according to claim 1. Further comprising a weighting processing unit which performs weighting processing for the previous SL onset language phrase objects,
The control unit compares the results with the predetermined value of the weighting process based on the comparison result, it controls the distribution of data according to the talk word object,
The information processing server according to claim 1 . The information processing server according to claim 3 , wherein the weighting processing unit performs the weighting processing based on an appearance frequency of the word or phrase in the call. The information processing server according to claim 3 , wherein the weighting processing unit performs the weighting processing based on the degree of abstraction of the phrase. The information processing server according to claim 3 , wherein the weighting processing unit performs the weighting processing based on a part-of-speech category of the word. The information processing server according to claim 3 , wherein the weighting processing unit performs the weighting processing based on data regarding sound pressure of speech of the phrase included in the voice data of the call. Wherein, in association cord to the remarks phrase objects, and controls the distribution of data according to the result of the weighting processing, the information processing server according to any one of claims 3 7. Based on the position of the speaker who spoke the words according to the talk word object, it determines the display position of the speech phrase objects, further comprising a speech status calculation unit for distributing the determined the display position, from claim 1 8. The information processing server according to any one of 8 . The control unit , based on the positional relationship between one of the plurality of speakers involved in the call and a user who is not participating in the call in the real space , controlling the delivery, the information processing server according to claim 1. The control unit , based on the positional relationship between one of the plurality of speakers involved in the call and a user who is not participating in the call in the real space , The information processing server according to claim 1, which controls distribution. An interest level control unit that acquires an input of the interest level of the user with respect to the call and associates the acquired interest level with the communication identification information of the user with respect to the conversation event object is further provided.
The information processing server according to claim 1. A display body control unit configured to generate a display body related to the user, and determine a virtual positional relationship between the position of the conversation event object and the display body based on the acquired degree of interest,
The information processing server according to claim 12 . Further comprising a weighting processing unit which performs weighting processing on the calling language phrase objects,
The control unit compares the result of the weighting processing, and a predetermined value, based on the comparison result, controls the distribution of data according to the talk word object,
The predetermined value is changed based on the acquired degree of interest,
The information processing server according to claim 12 or 13 . The degree-of-interest control unit acquires an input of the degree-of-interest ratio of the plurality of speakers involved in the call of the user, associates the input with the conversation event object, and obtains the data regarding the acquired degree-of-interest ratio. delivering, the information processing server according to claim 12 or 13. Further comprising a weighting processing unit which performs weighting processing on the calling language phrase objects,
The control unit compares the result of the weighting processing, and a predetermined value, based on the comparison result, controls the distribution of data according to the talk word object,
The predetermined value is changed based on the acquired interest level and the ratio of the interest levels,
The information processing server according to claim 15 . The interest degree control unit associates the communication identification information of the plurality of speakers associated with the conversation event object with the communication identification information of the user who inputs the interest degree for the call.
The information processing server according to any one of claims 12 to 16 . Generate a three-dimensional virtual space corresponding to the real space where there are multiple speakers involved in the call,
A plurality of objects corresponding to the respective communication identification information of the plurality of speakers involved in the call, and a conversation event object related to the call, which associates the plurality of objects with each other, to generate voice data of the call. Generate a language phrase object related to the phrase extracted from the above , and arrange the data related to the conversation event object and the data related to the language phrase object in the three-dimensional virtual space,
An information processing server including a control unit . Getting an input of the user's interest in the call,
Arranging a user object corresponding to the user in the three-dimensional virtual space,
Determining a virtual distance between the conversation event object and the user object in the three-dimensional virtual space based on the acquired degree of interest,
Further comprising a display body control unit,
The information processing server according to claim 18 . Obtaining an input of interest rate ratios for the plurality of speakers involved in the call of the user,
Arranging a reference point indicating the interest rate on the generated conversation event object based on the obtained interest rate.
Further comprising an interest degree control unit,
The information processing server according to claim 18 . An information processing system including an information processing server and a plurality of terminal devices,
The information processing server associates communication identification information of a plurality of speakers involved in a call, generates a conversation event object related to the call, and generates a language phrase object related to a phrase extracted from the voice data of the call. Then, the data related to the conversation event object and the data related to the language phrase object are distributed to the plurality of terminal devices,
Information processing system. A terminal device, comprising: a display unit that associates communication identification information of a plurality of speakers involved in a call with each other, and displays a conversation event object related to the call and a language phrase object related to a phrase extracted from voice data of the call . The result of the weighting process performed on the language phrase object is acquired, the result of the weighting process is compared with a predetermined value, and the display unit is controlled based on the comparison result. The terminal device according to claim 22 , further comprising an object control unit. The result of the weighting process performed on the language phrase object is acquired, and any one of the size, color, contrast, and display position of the language phrase object is acquired based on the result of the weighting process. 23. The terminal device according to claim 22 , further comprising a language phrase object control unit for controlling. The terminal according to any one of claims 22 to 24 , further comprising: a voice output control unit that acquires a user's interest in the call and controls an output of voice related to the call based on the interest. apparatus. By the user, based on the operation for the user object according to the user which is displayed on the display unit further comprises a conversational interest setting unit that acquires interest for the call of the user, any one of claims 22 to 24, The terminal device according to item 1. 27. The terminal device according to claim 26 , further comprising an imaging unit that acquires a face image of the user for displaying the user object. The terminal device according to any one of claims 22 to 24 , further comprising: a conversation interest degree notification unit that notifies the presence of the user based on acquisition of the interest degree of the user for the call. The conversation interest level notification unit,
Displaying a notification display indicating the presence of the user on the display unit,
Based on the acquired degree of interest, any one of the size, color, movement, contrast, and display position of the notification display is controlled.
The terminal device according to claim 28 . The conversation interest level notification unit,
Causing the voice output unit to perform voice output indicating the presence of the user,
Controlling the volume of the audio output based on the acquired degree of interest;
The terminal device according to claim 28 . Computer,
A conversation event object relating to the call, which associates communication identification information of a plurality of speakers involved in the call, generates a language phrase object relating to the phrase extracted from the voice data of the call, and the conversation event object A program for functioning as a control unit that controls the distribution of the data related to the above and the data related to the language phrase object .

Images