JP2017107597A - Information processing device, information processing method, program, and information processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable the display of a picked-up image obtained by imaging a wider area or a picked-up image obtained by imaging a narrower area by switching the picked-up images.SOLUTION: An information processing device includes a real space information providing part for displaying a display screen of a first display mode for displaying a first picked-up image related to a first area of a real space in a display part. The real space information providing part switches the display screen of the first display mode to a display screen of a second display mode for displaying a second picked-up image related to a second area of the real space narrower than the first area to display the display screen in the display part in accordance with a designation of a position in the first picked-up image by a user in the display screen of the first display mode.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, a program, and an information processing system.

  In recent years, softphones implemented with application software have become popular in place of telephones implemented with hardware as in the past. Since the softphone is implemented by software, it is possible to relatively easily realize addition of functions to the softphone, cooperation between the softphone and other application software, and the like. Therefore, various applied technologies related to softphones have been proposed.

  For example, in Patent Document 1, when a person who appears in the captured image is touched on the display screen of the captured image obtained from the camera, the person is identified by face recognition, the telephone number of the person is acquired, A technique for making a telephone call to a person has been disclosed.

JP 2007-208863 A

  However, Patent Document 1 does not disclose switching between a captured image obtained by imaging a wider area and a captured image obtained by imaging a narrower area.

  Therefore, the present invention provides a new and improved information processing apparatus, information processing method, and program capable of switching and displaying a captured image obtained by capturing a wider area and a captured image obtained by capturing a narrower area. And an information processing system.

  According to the present invention, the real space information providing unit includes a real space information providing unit that causes the display unit to display a display screen in the first display mode for displaying the first captured image related to the first region of the real space. The second captured image related to the second area of the real space that is narrower than the first area is displayed according to the designation of the position in the first captured image by the user on the display screen in the first display mode. An information processing apparatus is provided that switches the display screen of the first display mode to the display screen of the second display mode and displays the display screen on the display unit.

  The second area may be an area including a position in the real space corresponding to a position in the first captured image designated by the user.

  The second area may be an area centered on a position in the real space corresponding to a position in the first captured image designated by the user.

  At least the first captured image may be a captured image of a part or the whole of the first region.

  At least the second captured image may be a captured moving image.

  The display screen in the second display mode may include a button display for switching the display mode from the second display mode to the first display mode.

  On the display screen in the second display mode, the button display may be displayed in the vicinity of the second captured image without being superimposed on the second captured image.

  The first captured image and the second captured image may be a captured image generated based on an image captured by a single imaging device arranged in the real space.

  The information processing apparatus may further include a communication unit connected to a network, and the communication unit may acquire the first captured image and the second captured image via the network.

  The communication unit may acquire the first captured image and the second captured image from a server connected to the network via the network.

  The real space information providing unit sends a request to the server to change the orientation of the imaging device so that the second area can be imaged in accordance with the designation of the position in the first captured image by the user. You may make it transmit to a part.

  The network may be a LAN and / or the Internet.

  The information processing apparatus may further include the display unit, and a position in the first captured image may be designated by a user's touch operation on the display unit.

  The first display mode may be an overhead mode, and the second display mode may be a proximity mode.

  The real space information providing unit requests the server to change the optical zoom or digital zoom zoom rate of the imaging device to a specified zoom rate in response to a user's zoom rate specifying operation on the display unit. When the communication unit transmits the second captured image generated based on imaging by the imaging device after the change to the designated zoom rate via the network, the real space information providing unit May display a display screen in the second display mode for displaying the second captured image after the change to the designated zoom ratio on the display unit.

  In addition, according to the present invention, the step of causing the display unit to display a display screen in the first display mode for displaying the first captured image relating to the first region of the real space, and the user in the display screen in the first display mode. The first display to the display screen in the second display mode for displaying the second captured image related to the second region of the real space narrower than the first region in accordance with the designation of the position in the first captured image by And switching the mode display screen to display on the display unit.

  In addition, according to the present invention, it is possible to cause a computer to function as a real space information providing unit that causes a display unit to display a display screen in a first display mode that displays a first captured image related to the first region of the real space. The real space information providing unit is the real space that is narrower than the first region in accordance with the designation of the position in the first captured image by the user on the display screen in the first display mode. There is provided a program for switching the display screen in the first display mode to the display screen in the second display mode for displaying the second captured image related to the second area, and displaying the first display mode on the display unit.

  In addition, according to the present invention, the real space information provision unit includes a real space information provision unit that causes the display unit to display a display screen in the first display mode for displaying the first captured image related to the first region of the real space. The unit displays a second captured image related to the second region of the real space that is narrower than the first region in response to the designation of the position in the first captured image by the user on the display screen in the first display mode. An information processing system is provided in which the display screen of the first display mode is switched to the display screen of the second display mode to be displayed and displayed on the display unit.

  As described above, according to the present invention, it is possible to switch and display a captured image obtained by capturing a wider area and a captured image obtained by capturing a narrower area.

It is explanatory drawing which shows an example of a schematic structure of the information processing system which concerns on one Embodiment. It is a block diagram which shows an example of the hardware constitutions of the terminal device which concerns on one Embodiment. It is a block diagram which shows an example of a function structure of the terminal device which concerns on one Embodiment. It is explanatory drawing for demonstrating an example of the display screen displayed in bird's-eye view mode. It is explanatory drawing for demonstrating an example of the display screen displayed in proximity | contact mode. It is explanatory drawing for demonstrating the 1st example of the position in the close proximity captured image designated by the user. It is explanatory drawing for demonstrating the 2nd example of the position in the close proximity captured image designated by the user. It is explanatory drawing for demonstrating the 1st example of the three-dimensional virtual space corresponding to a center office. It is explanatory drawing for demonstrating the example of selection of the object arrange | positioned in the three-dimensional virtual space shown by FIG. It is explanatory drawing for demonstrating the 2nd example of the three-dimensional virtual space corresponding to a center office. It is explanatory drawing for demonstrating the example of selection of the object arrange | positioned in the three-dimensional virtual space shown by FIG. It is explanatory drawing for demonstrating an example of the display screen displayed in conversation mode. It is a transition diagram for demonstrating an example of transition of a display mode. It is a block diagram which shows an example of the software structure of the terminal device 100 which concerns on one Embodiment. It is a block diagram which shows an example of the hardware constitutions of the information management server which concerns on one Embodiment. It is a block diagram which shows an example of a function structure of the information management server which concerns on one Embodiment. It is a flowchart which shows an example of the schematic flow of the information processing which concerns on one Embodiment. It is a flowchart which shows an example of the schematic flow of the starting process which concerns on one Embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  Thereafter, <1. Schematic configuration of information processing system>, <2. Configuration of terminal device>, <3. Configuration of information management server>, <4. Embodiments of the present invention will be described in the order of processing flow>.

<< 1. Schematic configuration of information processing system >>
First, a schematic configuration of an information processing system according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is an explanatory diagram illustrating an example of a schematic configuration of an information processing system according to the present embodiment. Referring to FIG. 1, the information processing system exists over a plurality of locations, for example. In this example, the information processing system exists over the center office 10 and the satellite office 20 (or the home office 20). The center office 10 is a relatively large office, and the satellite office 20 (or the home office and 20) is a relatively small office.

  In the center office 10, the information processing system includes a camera 11, a microphone 13, a sensor 15, a media distribution server 17, an information management server 200, and a LAN (Local Area Network) 19. The information processing system includes a terminal device 100, a display 21, and a LAN 23 in the satellite office 20 (or the home office 20). The information processing system further includes a PBX (Private Branch eXchange) 40.

(Camera 11)
The camera 11 captures an area in the direction in which the camera 11 is facing (that is, the imaging direction). A plurality of cameras 11 are installed in the center office 10. Then, each installed camera 11 images a part or the whole of the center office from each installation position. Thus, in the information processing system, the center office is imaged from various positions. The captured image generated through the camera 11 may be a still image or a moving image (that is, a video).

  Moreover, the camera 11 can change direction automatically, for example. Furthermore, the camera 11 has a zoom function, for example. The zoom function may be an optical zoom function or a digital zoom function.

  Further, the camera 11 may be capable of changing the position. As an example, the camera 11 may be movable by a dolly. That is, the camera 11 may be movable along the rail. In this case, the camera 11 may move back and forth by controlling the motor for moving along the rail. As a result, even one camera 11 can generate a captured image in which the subject is captured from different positions.

  Further, when the camera 11 can change the position, the zoom function may be a zoom function by changing the position of the camera 11. As an example, the zoom function may be a dolly zoom function. For example, zooming in may be performed by moving the camera 11 toward the subject, and zooming out by moving the camera 11 in a direction away from the subject. Note that the zoom by dolly does not need to be a precise zoom like the optical zoom or the digital zoom. For example, in zooming in, it is only necessary to generate a captured image that shows a larger subject, and in zooming out, it is only necessary to generate a captured image that shows a smaller subject.

(Microphone 13)
The microphone 13 collects sound around the microphone 13. A plurality of microphones 13 are installed in the center office 10. And each installed microphone 13 collects the sound around each installation position in the center office. In this way, in the information processing system, sounds at various positions in the center office 10 are collected.

(Sensor 15)
The sensor 15 can include various types of sensors. For example, the sensor 15 includes a seat sensor that determines whether a person is in the seat. The seat sensor is installed in each seat and determines whether a person is sitting in each seat. The seat sensor is, for example, any sensor that can detect pressing.

(Media distribution server 17)
The media distribution server 17 distributes media (for example, audio, video, etc.) to the terminal device in response to a request.

(Information management server 200)
The information management server 200 manages various information used in the information processing system. That is, the information management server 200 stores the various information and updates the various information in a timely manner.

  For example, the information management server 200 manages parameters regarding the camera 11, the microphone 13, and the sensor 15. Specifically, for example, the information management server 200 stores and updates information such as an installation position of the camera 11, an imaging direction (for example, a direction perpendicular to the camera lens), a zoom ratio, and the like as parameters of the camera 11. .

  For example, the information management server 200 manages data in a three-dimensional virtual space corresponding to the real space. The three-dimensional virtual space is, for example, a three-dimensional virtual space that simulates the center office 10. An object is arranged in the three-dimensional virtual space. For example, the object corresponds to a person. Then, the object is arranged at a three-dimensional virtual position in the three-dimensional virtual space corresponding to the position of each seat of the center office 10. That is, when a person is sitting on the seat, the object is arranged at a three-dimensional virtual position corresponding to a position where the person will be present. As an example, the object is a cylindrical object. The three-dimensional virtual space will be described later.

(LAN19)
The LAN 19 is a network that connects each device in the center office 10. The LAN 19 connects each device in the center office 10 and a device outside the center office 10 via the external network 30. The external network 30 includes the Internet, for example.

(Terminal device 100)
The terminal device 100 is used by a user. For example, the terminal device 100 provides a user with a function for performing communication such as telephone and e-mail. The terminal device 100 is a tablet terminal as an example. Note that the terminal device 100 may be another device having a display function and a communication function, such as a smartphone, a PC (Personal Computer), and a display-equipped telephone, instead of the tablet terminal.

(Display 21)
The display 21 displays one of the screens. For example, the display 21 displays a screen including a captured image generated through the camera 11. Thereby, many persons including the user of the terminal device 100 can see the state of the center office 10 via the display 21.

  Further, the display 21 outputs any sound, for example. For example, the display 21 outputs sound collected by the microphone 13. Thereby, many persons including the user of the terminal device 100 can hear the sound of the center office 10 via the display 21.

(LAN23)
The LAN 23 is a network that connects devices in the satellite office 20 (or the home office 20). The LAN 23 connects each device in the satellite office 20 to a device outside the satellite office 20 via the external network 30.

(PBX40)
The PBX 40 enables communication between devices via the external network 30. PBX40 is, for example, H.264. It operates according to H.323 or SIP (Session Initiation Protocol).

  Specifically, for example, the PBX 40 stores identification information for communication (for example, a telephone number) and an IP (Internet Protocol) address in association with each other. In response to the request, the PBX 40 converts the communication identification information into an IP address, and provides the IP address to the request source.

  The PBX 40 may be connected to the LAN 19 or the LAN 23.

<< 2. Terminal device configuration >>
Next, an example of the configuration of the terminal device 100 according to the present embodiment will be described with reference to FIGS.

<2-1. Hardware configuration >>
First, an example of the hardware configuration of the terminal device 100 according to the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the terminal device 100 according to the present embodiment. Referring to FIG. 2, the terminal device 100 includes a CPU (Central Processing Unit) 801, a ROM (Read Only Memory) 803, a RAM (Random Access Memory) 805, a bus 807, a storage device 809, a communication interface 811, a camera 813, a microphone. 815, a speaker 817, and a touch panel 820.

  The CPU 801 executes various processes in the terminal device 100. The ROM 803 stores a program and data for causing the CPU 801 to execute processing in the terminal device 100. The RAM 805 temporarily stores programs and data when the CPU 801 executes processing.

  A bus 807 connects the CPU 801, the ROM 803, and the RAM to each other. A storage device 809, a communication interface 811, a camera 813, a microphone 815, a speaker 817, and a touch panel 820 are further connected to the bus 807. The bus 807 includes, for example, a plurality of types of buses. As an example, the bus 807 includes a high-speed bus that connects the CPU 801, the ROM 803, and the RAM 805, and one or more other buses that are slower than the high-speed bus.

  The storage device 809 stores data to be temporarily or permanently stored in the terminal device 100. The storage device 809 may be, for example, a magnetic storage device such as a hard disk, or an EEPROM (Electrically Erasable and Programmable Read Only Memory), a flash memory, an MRAM (Magnetoresistive Random Access Memory). Further, non-volatile memories such as FeRAM (Ferroelectric Random Access Memory) and PRAM (Phase change Random Access Memory) may be used.

  The communication interface 811 is a communication unit included in the terminal device 100 and communicates with an external device via a network (or directly). The communication interface 811 may be an interface for wireless communication. In this case, for example, a communication antenna, an RF circuit, and other communication processing circuits may be included. The communication interface 811 may be an interface for wired communication. In this case, for example, a LAN terminal, a transmission circuit, and other communication processing circuits may be included.

  The camera 813 images a subject. The camera 813 includes, for example, an optical system, an image sensor, and an image processing circuit.

  The microphone 815 collects ambient sounds. The microphone 815 converts ambient sound into an electric signal, and converts the electric signal into digital data.

  The speaker 817 outputs sound. The speaker 817 converts digital data into an electric signal, and converts the electric signal into sound.

  The touch panel 820 includes a touch detection surface 821 and a display surface 823.

  Touch detection surface 821 detects a touch position on touch panel 820. More specifically, for example, when the user touches the touch panel 820, the touch detection surface 821 senses the touch, generates an electrical signal corresponding to the position of the touch, and converts the electrical signal to the touch position. Convert to information. The touch detection surface 821 can be formed according to an arbitrary touch detection method such as a capacitance method, a resistance film method, and an optical method, for example.

  The display surface 823 displays an output image (that is, a display screen) from the terminal device 100. The display surface 823 can be realized using, for example, a liquid crystal, an organic EL (Organic Light-Emitting Diode: OLED), a CRT (Cathode Ray Tube), or the like.

<2-2. Functional configuration >>
Next, an example of a functional configuration of the terminal device 100 according to the present embodiment will be described. FIG. 3 is a block diagram illustrating an example of a functional configuration of the terminal device 100 according to the present embodiment. Referring to FIG. 3, the terminal device 100 includes a communication unit 110, an input unit 120, an imaging unit 130, a sound collection unit 140, a display unit 150, an audio output unit 160, a storage unit 170, and a control unit 180.

(Communication unit 110)
The communication unit 110 communicates with other devices. For example, the communication unit 110 is directly connected to the LAN 23 and communicates with each device in the satellite office 20. The communication unit 110 communicates with each device in the center office 10 via the external network 30 and the LAN 19. Specifically, for example, the communication unit 110 communicates with the camera 11, the microphone 13, the sensor 15, the media distribution server 17, and the information management server 200. Note that the communication unit 110 can be implemented by the communication interface 811, for example.

(Input unit 120)
The input unit 120 receives input from the user of the terminal device 100. The input unit 120 provides the input result to the control unit 180.

  For example, the input unit 120 detects a position designated by the user on the display screen. More specifically, for example, the input unit 120 is mounted by a touch detection surface 821 and detects a touch position on the touch panel 820. Then, the input unit 120 provides the detected touch position to the control unit 820.

(Imaging unit 130)
The imaging unit 130 images a subject. For example, the imaging unit 130 captures an area in the front direction of the terminal device 100. In this case, the imaging unit 130 images the user of the terminal device 100. The imaging unit 130 provides the imaging result (that is, the captured image) to the control unit 180. Note that the imaging unit 130 may be implemented by a camera 813, for example.

(Sound collector 140)
The sound collection unit 140 collects sounds around the terminal device 100. For example, the sound collection unit 140 collects the voice of the user of the terminal device 100. The sound collection unit 140 provides a sound collection result (that is, sound data) to the control unit 180. Note that the sound collection unit 140 can be implemented by a microphone 815, for example.

(Display unit 150)
The display unit 150 displays an output image (that is, a display screen) from the terminal device 100. The display unit 150 displays a display screen according to control by the control unit 180. Note that the display unit 150 can be implemented by the display surface 823, for example.

(Audio output unit 160)
The audio output unit 160 outputs audio from the terminal device 100. The sound output unit 160 outputs sound in accordance with control by the control unit 180. Note that the audio output unit 160 can be implemented by a speaker 817, for example.

(Storage unit 170)
The storage unit 170 stores a program and data for the operation of the terminal device 100. Note that the storage unit 170 may be implemented by the storage device 809, for example.

  For example, the storage unit 170 stores data in a three-dimensional virtual space corresponding to the real space. Specifically, for example, since the information management server 200 stores the data of the three-dimensional virtual space corresponding to the center office 10, the control unit 180 transmits the data of the three-dimensional virtual space via the communication unit 110. Get the data. Then, the storage unit 170 stores the acquired data of the three-dimensional virtual space.

(Control unit 180)
The control unit 180 provides various functions of the terminal device 100. The control unit 180 includes a real space information providing unit 181, a position acquisition unit 183, an object selection unit 185, an ID acquisition unit 187, and a telephone unit 189. The control unit 180 can be implemented by, for example, the CPU 801, the ROM 803, and the RAM 805.

(Real space information provider 181)
The real space information providing unit 181 provides real space information to the user of the terminal device 100.

  For example, the real space information providing unit 181 causes the display unit 150 to display a display screen of the captured image of the real space. More specifically, for example, the captured image is a captured image generated through the camera 11 in the real space (center office 10). The captured image may be a captured image generated by the camera 11, or may be a captured image generated by processing the captured image generated by the camera 11. The display screen is a screen that includes the captured image in part or in whole.

  The real space information providing unit 181 acquires a captured image generated by the camera 11 via the communication unit 110. Then, the real space information providing unit 181 generates a display screen including the captured image and causes the display unit 150 to display the display screen.

  Further, for example, the captured image is a captured image generated through one selected imaging device among a plurality of imaging devices in real space. More specifically, for example, the captured image is a captured image generated through one selected camera 11 among a plurality of cameras 11 arranged in the center office 10. A specific method for the user to select the camera 11 will be described later. Since the camera 11 can be selected, the user can see a captured image from a desired position. Then, when specifying the position of the captured image as described later, the user can specify the position with a more desirable captured image.

  For example, the display screen includes a captured image corresponding to a display mode. More specifically, for example, the display screen includes a first captured image in which a first area of real space is captured in the first display mode, and the first area in the second display mode. A second captured image in which a narrower second region is captured is included. That is, the real space information providing unit 181 displays the first captured image on the display unit 150 in the first display mode, and displays the second captured image on the display unit 150 in the second display mode. Let

  More specifically, for example, the first captured image is a captured image corresponding to the first zoom rate. The second captured image is a captured image corresponding to a second zoom factor that is larger than the first zoom factor. For example, the real space information providing unit 181 makes a request regarding the zoom (optical zoom, digital zoom, or zoom by changing the position of the imaging device (for example, zoom by dolly)) to the camera 11 via the communication unit 110. Thus, a captured image corresponding to the first zoom factor or a captured image corresponding to the second zoom factor is acquired. Alternatively, the real space information providing unit 181 may generate a captured image corresponding to the first zoom rate or a captured image corresponding to the second zoom rate by digital zooming on the captured image generated by the camera 11. . Note that the zoom rate here does not need to be a precise value such as 1.5 times or 2 times, and may be any value that directly or indirectly indicates the size of the subject in the captured image. . For example, in particular, when zooming by changing the position of the camera 11 (for example, zooming in and out by dolly) is used, the zoom rate is not a precise value such as 1.5 times or 2 times, but the size of the subject. Directly indicating the degree of the subject (for example, a parameter indicating the degree of the approximate size of the subject) or indirectly indicating the degree of the subject (eg, the position of the camera 11 on the rail) , Etc.). The captured image corresponding to the first zoom factor is a captured image in which the subject is smaller, and the captured image corresponding to the second zoom factor that is larger than the first zoom factor is larger in the subject. Any captured image may be used.

  As an example, the above-described display screen includes a bird's-eye shot image generated by the camera 11 by shooting at a zoom factor of X (for example, X = 1) in the bird's-eye view mode. It includes a close-up captured image generated by the camera 11 during imaging with Y> X). That is, the bird's-eye-view captured image is a captured image in which a wider area of the center office 10 is captured, and the close-up captured image is a captured image in which a narrower area of the center office 10 is captured. Hereinafter, a specific example of this point will be described with reference to FIGS.

—Display Screen Displayed in Overhead Mode FIG. 4 is an explanatory diagram for explaining an example of a display screen displayed in the overhead view mode. Referring to FIG. 4, a display screen 60 displayed in the overhead view mode is shown. The display screen 60 includes a bird's-eye image 61, a button image 63, a presence icon 65, a balloon image 67, and a map image 69.

  The overhead view captured image 61 is, for example, a captured image generated by the camera 11 by imaging at a zoom factor of X times. As an example, X = 1. In other words, the overhead view captured image 61 is a captured image generated by the camera 11 with imaging without zooming.

  For example, when the user designates the position of the overhead view captured image 61, the real space information providing unit 181 switches the display mode from the overhead view mode to the proximity mode. More specifically, for example, when the user touches the position of the overhead view captured image 61 and a touch position corresponding to the overhead view captured image 61 is detected, the real space information providing unit 181 changes the display mode from the overhead view mode. Switch to mode.

  The button image 63 is an image for selecting another camera 11. For example, when the user designates the position of the button image 63, the real space information providing unit 181 acquires an overhead view captured image generated by another camera 11 and causes the display unit 150 to display the overhead view captured image. More specifically, for example, when the user touches the position of the button image 63 and a touch position corresponding to the button image 63 is detected, the real space information providing unit 181 has an overhead view generated by another camera 11. A captured image is acquired, and the overhead image captured image is displayed on the display unit 150. For example, when the position of the button image 63A is designated by the user, the camera 11 located on the left side of the current camera 11 is selected. When the position of the button image 63B is designated by the user, the camera 11 positioned on the right side of the current camera 11 is selected. Then, the real space information provision unit 181 acquires the overhead view captured image generated by the selected camera 11 and causes the display unit 150 to display the overhead view captured image.

  The presence icon 65 is an icon indicating the busyness of the person shown in the captured image 61. For example, the presence icon 65 changes color according to the busyness of the person. As an example, the presence icon indicates that the busyness is high when it is red, indicates that the busyness is normal when it is yellow, and indicates that the busyness is low twice that of blue. As will be described later, since it can be seen where the person should be in the captured image 61, such an icon can also be displayed.

  For example, the display screen 60 includes, for example, information related to a person shown in the overhead view captured image 61 (hereinafter referred to as “person related information”). The person related information includes, for example, state information indicating the state of the person. As described above, the state information is the presence icon 65 as an example. The person-related information may include a state history indicating the state of the person at two or more points in time. As an example, the state history information may include a history of busyness of a person shown in the captured image 61. That is, the history of the busyness of the person may be displayed on the display screen 60. For example, the real space information providing unit 181 acquires person-related information or information necessary for displaying the person-related information from the information management server 200 via the communication unit 110.

  With such person-related information, the user can more accurately grasp the situation where the person is placed. In addition, the status information allows the user to more accurately determine whether or not to contact a person. Moreover, since the user can grasp not only the instantaneous user state but also the user state in a certain period from the state history, the user can more accurately determine whether or not to contact a person. be able to.

  The balloon image 67 is an image including character information presented by a person shown in the captured image 61. The balloon image 67 is also an example of person related information.

  The map image 69 is an image showing a map of the center office 10. The map image 69 further indicates the camera 11 being used by an icon. The map image 69 may be omitted particularly when only one or a few cameras 11 are installed in the center office 10.

—Display Screen Displayed in Proximity Mode FIG. 5 is an explanatory diagram for explaining an example of a display screen displayed in the proximity mode. Referring to FIG. 5, a display screen 70 displayed in the proximity mode is shown. The display screen 70 includes a close-up captured image 71, a button image 73, and a map image 75.

  The close-up captured image 71 is, for example, a captured image generated by the camera 11 by capturing at a zoom factor (Y> X) of Y times. As an example, Y = 1.5. That is, the overhead view captured image 61 is a captured image generated by the camera 11 with 1.5 × zoom imaging.

  The button image 73 is an image for switching the display mode from the proximity mode to the overhead view mode. For example, when the user designates the position of the button image 73, the real space information providing unit 181 switches the display mode from the proximity mode to the overhead mode. More specifically, for example, when the user touches the position of the button image 73 and a touch position corresponding to the button image 73 is detected, the real space information providing unit 181 changes the display mode from the proximity mode to the overhead view mode. Switch.

  Further, the map image 75 is an image showing a map of the center office 10 in the same manner as the map image 69 in the overhead view mode. The map image 75 further shows the camera 11 being used. For example, in the proximity mode, the icon of the camera used in the map image 75 is displayed at a more advanced position in order to symbolically indicate that the zoom has been performed. Similar to the map image 69 in the overhead view mode, the map image 75 can be omitted particularly when only one or a few cameras 11 are installed in the center office 10.

  Note that the display screen 70 displayed in the proximity mode may also include person-related information such as a presence icon and a balloon image.

  As described above, according to the display mode, a captured image obtained by capturing a wider area is displayed, so that the user can see the overall situation of the real space and easily find a specific person. Can do. Then, by displaying a captured image obtained by capturing a narrower region, the user can more easily designate the position of a specific person. Further, since only the display mode is switched, the user is only required to perform an easy operation. In addition, since the captured image obtained by capturing a wider area and the captured image obtained by capturing a narrower area are captured images having different zoom rates, the user easily grasps the relationship between these captured images intuitively. be able to. Therefore, even if the display mode is switched, the user can easily find a specific person and specify the position of the specific person.

-Other real space information As mentioned above, the real space information provision part 181 provides the visual information of real space by making the display part 150 display the display screen of the captured image of real space. Further, the real space information providing unit 181 may also provide auditory information on the real space. That is, the real space information providing unit 181 may cause the audio output unit 160 to output the audio of the audio data obtained by collecting the sound in the real space.

  For example, the real space information providing unit 181 selects the microphone 13 close to the selected camera 11. Then, the real space information providing unit 181 acquires voice data obtained by sound collection at the center office 10 from the microphone 13 via the communication unit 110. And the real space information provision part 181 may make the audio | voice output part 160 output the audio | voice of the acquired audio | voice data.

(Position acquisition unit 183)
The position acquisition unit 183 acquires a position in the captured image specified by the user on the display screen of the captured image in real space. More specifically, for example, when the input unit 120 detects a position in the captured image specified by the user on the display screen of the captured image of the real space, the position acquisition unit 183 acquires the position. Then, the position acquisition unit 183 provides the position in the captured image to the object selection unit 185.

  As an example, when the position acquisition unit 183 acquires any position in the close-up captured image 71, the position acquisition unit 183 provides the position to the object selection unit 185. Hereinafter, a specific example of this point will be described with reference to FIGS.

  FIG. 6 is an explanatory diagram for describing a first example of a position in the close-up captured image designated by the user. Referring to FIG. 6, a close-up captured image 71 and a user's hand 3 included in a display screen displayed in the close-up mode are shown. A person 77 shown in the close-up captured image 71 is also shown. Then, the user designates the position of the person 77 in the close-up captured image 71 by touching the position of the person 77 with the hand 3. In this case, the input unit 120 detects the position of the person 77 in the close-up captured image 71, and the position acquisition unit 183 acquires the position. Then, the input unit 120 provides the acquired position to the object selection unit 185.

  FIG. 7 is an explanatory diagram for describing a second example of the position in the close-up captured image designated by the user. Referring to FIG. 7, people 77A and 77B appearing in the close-up captured image 71 are shown. Then, the user designates the position of the person 77 </ b> A in the close-up captured image 71 by touching the position of the person 77 </ b> A with the hand 3. In this case, the input unit 120 detects the position of the person 77A in the close-up captured image 71, and the position acquisition unit 183 acquires the position. Then, the input unit 120 provides the acquired position to the object selection unit 185.

(Object selection unit 185)
The object selection unit 185 selects an object arranged in the three-dimensional virtual space corresponding to the real space based on the position in the acquired captured image. For example, when the position acquisition unit 183 acquires a position in the captured image specified by the user on the display screen of the captured image in the real space, the object selection unit 185 corresponds to the real space based on the position. Select an object placed in the three-dimensional virtual space.

  The object is an object arranged at a three-dimensional virtual position in the three-dimensional virtual space corresponding to the position in the captured image. Further, for example, the captured image is a captured image included in the display screen in the second mode (for example, the proximity mode).

  As an example, when the position acquisition unit 183 acquires a position in the close-up captured image generated by the camera 11, the object selection unit 185 includes, among the objects arranged in the three-dimensional virtual space corresponding to the center office 10. The object corresponding to the position is selected. Note that the object selection unit 185 acquires, for example, data of the three-dimensional virtual space corresponding to the center office 10 from the storage unit 170.

-Example in which one object is arranged Hereinafter, a specific example in the case where one object is arranged in the three-dimensional virtual space will be described with reference to FIGS. 8 and 9.

  FIG. 8 is an explanatory diagram for describing a first example of a three-dimensional virtual space corresponding to the center office 10. Referring to FIG. 8, a three-dimensional virtual space 90 corresponding to the center office 10 is shown. An object 91 is arranged in the three-dimensional virtual space 90. The object 91 corresponds to a person (for example, Mr. A). Then, the object is arranged at a three-dimensional virtual position corresponding to the position of the seat of the person (for example, Mr. A) in the center office 10. That is, when the person is sitting on the seat, the object 91 is arranged at a three-dimensional virtual position corresponding to a position where the person will be present. In this example, the object 91 is a cylindrical object. The cylinder object is a cylinder object with a radius R and a height H. The radius R and the height H are determined in advance, for example. The data in the three-dimensional virtual space 90 includes information related to each camera 11. For example, the information related to each camera 11 includes a three-dimensional virtual position corresponding to the installation position of each camera 11, an imaging direction (for example, a direction perpendicular to the camera lens), an angle of view, and the like.

  FIG. 9 is an explanatory diagram for explaining an example of selection of objects arranged in the three-dimensional virtual space shown in FIG. Referring to FIG. 9, for easy understanding, the positional relationship in the horizontal plane in the three-dimensional virtual space 90 is shown. Specifically, the object 91 arranged in the three-dimensional virtual space 90, the three-dimensional virtual position (hereinafter referred to as “virtual camera position”) O corresponding to the installation position of the camera 11 used for imaging, and the imaging of the camera 11 An axis y corresponding to a direction (eg, a direction perpendicular to the camera lens) and an axis x orthogonal to the axis y are shown. In this example, for easy understanding, it is assumed that the camera 11 is installed so that the imaging direction of the camera 11 is parallel to the horizontal plane.

  Further, the angle of view θ of the camera 11 is also shown. In addition, a virtual plane 93 that is perpendicular to the axis y corresponding to the imaging direction of the camera 11 and has a width corresponding to the angle of view θ is shown. Further, the virtual plane 93 is separated from the virtual camera position O by a distance I. The virtual surface 93 is a quadrangular surface and has the same aspect ratio as the captured image. That is, the virtual surface 93 is a surface corresponding to the captured image.

  For example, the object selection unit 185 converts the position in the captured image designated by the user into a three-dimensional virtual position A as shown in FIG. Then, the object selection unit 185 identifies an object that intersects a straight line passing through the virtual camera position O and the three-dimensional virtual position A. In this example, the object selection unit 185 specifies the object 91. Then, the object selection unit 185 selects the object 91.

  In the example of FIG. 9, for example, the position in the captured image that is converted to the three-dimensional virtual position between the three-dimensional virtual position B and the three-dimensional virtual position D in the virtual surface 93 is the user in the captured image. The object 91 is selected. Note that such a position is generally a position where a person corresponding to the object 91 is captured in the captured image.

  The distance I is determined so that the virtual plane 93 is positioned between the virtual camera position O and the object 91. As an example, the distance I is the focal length of the camera 11, but is not limited to this.

  Further, for the sake of easy explanation, the method of selecting an object by focusing on the horizontal plane of the three-dimensional virtual space 90 has been described, but of course, even if a vertical direction (for example, the z-axis) is added, Can be converted into a three-dimensional virtual position, and the object 91 can be specified from the three-dimensional virtual position.

  For example, as described above, the object corresponding to the position in the captured image is selected. In the above-described example, the object corresponding to the position in the captured image is identified by converting the position in the captured image to the three-dimensional virtual position. However, the method for identifying the object corresponding to the position in the captured image is as follows. Not limited to this.

  As an example, the object selection unit 185 may project the object 91 onto the virtual plane 93 with the virtual camera position O as the origin, and convert the projection range of the object 91 into a range in the captured image. Then, when the position in the captured image specified by the user is included in the range, the object selection unit 185 may select the object 91.

  As another example, a rendering image is generated by rendering the three-dimensional virtual space 90 using the virtual camera position O, the axis y, and the angle of view θ, and the object 91 in the rendering image is captured. The range in the captured image corresponding to the object 91 may be obtained from the existing range. Then, when the position in the captured image specified by the user is included in the range, the object selection unit 185 may select the object 91.

-Selection of object in consideration of state For example, the object 91 arranged in the three-dimensional virtual space 90 corresponds to state information indicating one of a plurality of states. For example, the object 91 corresponds to a person. The object 91 corresponds to state information (hereinafter referred to as “existence information”) indicating whether or not the person is sitting on the seat. The presence / absence information indicates a state where the person is sitting on the seat or a state where the person is not sitting on the seat. For example, the object selection unit 185 acquires a determination result as to whether or not the person corresponding to the object 91 is sitting on the seat from the sensor 15 via the communication unit 110. Then, the determination result becomes presence / absence information.

  For example, the object selection unit 185 selects an object that is arranged in the three-dimensional virtual space and that corresponds to state information indicating a predetermined state among the plurality of states. For example, the object selection unit 185 selects an object that is arranged in the three-dimensional space and corresponds to state information indicating a state where a person is sitting on a seat. That is, when the sensor 15 determines that the person is sitting on the seat, the object selection unit 185 can select the object corresponding to the person, but the sensor 15 determines that the person is not sitting on the seat. If it is, the object corresponding to the person is not selected.

  Thus, since an object is selected according to a person's state, it can avoid selecting the object which should not be selected originally. For example, it can be avoided that an object corresponding to a person is selected when there is no person.

-Example in which two objects are arranged Two or more objects may be arranged in the three-dimensional virtual space. Hereinafter, a specific example in which two objects are arranged will be described with reference to FIG.

  FIG. 10 is an explanatory diagram for describing a second example of the three-dimensional virtual space corresponding to the center office 10. Referring to FIG. 10, a three-dimensional virtual space 90 corresponding to the center office 10 is shown. In the three-dimensional virtual space 90, an object 91A and an object 91B are arranged. The object 91A corresponds to a certain person (for example, Mr. A) and is arranged at a three-dimensional virtual position corresponding to the position of the seat of the certain person in the center office 10. The object 91 </ b> B corresponds to a certain person (for example, Mr. B) and is arranged at a three-dimensional virtual position corresponding to the position of the seat of the certain person in the center office 10. Similar to the example of FIG. 8, the object 91 is a cylindrical object with a radius R and a height H.

  As described above, when two or more objects are arranged in the three-dimensional virtual space, the plurality of objects may correspond to the positions in the captured image designated by the user. For example, in the example of FIG. 10, both the object 91A and the object 91B can correspond to positions in the captured image specified by the user. As an example, in the case where two persons can be captured in the captured image as shown in FIG. 7, if the user specifies a position where the two persons overlap in the captured image, the two persons are Two corresponding objects may correspond to the position.

  Therefore, in such a case (that is, when a plurality of objects correspond to the positions of the captured images specified by the user), the object selection unit 185 selects any one of the plurality of objects. select. That is, when there are a plurality of objects respectively arranged at the three-dimensional virtual position in the three-dimensional virtual space corresponding to the position in the captured image designated by the user, the object selection unit 185 Any one of the objects is selected.

  For example, the captured image is generated through an imaging device in real space. Then, the object selection unit 185 selects an object closer to the three-dimensional virtual position in the three-dimensional virtual space corresponding to the imaging device among the plurality of objects. More specifically, for example, the captured image is generated through the camera 11 in the center office 10. Then, when there are a plurality of objects corresponding to the positions in the captured image designated by the user, the object selection unit 185 uses the three-dimensional virtual position corresponding to the installation position of the camera 11 (that is, the virtual camera position O). Select a close object. Hereinafter, a specific example of this point will be described with reference to FIGS. 10 and 11.

  FIG. 11 is an explanatory diagram for explaining an example of selection of objects arranged in the three-dimensional virtual space shown in FIG. Referring to FIG. 11, for easy understanding, the positional relationship in the horizontal plane in the three-dimensional virtual space 90 is shown. Specifically, an object 91A and an object 91B arranged in the three-dimensional virtual space 90 are shown. Similarly to FIG. 9, the virtual camera position O, the axis y, the axis x, the angle of view θ, and the virtual plane 93 are shown. Also in this example, as in FIG. 9, it is assumed that the camera 11 is installed so that the imaging direction of the camera 11 is parallel to the horizontal plane for easy understanding.

  For example, as shown in FIG. 7, the position in the captured image is designated by the user. In this case, according to the method described with reference to FIG. 9, the position in the captured image is converted into a three-dimensional virtual position between the three-dimensional virtual position B ′ and the three-dimensional virtual position D. In this case, both the object 91A and the object 91B are specified as objects corresponding to the position in the captured image. Then, the object selection unit 185 selects an object 91A that is closer to the virtual camera position O among the objects 91A and 91B.

  In addition, when the position in the captured image designated by the user is converted into a three-dimensional virtual position between the three-dimensional virtual position B and the three-dimensional virtual position B ′, the object 91A is the above-described captured image. It is specified and selected as an object corresponding to the position within. In addition, when the position in the captured image designated by the user is converted into a three-dimensional virtual position between the three-dimensional virtual position D and the three-dimensional virtual position D ′, the object 91B is converted into the captured image. It is specified and selected as an object corresponding to the position within.

  By selecting one object from a plurality of objects in this way, it is avoided that an error (an error caused by acquiring a plurality of communication IDs) occurs after the selection of a plurality of objects. be able to. In addition, by selecting an object close to the virtual camera position corresponding to the camera 11 from among a plurality of objects, for example, even when people are overlapped in the captured image, it corresponds to the person in the foreground The object to be selected is selected. Therefore, an object corresponding to the person intended by the user is selected.

  As described above, the object selection unit 185 selects an object. Then, the object selection unit 185 provides identification information (hereinafter referred to as “object ID”) of the selected object to the ID acquisition unit 187. The object ID may be identification information of a person corresponding to the selected object, or may be a mere number assigned to the selected object.

(ID acquisition unit 187)
The ID acquisition unit 187 acquires identification information corresponding to the selected object. For example, the identification information is identification information for communication corresponding to the selected object (hereinafter referred to as “communication ID”). The communication ID is a telephone number as an example.

  Specifically, for example, when the object selection unit 185 selects an object, the ID acquisition unit 187 acquires the object ID of the selected object. Then, the object selection unit 185 transmits the object ID to the information management server 200 via the communication unit 110, and acquires the communication ID corresponding to the object ID. The communication ID is a communication ID of a person corresponding to the selected object.

  For example, as described above, the ID acquisition unit 187 acquires the communication ID corresponding to the selected object. The communication ID is a communication ID of the communication device of the person corresponding to the object (that is, the person shown in the position of the captured image designated by the user). Then, the ID acquisition unit 187 provides the acquired communication ID to the telephone unit 189.

  As described above, when a position in the captured image is designated by the user, an object corresponding to the position is selected, and a communication ID corresponding to the object is acquired. Thereby, the user can contact the target person with an intuitive operation. In addition, an object corresponding to the person is selected and a communication ID is acquired regardless of how the person appears in the captured image, so that the person can be contacted more reliably.

(Telephone part 189)
The telephone unit 189 provides a function for making a call. For example, the telephone unit 189 provides a softphone function.

  For example, when the telephone unit 189 acquires the communication ID provided by the ID acquisition unit 187, the telephone unit 189 performs a telephone call using the communication ID. More specifically, for example, when the telephone unit 189 acquires the communication ID, the telephone unit 189 provides the communication ID to the PBX 40 via the communication unit 110 and acquires the IP address from the PBX 40. Then, the telephone unit 189 executes a series of sequences for establishing a session with a communication device having the IP address (that is, a communication device that is a destination).

  As described above, the telephone unit 189 performs a telephone call for a telephone call to a person shown at a position in the captured image designated by the user on the display screen. That is, the telephone unit 189 makes a telephone call to the communication device of the person.

  When the voice data from the communication device of the other party of the telephone is received by the communication unit 110, the telephone unit 189 causes the voice output unit 160 to output the voice of the voice data. Further, the telephone unit 189 causes the communication unit 110 to transmit the voice data provided by the sound collection unit 140 to the communication device of the other party of the telephone. In addition, for example, the telephone unit 189 causes the communication unit 110 to transmit a captured image provided by the imaging unit 130 (for example, a captured image in which the user of the terminal device 100 is captured) to the communication apparatus of the other party of the telephone.

  Further, the telephone unit 189 causes the display unit 150 to display a display screen during a telephone call. For example, a display mode that displays a display screen during a call is called a conversation mode. In this case, when the telephone unit 189 acquires the communication ID provided by the ID acquisition unit 187, the telephone unit 189 switches the display mode from the proximity mode to the conversation mode. Hereinafter, a specific example of the conversation mode display screen will be described with reference to FIG.

  FIG. 12 is an explanatory diagram for explaining an example of a display screen displayed in the conversation mode. Referring to FIG. 12, a display screen 80 displayed in the conversation mode is shown. The display screen 60 includes a partner-side captured image 81, a button image 83, and a self-side captured image 85.

  The partner-side captured image 81 is, for example, a captured image acquired from a communication device of a telephone partner. For example, when the communication unit 110 receives a captured image from a communication apparatus that is the other party of the telephone, the telephone unit 189 uses the captured image as the partner-side captured image 81.

  The button image 63 is an image for ending the call. For example, when the user designates the position of the button image 83, the telephone unit 189 ends the telephone call. More specifically, for example, when the user touches the position of the button image 83 and a touch position corresponding to the button image 83 is detected, the telephone unit 189 executes a sequence for ending the call including disconnection of the session. To do. For example, the telephone unit 189 switches the display mode from the conversation mode to the overhead mode.

  The own side captured image 85 is a captured image provided by the imaging unit 130.

(Other: Display mode transition)
Here, a specific example of the transition between the overhead view mode, the proximity mode, and the conversation mode will be described with reference to FIG.

  FIG. 13 is a transition diagram for explaining an example of the transition of the display mode. Referring to FIG. 13, for example, when connection processing with the camera 11, the microphone 13, the sensor 15, the information management server 200, and the like is performed at the time of starting up the software, the display mode becomes the overhead mode 301 (ENTRY).

  In the overhead view mode 301, the overhead view mode process is executed (DO). When the user designates the position of the overhead view captured image 61, a mode change process is performed (EXIT), and the display mode is switched from the overhead view mode 301 to the proximity mode 303. The mode change process includes a zoom process of the camera 11 (ENTRY).

  In the proximity mode 303, proximity mode processing is executed (DO). When the user designates a position in the close-up captured image 71 where a person is shown, a mode change process is performed (EXIT), and the display mode is switched from the close-up mode 303 to the conversation mode 305. The mode change process in this case includes a process for a call (ENTRY). When the user designates the position of the button image 73, a mode change process is performed (EXIT), and the display mode is switched from the proximity mode 303 to the overhead mode 301. The mode change process in this case includes the connection process described above (ENTRY).

  In conversation mode 305, conversation mode processing is executed (DO). When the user designates the position of the button image 83, a mode change process is performed (EXIT), and the display mode is switched from the conversation mode 305 to the overhead mode 301. The mode change process in this case includes the connection process described above (ENTRY).

(Other: Another example of display screen)
-Captured image according to display mode As described above, for example, in the first display mode (for example, the bird's-eye view mode), the real space information providing unit 181 captures the first region in which the first area of the real space is captured. A captured image (for example, an overhead image captured image) is displayed on the display unit 150. In addition, the real space information providing unit 181 is configured such that, in the second display mode (for example, the proximity mode), the second captured image (for example, the proximity captured image) in which the second region narrower than the first region is captured. ) Is displayed on the display unit 150. In the above-described example, the first captured image obtained by capturing the first area in the real space is a captured image corresponding to the first zoom rate, and is narrower than the first area. The second captured image in which the region is captured is a captured image corresponding to a second zoom factor that is larger than the first zoom factor. However, the first captured image and the second captured image are not limited to this.

  For example, the first captured image is a captured image generated through a first image capturing device among a plurality of image capturing devices in real space, and the second captured image is the image of the plurality of image capturing devices. It may be a captured image generated through the second imaging device. In this case, for example, in the overhead mode, the real space information providing unit 181 selects the camera 11 that captures a wide area of the center office 10 and uses the captured image generated by the camera 11 as the overhead image. get. Then, in the proximity mode, the real space information providing unit 181 selects a camera 11 (for example, a camera in front of the center office 10) that captures a narrower area of the center office 10, and the imaging generated by the camera 11 is selected. An image is acquired as a close-up image.

  This makes it easier to specify the position of the person depending on the arrangement of the camera 11. Further, it is not necessary to request the camera 11 to zoom. Therefore, for example, as in the case of requesting optical zoom or zoom by dolly, requests from individual terminal devices do not compete and a waiting state does not occur in any terminal device. Further, for example, the processing amount does not increase as in the case of using digital zoom.

-Captured image generated by imaging under more free conditions In the above-described example, the example in which the display mode is switched has been described, but the display screen is not limited thereto. For example, instead of switching the display mode, a captured image may be generated by imaging under more free conditions, and a display screen including the captured image may be displayed.

  For example, the captured image in the real space may be a captured image corresponding to a zoom rate selected from a plurality of zoom factors. In this case, for example, the real space information providing unit 181 requests the camera 11 via the communication unit 120 for a zoom rate specified by the user via the input unit 120. As a result, the camera 11 changes the zoom rate according to the request, and provides the terminal device 100 with a captured image generated by imaging at the changed zoom rate. Then, the real space information providing unit 181 causes the display unit 150 to display a display screen including the provided captured image. Then, when the user designates a position in the captured image, the position acquisition unit 183 acquires the position. Then, the position acquisition unit 183 provides the object selection unit 185 with the position in the captured image.

  Thereby, the user can designate a fine zoom ratio and display a desired captured image. Therefore, it becomes easier for the user to specify the position of a specific person.

  As described above, the zoom rate here does not need to be a precise value such as 1.5 times or 2 times, and directly or indirectly indicates the size of the size of the subject in the captured image. Is. For example, in particular, when zooming by changing the position of the camera 11 (for example, zooming in and out by dolly) is used, the zoom rate is not a precise value such as 1.5 times or 2 times, but the size of the subject. Directly indicating the degree of the subject (for example, a parameter indicating the degree of the approximate size of the subject) or indirectly indicating the degree of the subject (eg, the position of the camera 11 on the rail) , Etc.).

<2-3. Software configuration>
Next, an example of a software configuration of the terminal device 100 according to the present embodiment will be described. FIG. 14 is a block diagram illustrating an example of a software configuration of the terminal device 100 according to the present embodiment. Referring to FIG. 14, the terminal device 100 includes an OS (Operating System) 840 and application software. The terminal device 100 includes a soft phone 851, a super-realistic client 853, and a telephone call control function 855 as application software.

  The OS 840 is software that provides basic functions for operating the terminal device 100. The OS 840 executes each application software.

  The soft phone 851 is application software for making a phone call using the terminal device 100. The telephone unit 189 can be implemented by, for example, a soft phone 851.

  The ultra-realistic client 853 is application software for providing real space information to the terminal device 100. The real space information providing unit 181 can be implemented by, for example, a super-realistic client 853.

  Note that the super-realistic client 853 may acquire state information indicating the state of a person in the real space (for example, the center office 10) and provide it to the soft phone 851 via the OS. Then, the soft phone 851 may control outgoing calls based on the state information.

  The telephone call control function 855 is application software that acquires the communication ID of a person's communication device shown in the captured image in the display screen. The position acquisition unit 183, the object selection unit 185, and the ID acquisition unit 187 can be implemented by the telephone call control function 855.

  The telephone call control function 855 obtains the communication ID and provides it to the softphone 851 via the OS. Then, the soft phone 851 makes a call using the communication ID.

<< 3. Configuration of information management server >>
Next, an example of the configuration of the information management server 200 according to the present embodiment will be described with reference to FIGS. 15 and 16.

<3-1. Hardware configuration >>
First, an example of the hardware configuration of the information management server 200 according to the present embodiment will be described with reference to FIG. FIG. 15 is a block diagram illustrating an example of a hardware configuration of the information management server 200 according to the present embodiment. Referring to FIG. 15, the information management server 200 includes a CPU 901, a ROM 903, a RAM 905, a bus 907, a storage device 909, and a communication interface 911.

  The CPU 901 executes various processes in the information management server 200. The ROM 903 stores a program and data for causing the CPU 901 to execute processing in the information management server 200. The RAM 905 temporarily stores a program and data when the CPU 901 executes the process.

  A bus 907 connects the CPU 901, the ROM 903, and the RAM to each other. A storage device 909 and a communication interface 911 are further connected to the bus 907. The bus 907 includes, for example, a plurality of types of buses. As an example, the bus 907 includes a high-speed bus that connects the CPU 901, the ROM 903, and the RAM 905, and one or more other buses that are slower than the high-speed bus.

  The storage device 909 stores data to be temporarily or permanently stored in the information management server 200. The storage device 909 may be a magnetic storage device such as a hard disk, or may be a non-volatile memory such as EEPROM, flash memory, MRAM, FeRAM, and PRAM.

  The communication interface 911 is a communication unit included in the information management server 200, and communicates with an external device via a network (or directly). The communication interface 911 may be an interface for wireless communication. In this case, for example, a communication antenna, an RF circuit, and other communication processing circuits may be included. The communication interface 911 may be an interface for wired communication. In this case, for example, the communication interface 911 may include a LAN terminal, a transmission circuit, and other communication processing circuits.

<3-2. Functional configuration >>
Next, an example of a functional configuration of the information management server 200 according to the present embodiment will be described. FIG. 16 is a block diagram illustrating an example of a functional configuration of the information management server 200 according to the present embodiment. Referring to FIG. 16, the information management server 200 includes a communication unit 210, a storage unit 220, and a control unit 230.

(Communication unit 210)
The communication unit 210 communicates with other devices. For example, the communication unit 210 is directly connected to the LAN 19 and communicates with each device in the center office 10. Specifically, for example, the communication unit 210 communicates with the camera 11, the microphone 13, the sensor 15, and the media distribution server 17. The communication unit 210 communicates with each device in the satellite office 20 via the external network 30 and the LAN 23. Specifically, for example, the communication unit 210 communicates with the terminal device 100 and the display 21. The communication unit 210 can be implemented by the communication interface 911, for example.

(Storage unit 220)
The storage unit 220 stores a program and data for the operation of the information management server 200.

  In particular, in the present embodiment, the storage unit 220 stores various information used in the information processing system.

  As a first example, the storage unit 220 stores parameters regarding the camera 11, the microphone 13, and the sensor 15. The specific contents of the parameter are as described above.

  As a second example, the storage unit 220 stores data in a three-dimensional virtual space corresponding to the real space. The three-dimensional virtual space is, for example, a three-dimensional virtual space that simulates the center office 10. The specific contents of the three-dimensional virtual space are as described above.

  As a third example, the storage unit 220 stores person-related information. The person related information is, for example, person related information of a person in the center office 10. The specific content of the person related information is as described above.

  As a fourth example, the storage unit 220 stores object IDs and communication IDs of objects arranged in a three-dimensional virtual space in association with each other. The specific contents of the object ID and communication ID are as described above.

(Control unit 230)
The control unit 230 provides various functions of the information management server 200.

  For example, the control unit 230 provides various information used in the information processing system in response to a request. For example, in response to a request from the terminal device 100, the control unit 230 displays data in the three-dimensional virtual space, person-related information, a communication ID corresponding to the object ID, parameters regarding the camera 11, the microphone 13, and the sensor 15. provide.

  For example, the control unit 230 appropriately updates various information used in the information processing system. The control unit 230 updates the information automatically or in response to a manual instruction.

<< 4. Process flow >>
Next, an example of information processing according to the present embodiment will be described with reference to FIG. FIG. 17 is a flowchart illustrating an example of a schematic flow of information processing according to the present embodiment.

  In step S401, the position acquisition unit 183 determines whether the position in the captured image is specified by the user on the display screen of the captured image in real space. More specifically, for example, the position acquisition unit 183 determines whether a position in the close-up captured image has been designated by the user. If the position is designated, the process proceeds to step S403. Otherwise, the process repeats step S401.

  In step S403, the position acquisition unit 183 acquires a position in the captured image designated by the user.

  In step S405, the object selection unit 185 acquires data of the three-dimensional virtual space corresponding to the real space from the storage unit 170.

  In step S407, an object arranged in the three-dimensional virtual space is selected based on the acquired position in the captured image.

  In step S409, the ID acquisition unit 187 acquires the communication ID corresponding to the selected object as the communication ID on the called side.

  In step S411, the ID acquisition unit 187 acquires the communication ID on the transmission side (that is, the communication ID of the terminal device 100).

  In step S413, the telephone unit 189 performs a telephone call using the communication ID on the receiving side. Thereafter, the telephone unit 189 executes various processes for the telephone, and the process ends.

  As described above, the information processing according to the present embodiment is executed. Furthermore, an example of the startup process performed before the start of the information processing will be described with reference to FIG.

  FIG. 18 is a flowchart illustrating an example of a schematic flow of the activation process according to the present embodiment.

  In step S501, the activation process of the softphone 851 is executed. As a result, the soft phone 851 is activated.

  In step S503, registration processing related to the softphone 851 is executed. For example, as one of the registration processes of the softphone 851, registration (for example, SIP REGISRATION) is performed at the PBX 40.

  In step S505, the ultra-realistic client activation process is executed. For example, the camera 11, microphone 13, sensor 15, media distribution server 17, information management server 200, etc. used in the ultra-realistic client are specified.

  In step S507, the overhead view mode process of the super-realistic client is executed. And a series of starting processes are complete | finished.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to the example which concerns. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  For example, an example has been described in which the three-dimensional virtual space of the center office is prepared as the three-dimensional virtual space corresponding to the real space, but the present invention is not limited to this. For example, a plurality of three-dimensional spaces may be prepared. As an example, a three-dimensional virtual space corresponding to the real space may be prepared for each of a plurality of offices. For example, a three-dimensional virtual space for a satellite office, a home office, another center office, or the like may be prepared. In this case, the three-dimensional virtual space of each office may be a three-dimensional virtual space having a size corresponding to the size of each office. Further, the object selection unit of the terminal device may acquire data of a desired three-dimensional virtual space among a plurality of three-dimensional virtual spaces. Further, a three-dimensional virtual space other than the office may be prepared.

  Further, although an example in which an object corresponding to a person is selected only when the person is sitting on the seat has been described, the present invention is not limited to this. For example, an object may be selected even when a person is not sitting on a seat. As an example, when a person is sitting on the seat, the communication ID of the communication device installed in the seat is acquired, and when the person is not sitting on the seat, the communication ID of the person's portable terminal May be acquired.

  Moreover, although the example in which the communication ID is a telephone number has been described, the present invention is not limited to this. The communication ID may be an ID other than a telephone number. As an example, the communication ID may be a softphone ID other than a telephone number. As another example, the communication ID may be an ID for communication other than a telephone. For example, the communication ID may be a mail address or a short message ID. In this case, a mail or a short message may be transmitted using the communication ID.

  Further, although an example has been described in which the communication ID corresponding to the object is acquired when the object is selected, the present invention is not limited to this. For example, when an object is selected, any identification information corresponding to the object may be acquired. As an example, when an object is selected, any identification information of a person corresponding to the object may be acquired. For example, the communication ID may be acquired from this identification information.

  Moreover, although the example (object selected by the object selection part) arrange | positioned in the three-dimensional virtual space corresponding to real space corresponds to a person and it is a cylindrical object, this invention is not limited to this. . For example, the object may be an object having a different shape instead of a cylindrical object. For example, the object may correspond to something other than a person. As an example, the object may correspond to a real space region. Specifically, for example, the object may correspond to a seat and may be arranged at a three-dimensional virtual position corresponding to the position of the seat. Then, when the communication ID of the communication apparatus installed in the seat corresponds to the object, and the object is selected, the communication ID may be acquired. The object may correspond to a region wider than the seat and may exist over a three-dimensional virtual range corresponding to the range of the region. Then, when the communication ID of the communication device installed in the area corresponds to the object, and the object is selected, the communication ID may be acquired.

  Moreover, although the example in which the position of the captured image is designated by a user touch on the display screen has been described, the present invention is not limited to this. For example, the position of the captured image may be specified by the user using input means other than the touch panel. For example, the position of the captured image may be specified by clicking with a mouse, or may be specified using another input means such as a button or a keyboard.

  In addition, an example has been described in which a captured image generated by a camera, audio data generated by a microphone, and a determination result by a sensor are directly provided to the terminal device by the camera, the microphone, and the sensor, respectively. Is not limited to this. For example, these data may be provided by another device. As an example, any server (for example, a media distribution server) may acquire these data and provide these data to the terminal device.

  Moreover, although the example in which functions, such as a position acquisition part, an object selection part, and an ID acquisition part, are provided with a terminal device was demonstrated, this invention is not limited to this. For example, these functions may be provided by a device other than the terminal device. As an example, these functions may be provided by any server.

  Moreover, although the example in which the display screen of the captured image of the real space is displayed by the terminal device has been described, the present invention is not limited to this. For example, the display screen may be displayed by another device. As an example, the display screen may be displayed by a display installed in a satellite office. Then, the user may specify the position of the captured image included in the display image on the display.

  Further, the processing steps in the information processing of the present specification do not necessarily have to be executed in time series in the order described in the flowchart. For example, the processing steps in the information processing may be executed in an order different from the order described in the flowchart, or may be executed in parallel.

  In addition, it is possible to create a computer program for causing hardware such as a CPU, a ROM, and a RAM built in an information processing apparatus (for example, a terminal apparatus) to exhibit functions equivalent to the respective configurations of the information processing apparatus. A storage medium storing the computer program is also provided.

11 Camera 13 Microphone 15 Sensor 17 Media distribution server 19, 23 LAN (Local Area Network)
21 Display 30 External network 40 PBX (Private Branch eXchange)
60 Display screen (Overview mode)
61 Bird's-eye view image 63, 73, 83 Button image 70 Display screen (proximity mode)
71 Close-up image 80 Display screen (Conversation mode)
81 Opposite side captured image 81
85 Own side captured image 85
90 three-dimensional virtual space 91 object 93 virtual surface 100 terminal device 180 control unit 181 real space information providing unit 183 position acquisition unit 185 object selection unit 187 ID acquisition unit 189 telephone unit

Claims (18)

A real space information providing unit for displaying on the display unit a display screen in the first display mode for displaying the first captured image relating to the first region of the real space;
With
The real space information providing unit relates to a second region of the real space that is narrower than the first region in accordance with a designation of a position in the first captured image by a user on the display screen in the first display mode. An information processing apparatus that switches the display screen in the first display mode to the display screen in the second display mode that displays the second captured image and displays the display screen on the display unit.   The information processing apparatus according to claim 1, wherein the second area is an area including a position in the real space corresponding to a position in the first captured image designated by the user.   The information processing apparatus according to claim 2, wherein the second area is an area centering on a position in the real space corresponding to a position in the first captured image designated by the user.   The information processing apparatus according to claim 3, wherein at least the first captured image is a captured image of a part of or the entire first region.   The information processing apparatus according to claim 4, wherein at least the second captured image is a captured moving image.   The information processing apparatus according to claim 5, wherein the display screen in the second display mode includes a button display for switching a display mode from the second display mode to the first display mode.   The information processing apparatus according to claim 6, wherein, on the display screen in the second display mode, the button display is not superimposed on the second captured image and is displayed in the vicinity of the second captured image.   The information processing apparatus according to claim 7, wherein the first captured image and the second captured image are captured images generated based on an image captured by a single image capturing apparatus arranged in the real space. The information processing apparatus includes:
A communication unit for connecting to the network;
The information processing apparatus according to claim 8, wherein the communication unit acquires the first captured image and the second captured image via the network.   The information processing apparatus according to claim 9, wherein the communication unit acquires the first captured image and the second captured image from a server connected to the network via the network.   The real space information providing unit sends a request to the server to change the orientation of the imaging device so that the second area can be imaged in accordance with the designation of the position in the first captured image by the user. The information processing apparatus according to claim 10, wherein the information processing apparatus is transmitted to a unit.   The information processing apparatus according to claim 11, wherein the network is a LAN and / or the Internet. The information processing apparatus further includes the display unit,
The information processing apparatus according to claim 12, wherein the position in the first captured image is designated by the user's touch operation on the display unit. The real space information providing unit requests the server to change the optical zoom or digital zoom zoom rate of the imaging device to a specified zoom rate in response to a user's zoom rate specifying operation on the display unit. Send it to the communication department,
When the communication unit acquires the second captured image generated based on imaging by the imaging device after being changed to the specified zoom rate via the network, the real space information providing unit is configured to The information processing apparatus according to claim 13, wherein a display screen in the second display mode for displaying the second captured image after being changed to a zoom rate is displayed on the display unit. The first display mode is an overhead view mode,
The information processing apparatus according to claim 1, wherein the second display mode is a proximity mode. Displaying a display screen of a first display mode for displaying a first captured image relating to a first region of real space on a display unit;
A second captured image related to the second region of the real space that is narrower than the first region is displayed in response to the designation of the position in the first captured image by the user on the display screen in the first display mode. A step of switching the display screen of the first display mode to a display screen of two display modes and displaying on the display unit;
Including an information processing method. Computer
A real space information providing unit for displaying on the display unit a display screen in the first display mode for displaying the first captured image relating to the first region of the real space;
Is a program for functioning as
The real space information providing unit relates to a second region of the real space that is narrower than the first region in accordance with a designation of a position in the first captured image by a user on the display screen in the first display mode. The program which switches the display screen of the said 1st display mode to the display screen of the 2nd display mode which displays a 2nd captured image, and displays it on the said display part. A real space information providing unit for displaying on the display unit a display screen in the first display mode for displaying the first captured image relating to the first region of the real space;
With
The real space information providing unit relates to a second region of the real space that is narrower than the first region in accordance with a designation of a position in the first captured image by a user on the display screen in the first display mode. An information processing system for switching a display screen in the first display mode to a display screen in a second display mode for displaying a second captured image and causing the display unit to display the display screen.

Images