JP2007221437A - Remote conference system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a remote conference system in which an explainer can explain while grasping the whole atmosphere and the auditor of remote place and the auditor can listen with feeling of presence while viewing explainer's detail appearance of remote place. <P>SOLUTION: Auditor's reaction and aspect of a conference room photographed by a first imaging means 903 capable of taking a picture in omni-direction are displayed for an explainer, and the explainer's aspect photographed by a second imaging means 904 capable of taking a picture in a specified region is displayed for the auditor, so conference with feeling of presence can be carried out at remote places. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

The present invention provides a plurality of remote conference terminals by sharing media information composed of audio information and imaging information and document information composed of image information and text information via a communication line. The present invention relates to a teleconferencing system that allows a presenter and a listener in a remote location to have a more realistic sensation in a remote conference in which a conference is held between locations.

  Conventionally, in conferences, presentations, education, etc. between remote locations, media information consisting of audio information and imaging information between remote locations via a communication line, and document information consisting of image information and text information As a device realized by transmitting and receiving and sharing, a remote conference terminal device is used.

  As shown in FIG. 2, such remote conference terminal devices (2-1A to C) are installed in a plurality of conference rooms (conference rooms A to C) or the like, and a plurality of remote conference terminal devices are connected via communication lines. By connecting, users (2-2A1-3, 2-2B1-3, 2-2C3) between remote locations have realized meetings, presentations, education, and the like.

  In such a remote conference terminal device, input detecting means such as a touch panel is combined on the screen of the large screen display device shown in FIG. 3 (302), and the user (301) uses the digitizer, pen, etc. (303). There is also an apparatus that can draw text and illustrations (304) on a screen, and that the drawn data is shared between remote locations and the whiteboard is shared between remote locations (see, for example, Patent Document 1). .

  In addition, document information composed of image information and text information as described above is shared, text and illustrations drawn on the screen of the display device by the user are shared, and video information and audio information between remote locations There is also an apparatus for performing a conference by increasing the sense of reality by sharing the information (for example, see Non-Patent Document 1).

  In addition, an imaging device is used to share images between remote locations, but 360 degrees around the camera, that is, all directions can be imaged, and any location can be imaged in detail by panning, tilting, and zooming There are various imaging devices.

  By using the above-described imaging device, it is possible to capture the surrounding situation with an omnidirectional camera, and to capture and browse the attention point in detail with a pan, tilt, and zoom camera.

  As shown in FIG. 4, the omnidirectional imaging method is to perform software processing on a circular optical image obtained by the imaging means (402) using a conical mirror, a spherical mirror, a hyperboloidal mirror, etc. (401). There are a method of converting to a panoramic image or a fluoroscopic image that is an image easy to see for humans, a method of converting to a panoramic image that is an image easy to see for humans by imaging all directions using a plurality of cameras and performing software processing.

  Further, imaging that realizes imaging from the same viewpoint by making it the same as the central axis of the means (401, 402) for imaging in all directions and the means (403, 404) for imaging in detail by panning, tilting, and zooming. There is also an apparatus (for example, refer to Patent Document 2).

JP 2003-280817 A JP 2003-259350 A "Pioneer Corporation Cyber Board" 2004, http://wwwbsc.pioneer.co.jp/catalog/category/fset_cyberboard.html

  However, Patent Document 1 and Non-Patent Document 1 do not describe an omnidirectional video by an omnidirectional camera. In addition, even if an omnidirectional camera is used as the imaging device of Non-Patent Document 1, an image of an omnidirectional camera for grasping the entire remote location according to the user's situation and a partially more detailed situation are grasped. Therefore, the pan, tilt and zoom camera images could not be displayed properly.

  For this reason, it has been difficult to realize a remote conference system that is realistic and easy to use even in remote locations as if a conference was held at the same point.

  Further, since the camera is used for monitoring in Patent Document 2, pan, tilt, and zoom cameras are controlled in order to capture in detail the portion that has moved in the omnidirectional video captured by the omnidirectional camera.

  However, when the imaging device is used in a remote conference or distance learning, the video that you want to capture and display in detail with pan, tilt, and zoom cameras is not necessarily a moving part. It is required to image and display. Therefore, it was not suitable as an imaging device used in a remote conference system.

  The present invention has been made in view of the above-mentioned points, and the presenter can explain the remote audience and the overall atmosphere while grasping the details of the remote presenter. The purpose is to realize a remote conference system that can give a sense of reality by listening to the lecture.

  The remote conference system of the present invention includes a first imaging unit capable of capturing an omnidirectional image, a second imaging unit capable of capturing a detailed image in an arbitrary direction by panning, tilting, and zooming, A display unit that displays the imaging information captured by the first imaging unit or the second imaging unit and displays document information composed of image information and text information, and the imaging information and document information displayed by the display unit. The control unit is configured to control the contents and layout, and to control the transmission of imaging information and document information to a remote place, the input unit to detect an input operation to the display unit, the voice input unit, and the voice output unit. In a remote conference system for performing a conference with a remote place by sharing imaging information and document information using two or more remote conference terminal devices, Serial control means and performing different control depending on the state of said input means and voice input means.

According to the present invention, the presenter can explain while grasping the remote audience and the overall atmosphere.
Further, according to the present invention, it is possible to realize a remote conference system in which the listener can have a sense of reality by listening while viewing the detailed state of the remote presenter.

  Hereinafter, an embodiment of a control method in a remote conference system according to the present invention will be described with reference to the drawings.

  The system according to the present embodiment can realize a remote conference by simultaneously connecting a plurality of remote conference terminal devices via an IP network composed of a LAN and a WAN. A control method in the remote conference system is realized by appropriately controlling the display according to the status of the conference.

  FIG. 1 shows a configuration example of a network according to the present embodiment. Media conference server device (1-1), conference control server device (1-2), and remote conference terminal device (1-) equipped with a communication function to which imaging devices (1-4A) to (1-4B) are connected. It is a figure which shows the structure of the remote conference system connected to the IP network comprised by 3A)-(1-3B) by LAN and WAN.

  The configuration and control method of the remote conference terminal devices (1-3A) to (1-3B) will be described later.

  The media conference server device (1-1) mixes media information composed of audio information and imaging information received from a plurality of remote conference terminal devices (1-3), and mixes the media information into a plurality of remote conference terminals. This is distributed to the device (1-3).

  Further, document information composed of image information and text information received from the remote conference terminal device (1-3) may be distributed to a plurality of remote conference terminal devices (1-3).

  The conference control server device (1-2) manages terminal information including the IP addresses of the remote conference terminal devices (1-3A) to (1-3B) and the remote conference terminal device (1-3A) to (1-3B) connection control is performed.

  In the following, the remote conference terminal device (1-3A) and the remote conference terminal device (1-3B) are directly connected without using the media conference server device (1-1) and the conference control server device (1-2). A method for realizing a remote conference by transmitting and receiving the media information and the document information will be described.

  FIG. 5 is a block diagram showing a hardware configuration of the remote conference terminal device in this embodiment. The remote conference terminal device (1-3) includes a CPU (5-1), a RAM (5-2), a ROM (5-3), an HDD (5-4), a network I / F (5-5), and an input device. I / F (5-6), display I / F (5-7), imaging device I / F (5-8), audio input / output device I / F (5-9), keyboard (5-10), Digitizer (5-11), display (5-12), omnidirectional imaging device (5-13), PTZ imaging device (5-14), audio input device (5-15), audio output device (5-16) And a bus (5-17).

  In addition to controlling the entire remote conference terminal device, the CPU (5-1) performs processing instructed by program data for realizing a control method in the remote conference system.

  A RAM (5-2) is a RAM (Random Access Memory) and is used as a work memory necessary for the operation of the CPU (5-1).

  A ROM (5-3) is a ROM (Read Only Memory) and stores instructions and data necessary for the CPU (5-1).

  The HDD (5-4) is an HDD (Hard Disk Drive), which is necessary for the operation of the program data for realizing the control method in the remote conference system in addition to storing the program data itself for realizing the control method in the remote conference system. It is also assumed that data such as user database information and terminal information is stored.

  The network I / F (5-5) processes communication control for connecting to the network (5-18), but various communication interfaces are applied depending on the environment.

  When connecting directly to the access network, it is composed of various modems and terminal adapters, and when connecting to an ADSL modem or cable modem of an external device, it is composed of an Ethernet interface.

  The input device I / F (5-6) processes input from a keyboard (5-10), a mouse, a digitizer (5-11), a pen input device, and the like included in the remote conference terminal device. It is used for the operation of the terminal device.

  In the present embodiment, as shown in FIG. 5, the display (5-12) is provided with means for detecting an input operation by the digitizer (5-11) such as a touch panel, thereby directly operating the screen of the display (5-12). Is possible.

  The display I / F (5-7) and the display (5-12) are display units, and are configured by a CRT, a plasma display, a rear projection television or a liquid crystal display and a control circuit thereof. It is displayed in a part of.

  The imaging device I / F (5-8), the omnidirectional imaging device (5-13), and the PTZ imaging device (5-14) are used for capturing a real image including the situation in the vicinity of the remote conference terminal device. Yes, it is used when video communication of a remote conference is performed in a remote conference terminal device.

  As shown in FIG. 4, the omnidirectional imaging device (5-13) is an image that is easy for humans to view by performing software processing on a circular optical image obtained by using a conical mirror, a spherical mirror, a hyperboloidal mirror, etc. (401). The omnidirectional imaging is performed by a method of converting to a panoramic image or a fluoroscopic image, a method of converting a panoramic image, which is an image that is easy for humans to view, by imaging the omnidirectional image using a plurality of cameras and performing software processing.

  The PTZ imaging device (5-14) is an imaging device capable of panning, tilting, and zooming. The voice input / output device I / F (5-9) processes input / output from a voice input device (5-15) such as a microphone and a voice output device (5-16) such as a speaker provided in the remote conference terminal device. It is used when performing a teleconference voice call in the teleconference terminal device. The voice input / output device I / F (5-9) also has a function of detecting the direction in which the voice input from the voice input device (5-15) is uttered. Each block of (5-1) to (5-9) is connected to the system bus (5-17).

  FIG. 6 illustrates the arrangement of the remote conference terminal device in the present embodiment in the conference room. The omnidirectional imaging device (5-13) and the PTZ imaging device (5- The imaging information captured by 14) will be described.

  The display device (601) is a device constituting the main part of the remote conference terminal device except for the omnidirectional imaging device (5-13) and the PTZ imaging device (5-14) in FIG. As described above, the user (301, 604) can directly operate the screen of the display (5-12).

  The table (602) is installed in front of the display device (601), and users (605A) to (605H) are seated around the table for a meeting.

  The imaging device (603) is composed of the omnidirectional imaging device (5-13) and the PTZ imaging device (5-14), and the omnidirectional imaging device (5-13) is installed in the surrounding area as shown in FIG. The PTZ imaging device (5-14) captures a part of the surrounding area in detail as shown in FIG.

  Next, the functional configuration of the remote conference terminal device according to the present embodiment will be described with reference to the block diagram showing the functional configuration of the remote conference terminal device shown in FIG.

  The remote conference terminal device (1-3A) and the remote conference terminal device (1-3B) are directly connected via the IP network, and are composed of media information including voice information and imaging information, and image information and text information. The remote conference is realized in the conference room A and the conference room B, which are remote locations, by transmitting and receiving the document information to be transmitted. The voice input means (901), the voice output means (902), and the first imaging A means (903), a second imaging means (904), a display means (905), an input means (906), a control means (907), a receiving means (908), and a transmitting means (909).

  The voice input means (901) acquires the voice of the conference room using the voice input device (5-15), and the acquired voice is encoded into a format such as G.711 or G.729 using a voice codec. To do.

  The encoded audio data is transmitted to the remote conference terminal device connected by the transmission means (909).

  The voice input means (901) detects the direction in which the acquired voice is uttered, and transmits information on the detected direction of the utterance source to the control means (907).

  The voice output means (902) is a means for decoding the voice data transmitted from the remote conference terminal device at the remote location and received by the receiving means (908) by the voice codec and reproducing it using the voice output device (5-16). is there.

  The first imaging means (903) acquires video of the conference participants and the entire conference room using the omnidirectional imaging device (5-13), and the acquired imaging information uses a video codec for the acquired imaging information. Or it encodes to formats, such as H.263.

  The encoded imaging information is transmitted to the remote conference terminal device connected by the transmission means (909).

  The second image pickup means (904) uses the PTZ image pickup device (5-14) to pick up an image of a part of the conference room by panning, tilting and zooming, and more detailed video than the omnidirectional image pickup device (5-13). The acquired imaging information is encoded into a format such as H.264 or H.263 using a video codec.

  The encoded imaging information is transmitted to the remote conference terminal device connected by the transmission means (909). The pan, tilt and zoom controls for determining the area to be imaged are performed by a control means (907) described later.

  The display means (905) decodes the imaging information transmitted from the remote conference terminal device at the remote location and received by the receiving means (908) by the video codec, displays the imaging information using the display (5-12), and Document information composed of image information and text information received by the receiving means (908) is displayed.

  Control of the content and display position of the imaging information and document information displayed by the display means (905) is performed by a control means (907) described later.

  In the present embodiment, the same document information is displayed on the display means (905) of each connected remote conference terminal device and is updated in real time, thereby assisting collaboration between remote locations.

  The input means (906) can not only operate the remote conference terminal device but also can draw text and illustrations on the screen of the display means (905) using the digitizer (5-11).

  The drawn text and illustration are transmitted as the document information to the remote conference terminal device connected by the transmission means (909).

  The receiving means (908) transmits the imaging information and the document information to the remote conference terminal device connected as described above.

  The transmission means (909) receives the imaging information and the document information transmitted from the remote conference terminal device connected at the remote place as described above.

  Control performed by the control means (907) will be described below with reference to the flowchart of FIG. Here, the remote conference terminal device (1-3A) installed in the conference room A in FIG. 1 and the remote conference terminal device (1-3B) installed in the conference room B in FIG.

  When the remote conference terminal device (1-3A) and the remote conference terminal device (1-3B) are directly connected via the IP network, the media information and the document information are transmitted (909) and received ( 908), a remote conference is started (10-1).

  When the input means (906) detects the input from the digitizer (5-11) (10-2), the control means (907) uses the second input to image the vicinity input by the digitizer (5-11). The panning, tilting, and zooming of the image pickup means (904) are controlled (10-4).

  The second imaging means (904) controlled by the control means (907) images the vicinity input by the digitizer (5-11) as shown in FIG. 5, and the captured imaging information is transmitted by the transmission means (909). It is transmitted to the connected remote conference terminal device (10-5).

  In addition, when the voice input means (901) detects the voice produced by the user and detects the direction of the voice (10-3), the control means (907) is spoken to capture the vicinity of the voice. The pan, tilt and zoom of the second image pickup means (904) are controlled in the direction (10-4).

  The second image pickup means (904) controlled by the control means (907) picks up the vicinity of the utterance, and the picked-up image pickup information is transmitted to the remote conference terminal device connected by the transmission means (909). (10-5).

  When the input means (906) does not detect the input from the digitizer (5-11) and the voice input means (901) does not detect the voice generated by the user, the first imaging means (903) participates in the conference. The entire conference room including the person is imaged, and the captured image information is transmitted to the remote conference terminal device at the remote location connected by the transmission means (909) (10-5).

  Here, the imaging information of the entire conference room including the conference participants captured by the omnidirectional imaging device (5-13) will be described with reference to FIG. 7 for the conference room B in which the remote conference terminal device (1-3B) is installed. .

  The display device (1101) is a device constituting the main part of the remote conference terminal device excluding the omnidirectional imaging device (5-13) and the PTZ imaging device (5-14). As shown in FIG. 301, 604) can directly operate the screen of the display (5-12).

  The table (1102) is installed in front of the display device (1101), and users (1104A) to (1104H) are seated around the table to hold a conference.

  The imaging device (1103) is composed of the omnidirectional imaging device (5-13) and the PTZ imaging device (5-14), and the omnidirectional imaging device (5-13) is installed in the surrounding area as shown in FIG. Image all directions.

  Returning to the flowchart of FIG. 10 again, the control of the control means (907) will be described. The receiving means (908) receives the imaging information imaged by the first imaging means (903) or the imaging information imaged by the second imaging means (904) transmitted from the remote conference terminal device at the remote location. (10-7).

  Hereinafter, regarding the control of the display means (905) performed by the control means (907), the input means (906) of the remote conference terminal device (1-3A) receives the input from the digitizer (5-11) in the conference room A of FIG. In the conference room B of FIG. 1, the input means (906) of the remote conference terminal device (1-3B) does not detect the input by the digitizer (5-11), or the voice input means (901) A case will be described as an example where no voice is detected.

  FIG. 13 shows that in a conference room A, a user (1301) inputs a text (1304) using a digitizer (1303, 5-11) to a display means (1302) with input means of a remote conference terminal device (1-3A). This is shown in the figure.

  The control means (907) of the remote conference terminal device (1-3A) that has detected the input by the digitizer (1303, 5-11) as shown in FIG. 13 picks up the image picked up by the received first image pickup means (903). The information (1305) is displayed at a position that does not prevent the user (1301) from inputting the text (1304).

  Thus, the user (1301) who explains using the digitizer (1303, 5-11) can explain the situation of the entire conference room B in the remote location and the reaction of the audience.

  Further, as shown in FIG. 14, the image information (1405) captured by the first image capturing means (903) received is input by being translucently transmitted so as not to prevent the user (1401) from inputting the text (1404). It may be displayed in the vicinity.

  As a result, the user (1401) who explains using the digitizer (1403, 5-11) performs the explanation while grasping the state of the entire conference room B in the remote location and the reaction of the audience without moving the viewpoint in the input. It is possible.

  FIG. 15 illustrates that in the conference room B, no input is made by the digitizer (5-11) on the display means with input means (1501) of the remote conference terminal device (1-3B).

  Control of the remote conference terminal device (1-3B) when the input means (906) does not detect the input by the digitizer (5-11) and the voice input means (901) does not detect the voice by the user's speech The means (907) displays the received image information picked up by the second image pickup means (903) at a position that does not interfere with the text (1502) input by the user (1301) in the conference room A (1503).

  Thereby, the user of the conference room B who listens to the explanation of the user (1301) of the conference room A can listen to the explanation while grasping the detailed state of the presenter.

  By sharing media information composed of audio information and imaging information and document information composed of image information and text information via a communication line among a plurality of remote conference terminals having the above functions In a remote conference where a conference is held between a plurality of remote locations, the omnidirectional camera capable of capturing an omnidirectional image and the display of imaging information captured by a pan, tilt, and zoom camera capable of capturing an image of any location in detail are displayed. By controlling appropriately according to the situation, the presenter can not only explain the reaction of the listener and the state of the conference room, but the listener can explain the detailed state of the presenter. It is possible to realize a control method in the remote conference system that enables the user to listen.

(Other embodiments)
In the above-described embodiment, the remote conference has been described using two remote conference terminal devices. However, the remote conference system of the present invention is not limited to two remote conference terminal devices. When a conference is performed, the media conference server device and the conference control server device described in the above embodiment can also be used.

  In the above embodiment, the transmission of the imaging information imaged by the first imaging unit or the second imaging unit is performed alternatively, but the first imaging unit and the second imaging unit are used. It is also possible to transmit both pieces of imaging information imaged in accordance with the above and the selection of the imaging information to be displayed can be determined by the control means on the receiving side.

  Another object of the present invention is to supply a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and a computer such as the system reads and executes the program codes from the storage medium. Is also achieved.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  In addition, the case where the functions of the above-described embodiment are realized by performing part or all of the actual processing by an OS or the like running on the computer based on the instruction of the program code read by the computer. It is.

  Further, after the program code read from the storage medium is written in a memory provided in a function expansion unit connected to the computer, the CPU or the like performs actual processing based on the instruction of the program code, and the above-described processing is performed. The case where the functions of the embodiment are realized is also included.

It is a figure which shows the structure of the network in the remote conference system which concerns on embodiment of this invention. It is a figure which shows the structure of the network in the conventional remote conference system. It is a figure which shows the remote conference apparatus which can be directly input into the conventional screen. It is a figure which shows the conventional imaging device which can image omnidirectional and can pan, tilt, and zoom. It is a block diagram which shows the hardware constitutions of the remote conference terminal device in embodiment of this invention. It is a block diagram which shows arrangement | positioning of the conference room and remote conference terminal device in embodiment of this invention. It is a figure which shows the image | video which the omnidirectional imaging device in embodiment of this invention images. It is a figure which shows the image | video which the PTZ imaging device in embodiment of this invention images. It is a block diagram which shows the function structure of the remote conference terminal device in embodiment of this invention. It is a flowchart which shows the process of the remote conference terminal device in embodiment of this invention. It is a block diagram which shows arrangement | positioning of the remote conference room and remote conference terminal device in embodiment of this invention. It is a figure which shows the image | video which the omnidirectional imaging device of the remote place in embodiment of this invention images. It is a figure for demonstrating the display method to the display means in embodiment of this invention. It is a figure for demonstrating the display method to the display means in one Embodiment of this invention. It is a figure for demonstrating the display method to the display means in embodiment of this invention.

Explanation of symbols

1-1 Media Conference Server Device 1-2 Conference Control Server Device 1-3 Remote Conference Terminal Device 1-4 Imaging Device 1-5 User 2-1 Remote Conference Terminal Device 2-2 User 301 User 302 Display Device with Input Means 303 Digitizer 304 Text and illustration 401 Mirror 402 Imaging means 403 Pan head 404 Imaging means 5-1 CPU
5-2 RAM
5-3 ROM
5-4 HDD
5-5 Network I / F
5-6 Input device I / F
5-7 Display I / F
5-8 Imaging device I / F
5-9 Audio input / output equipment I / F
5-10 keyboard 5-11 digitizer 5-12 display 5-13 omnidirectional imaging device 5-14 PTZ imaging device 5-15 audio input device 5-16 audio output device 601 display device 602 table 603 imaging device 604 user 605 user 901 Audio input means 902 Audio output means 903 First image pickup means 904 Second image pickup means 905 Display means 906 input means 907 control means 908 reception means 909 transmission means 1101 display device 1102 table 1103 image pickup device 1104 user 1301 user 1302 with input means Display means 1303 Digitizer 1304 Text 1305 Imaging information 1401 User 1402 Display means with input means 1403 Digitizer 1404 Text 1405 Imaging information 1501 Display hand with input means 1502 text 1503 imaging information

Claims (5)

A first imaging means capable of imaging an omnidirectional image;
A second imaging means capable of capturing a detailed image in an arbitrary direction by panning, tilting and zooming;
Display means for displaying imaging information imaged by the first imaging means or the second imaging means at a remote location and displaying document information composed of image information and text information;
Control means for controlling the contents and layout of the imaging information and document information displayed by the display means, and for controlling the transmission of the imaging information and document information to a remote place;
Input means for detecting an input operation to the display means;
In a remote conference system for performing a conference with a remote place by sharing imaging information and document information using two or more remote conference terminal devices composed of voice input means and voice output means,
The remote conferencing system characterized in that the control means performs different control depending on the states of the input means and the voice input means.   When the control means detects an input to the input means by a user, the control means controls the second image pickup means to pick up an image in the vicinity where the input to the input means is detected. And displaying the imaging information captured by the first imaging unit in the remote conference terminal device at a remote location on the display unit. The remote conference system according to 1.   When the control means detects a voice input to the voice input means by the user's utterance, the control means controls the second imaging means to pick up an image of the vicinity where the user's utterance is detected. And displaying the image information captured by the first image capturing means in the remote teleconference terminal device at the local remote conference terminal device on the display means. Item 1. The teleconferencing system according to item 1.   2. The teleconference system according to claim 1, wherein the first image pickup means and the second image pickup means pick up images from the same viewpoint by matching the central axes.   The control unit is configured to display the imaging information at a position and a size that do not hinder the operation of the user and the display of the document information when the imaging information is displayed on the display unit. The remote conference system according to claim 1.

Images