JPWO2009004891A1 - Bidirectional communication system, communication terminal, communication server, communication method, and communication control program used in the bidirectional communication system - Google Patents

Abstract

Provided is a bidirectional communication system in which images and character strings are superimposed on a display screen of each communication terminal without imposing a processing load on the communication server. Each terminal state information is transmitted from the conference terminals 21, 22,..., 2n to the multipoint conference apparatus 1, and each image received by the multipoint conference apparatus 1 from the conference terminals 21, 22,. The encoded data is transmitted to each of the conference terminals 21, 22,..., 2n, and the impose image identifier and the impose position information corresponding to the received terminal state information are included in each encoded image data. .., 2n are inserted and transmitted to the conference terminals 21, 22,..., 2n. The selected impose image is implied in the images displayed on the conference terminals 21, 22,..., 2n based on the impose position information.

Description

  The present invention relates to a bidirectional communication system, a communication method and a communication control program used in the bidirectional communication system. For example, image encoded data is transmitted / received between a plurality of communication terminals arranged at multiple points. Bidirectional communication system suitable for use when a conference or the like is performed by superimposing a character string or an image on a display screen of a communication terminal, a communication terminal, a communication server, a communication method, and communication control used in the bidirectional communication system Regarding the program.

  In communication networks, recently, image encoded data and audio encoded data are transmitted at high speed using packets by ADSL (Asymmetric Digital Subscriber Line) or optical communication technology. Thereby, for example, a bidirectional communication system and a bidirectional communication system compatible with VoIP (Voice over Internet Protocol) and TVoIP (TV over Internet Protocol) are rapidly spreading.

  A two-way communication system related to this type of technology includes, for example, a conference server (communication server) disposed in a central part of a company or the like, and a conference terminal (communication terminal) disposed in a plurality of remote locations. ). Each conference terminal is connected to the conference server via a communication line such as the Internet or an intranet, and a stream (image stream) including encoded image data is transmitted between the conference terminals via the conference server. Sent and received. In the conference server, a character string or an image owned by the conference server may be imposed on the encoded image data transmitted from each conference terminal and transmitted to the conference terminal. . For example, when a new participant joins a conference, the conference server encodes the image data with the entrance mark imposed and transmits it to each conference terminal. In this case, an image stream received from each conference terminal is decoded by the conference server, and a character string or an image corresponding to the entry mark is imposed and then encoded and transmitted to each conference terminal.

In addition to the above two-way communication system, as a related technique, for example, there is one described in Patent Document 1.
In the two-way communication system described in Patent Document 1, characters and images to be combined designated by operation input means are converted into bitmap (BMP) image data by WWW (World Wide Web) control means in a terminal, The BMP image data is output to the image codec control means. In the image codec control means, this BMP image data is converted into data for one slice of MPEG-1 and transmitted to the server apparatus. In the server device, this data is replaced with the fourth slice image of the terminal and synthesized with the first to third slice images.

  Moreover, in the control method of the video conference communication terminal device described in Patent Document 2, character information input by the terminal itself is converted into image information and superimposed and displayed, so that character information is input. A user other than the user can confirm the content of the input character information, for example, the content of character information communication, and the communication conference session is smoothly performed.

  Further, in the interactive communication system described in Patent Document 3, the participation point name is superimposed and displayed on the monitor / display of each terminal device. Is confirmed by the participants.

Further, in the video conference system described in Patent Document 4, the imaging position (for example, the imaging direction and magnification) of the imaging means is registered according to the conference participant, and at the same time, information such as the person's name, affiliation, and title Is registered as additional information. When the imaging means matches or approaches the registered imaging position, an image in which additional information such as a name is combined with the person image is obtained. Thereby, introduction and confirmation of a person become unnecessary.
JP 2004-023706 (abstract, FIGS. 4 and 23) Japanese Patent Application Laid-Open No. 08-307845 (Abstract, FIG. 2) Japanese Patent Laid-Open No. 05-063830 (Abstract, FIG. 1) Japanese Patent Application Laid-Open No. 08-051614 (Abstract, FIG. 3)

However, the bidirectional communication system has the following problems.
That is, since the received image stream is decoded by the conference server, and character strings and images are encoded and encoded, there is a problem that the processing load on the conference server is heavy. The conference server is intended for a large number of unspecified users, and does not own the image for impose that corresponds to the individual user. There is a problem that it is not possible.

  Further, in the bidirectional communication system described in Patent Document 1, characters and images are uniquely converted into BMP image data by the WWW control unit at the terminal, but the processing load of the server device is not reduced. The above problems are not improved.

  The video conference communication terminal device described in Patent Document 2 is such that character information input at its own terminal is converted into image information, superimposed and displayed, and does not improve the above problems. .

  In the two-way communication system described in Patent Document 3, the name of a participating point is displayed on a monitor / display, and does not improve the above problem.

  The video conference system described in Patent Document 4 provides an image in which additional information such as a name is combined with a person image when an imaging unit matches or approaches a registered imaging position. Is not improved.

  The present invention has been made in view of the above-described circumstances. An image or a character string is imposed at a specified timing without imposing a processing load on the conference server, and an imposed image that suits the user's preference. It is an object to provide a two-way communication system, a communication terminal, a communication server, a communication method, and a communication control program used in the two-way communication system.

  In order to solve the above-described problem, the present invention includes a plurality of communication terminals and a communication server that transmits and receives each image encoded data to and from each of the communication terminals. The present invention relates to a bidirectional communication system having display means for displaying an image corresponding to the transmitted image encoded data, and each communication terminal is superimposed on the image displayed on the display means. One or a plurality of impose images are held in association with each impose image identifier in a one-to-one correspondence, and the impose image for selecting the impose image corresponding to the given impose image identifier is stored. The impose image selected by the hold selection means and the impose image hold selection means is displayed on the display means based on the given impose position information. Image composing means for superimposing in the image, and transmitting means for transmitting the encoded image data to the communication server. The communication server includes the impose image identifier and impose position information. And the encoded image data received from each communication terminal is transmitted to any communication terminal, and the impose image identifier and the imposed position information are transmitted to any communication terminal. It is characterized by being configured to transmit.

  In order to solve the above problem, the present invention provides a communication method for transmitting and receiving encoded image data to and from a communication terminal, and when transmitting the encoded image data to the communication terminal, An impose image identifier and an impose position information are also transmitted, an impose image is selected based on the transmitted impose image identifier, and the impose image identifier is included in the image based on the transmitted impose position information. It is characterized by superimposing a pause image.

  According to the present invention, there is provided a bidirectional communication system in which an image or a character string is imposed at a specified timing without imposing a processing load on the conference server, and an imposed image can be obtained according to the user's preference. A communication terminal, a communication server, a communication method, and a communication control program used for a bidirectional communication system can be provided.

The present invention is configured such that the communication server inserts the impose image identifier and the impose position information into each encoded image data and transmits it to each communication terminal. The communication server also includes the impose image identifier and the impose image information. The present invention relates to a bidirectional communication system configured to transmit pause position information to each communication terminal separately from each image encoded data, a communication method used in the bidirectional communication system, and a communication control program.
For example, as shown in FIG. 1, a communication server that transmits and receives each encoded image data between a plurality of conference terminals 2 ₁ to 2 _n as an example of a communication terminal and each of the conference terminals 2 ₁ to 2 _n. An example of display means for displaying images corresponding to the encoded image data transmitted from the multipoint conference device 1, each of the conference terminals 2 ₁ to 2 _n includes the multipoint conference device 1 as an example. In the interactive communication system having the display unit 16, each of the conference terminals 2 ₁ to 2 _n has one or a plurality of impose images for superimposing in the image displayed on the display unit 16. Are stored in association with each impose image identifier in a one-to-one manner, and an impose image holding / selecting unit 210 that selects an impose image corresponding to the given impose image identifier, and an impose image hold Select part The image composition unit 14 for superimposing the impose image selected in 10 on the image displayed on the display unit 16 on the basis of the given impose position information; The multipoint conference device 1 holds each impose image identifier and each impose position information, and is received from each conference terminal 2 ₁ to 2 _n. Each encoded image data is transmitted to any conference terminal 2 ₁ to 2 _n via the communication unit 230, and the impose image identifier and the imposed position information are communicated to any conference terminal 2 ₁ to 2 _n . It is assumed that the transmission is performed via the H.230.
According to this configuration, the encoded image data is transmitted to the multipoint conference apparatus 1 by the communication unit 211 of each conference terminal 2 ₁ to 2 _{n. In} the multipoint conference apparatus 1, each conference terminal 2 ₁ to 2 is transmitted. together with the respective coded image data received from the _n is transmitted to any conference terminals 2 ₁ to 2 _n, transmission superimpose image identifier and superimpose position information to any conference terminals 2 ₁ to 2 _n Is done. In each of the conference terminals 2 _{1 to} 2 _n , the impose image holding / selecting unit 210 selects an impose image corresponding to the impose image identifier given from the multipoint conference device 1, and the image compositing unit 14 The impose image selected by the impose image holding / selecting unit 210 is superimposed on the image displayed on the display unit 16 based on the impose position information given from the multipoint conference apparatus 1. The Thereby, in the multipoint conference apparatus, it is not necessary to perform decoding, imposing, and encoding, and the processing load can be reduced, and an image suitable for each user's preference of each of the conference terminals 2 ₁ to 2 _n can be displayed. Can be imposed.
Hereinafter, preferred embodiments of the present invention will be described.

[First Embodiment]
FIG. 2 is a block diagram showing an electrical configuration of a main part of the bidirectional communication system according to the first embodiment in which the present invention is preferably implemented.
As shown in the figure, the bidirectional communication system of this example is a multipoint conference system, and includes a multipoint conference device 1, conference terminals 2 ₁ , 2 ₂ ,..., 2 _n and an external device 3. It is configured.
Multipoint conference apparatus 1, for example, is disposed centrally of the corporate conference terminals _{2 1,} 2 2, ..., the image stream _vs 1, vs ₂ including an image coded data to and from the _{2 n,} ... , Vs _n are respectively transmitted and received. The conference terminals 2 ₁ , 2 ₂ ,..., 2 _n are, for example, disposed at a plurality of remote locations and connected to the multipoint conference device 1 via a communication line such as the Internet or an intranet. It has a display unit (not shown) for displaying images corresponding to the image streams vs ₁ , vs ₂ ,..., Vs _n transmitted from the multipoint conference apparatus 1. The conference terminals 2 ₁ , 2 ₂ ,..., 2 _n may be mobile devices such as mobile phone terminals. The external device 3 as an example of the external state detection device is composed of, for example, an earthquake sensor or an air temperature sensor, and detects the external state of the multipoint conference system to represent the state where the earthquake has occurred, the air temperature, and the like. Alternatively, a plurality of types of external state information ps are generated. The external device 3 may generate the external state information ps corresponding to the state in which the organizer participates in the conference, for example.

Figure 3 is a block diagram showing an electrical configuration of the conference terminal 2 ₁ in FIG.
The conference terminal 2 _1, as shown in FIG. 3, a communication control program for operating the CPU (central processing unit) 2a and the CPU2a as a computer for controlling the 2 ₁ whole the conference terminals is recorded And a ROM (Read Only Memory) 2b. In particular, in this embodiment, the conference terminal 2 ₁ includes a receiver 11, a superimpose information analyzing unit 12, a decoder 13 as an example of a decoding unit, an image synthesizing unit 14 as an example of the image combining means , An impose data selection unit 15 as an example of an imposition image holding selection unit, a display unit 16, and a transmission unit 17 as an example of a transmission unit. The receiving unit 11 receives the image stream vs ₁ transmitted from the multipoint conference apparatus 1. The impose information analysis unit 12 outputs the encoded image data va included in the image stream vs ₁ received by the receiving unit 11, and analyzes the encoded image data va to analyze the impose image identifier cm and the impose. The position information cn is acquired. The decoder 13 decodes an image corresponding to the encoded image data va output from the impose information analysis unit 12 and outputs an image signal vd.

The impose data selection unit 15 stores a plurality of impose images for superimposing in the image displayed on the display unit 16 in a one-to-one correspondence with each impose image identifier as a data set. The impose image vi corresponding to the impose image identifier cm given from the impose information analysis unit 12 is selected. The data set includes, for example, data provided and held in advance from the outside, data originally created by the user, data downloaded from the website by the user, and the like. The image synthesis unit 14 synthesizes the impose image vi selected by the impose data selection unit 15 with the image signal vd based on the impose position information cn given from the impose information analysis unit 12. Then, the image signal ve is output after being superimposed on the image displayed on the display unit 16. The display unit 16 displays an image based on the image signal ve. Transmitter 17, the use state of the image stream vs _1, and conference terminals 2 ₁ (e.g., a state in which the conference terminals 2 ₁ has entered the conference room, exit state, the state such as pending) a plurality of types that represent Terminal state information si ₁ is transmitted to the multipoint conference apparatus 1. Further, the conference terminals ₂ 2, ..., _{2 n} also has the same configuration as that of the conference terminals _{2 1.}
The reception unit 11 and the transmission unit 17 correspond to the communication unit 211, the impose information analysis unit 12, the decoder 13, and the impose data selection unit 15 in FIG. 1 correspond to the impose image selection holding unit 210 in FIG. 1. .

FIG. 4 is a block diagram showing an electrical configuration of the multipoint conference apparatus 1 in FIG.
As shown in FIG. 4, the multipoint conference apparatus 1 stores a CPU (Central Processing Unit) 1a as a computer that controls the entire multipoint conference apparatus 1 and a communication control program for operating the CPU 1a. ROM 1b. In particular, in this embodiment, the multipoint conference apparatus 1, the receiving unit _{21 1,} 21 2, ..., and 21 _n, imposed information insertion section _{22 1,} 22 2, ..., and 22 _n, an image selection section 23 , imposed information insertion section _{24 1,} 24 2, ..., and 24 _n, the transmission unit _{25 1,} 25 2, ..., and 25 _n, and a timing acquisition unit 26.. Reception unit _{21 1,} 21 2, ..., 21 _n is conference terminal _{2 1,} 2 2, ..., the image from the _{2 n} streams _{vs 1,} vs 2, ..., to receive vs _n. The timing acquisition unit 26 is based on the image streams vs ₁ , vs ₂ ,..., Vs _n received by the reception units 21 ₁ , 21 ₂ ,..., 21 _n and the external state information ps generated by the external device 3. Each timing for inserting each impose image identifier and each impose position information into each image encoded data of the image stream vs ₁ , vs ₂ ,..., Vs _n is acquired.

The impose information insertion units 22 ₁ , 22 ₂ ,..., 22 _n hold each impose image identifier and each impose position information in association with each terminal state information and each external state information. at each timing to be acquired, the terminal state information _{si 1, si 2, ...,} si n or external status information receiving section ₂₁ 1 a superimpose image identifier and imposing position information corresponding to _ps, 21 2, ..., 21 _n image stream _{vs 1,} vs 2 received in, ..., the image stream _{vt 1,} vt 2 is inserted into the coded image data of vs _n, ..., and outputs it as vt _n. In this case, for example, ITU-T (International Telecommunication Union-Telecommunication Standardization Sector recommendation) H.264. Based on a moving picture coding system such as H.264 and a moving picture compression standard such as MPEG (Moving Picture Experts Group) -4, the impose image identifier and the impose position information are inserted into each picture coded data. Is done. Note that the impose information insertion units 22 ₁ , 22 ₂ ,..., 22 _n store the impose image identifier and the impose position information in the encoded image data when the timing acquisition unit 26 does not acquire the respective timings. Not inserted inside.

Image selection section 23, superimpose information insertion section _{22 1,} 22 2, ..., the image stream _{vt 1,} vt 2 output from the 22 _n, ..., type the vt _n, conference terminals _{2 1, 2} 2, ..., of any in _{2 n} image stream _{vs 1,} vs 2, ..., you select whether to send a vs _n. The selection method in this case may be an arbitrary method such as a method arbitrarily performed by each user of the conference terminals 2 ₁ , 2 ₂ ,..., 2 _{n or} a method of switching at regular intervals. The impose information insertion units 24 ₁ , 24 ₂ ,..., 24 _n hold each impose image identifier and each impose position information in a one-to-one correspondence with each terminal state information and each external state information. at each timing acquired by the acquiring unit 26, terminal status information si _1, si 2, _..., outputted from the superimpose image identifier and superimpose position information image selecting unit 23 corresponding to si _n or external status information ps image stream _{vm 1, vm 2, ...,} the image is inserted into the coded image data of the vm _n streams _{vu 1, vu 2, ...,} and outputs it as vu _n. Note that the impose information insertion units 24 ₁ , 24 ₂ ,..., 24 _n store the impose image identifier and the impose position information in the encoded image data when the timing acquisition unit 26 does not acquire the respective timings. Do not insert into. Transmission unit _{25 1,} 25 2, ..., 25 _n, the image stream _{vu 1,} vu 2, ..., vu _n a conference terminal _{2 1,} 2 2, ..., transmits respectively to _{2 n.} In this case, for example, RTP (Real Time Protocol) is used based on a predetermined moving picture transmission protocol. The receiving units 21 ₁ , 21 ₂ ,..., 21 _n and the transmitting units 25 ₁ , 25 ₂ ,..., 25 _n correspond to the communication unit 230 in FIG.

FIG. 5 is a block diagram showing an electrical configuration of an image processing unit 30 provided in place of the image selection unit 23 in FIG.
As shown in FIG. 5, the image processing unit 30 includes decoders 31 ₁ , 31 ₂ ,..., 31 _n , mixers 32 ₁ , 32 ₂ ,..., 32 _n and encoders 33 ₁ , 33 ₂ ,. 33 _n . Decoder _{31 1, 31 2, ...,} 31 n , the image stream _{vt 1,} vt 2, ..., and outputs each decoded image data by decoding the respective coded image data for vt _n. Mixers _{32 1,} 32 2, ..., 32 _n, the decoder _{31 1,} 31 2, ..., mixes the respective decoded image data output from the 31 _n. The encoders 33 ₁ , 33 ₂ ,..., 33 _n encode the image data output from the mixers 32 ₁ , 32 ₂ ,..., 32 _n into image encoded data to generate image streams vm ₁ , vm ₂ ,. Output as vm _n .

6 and 7 are diagrams showing an example of the data structure when inserting the impose image identifier and the impose position information in each image encoded data based on the moving picture compression standard; MPEG-4.
With reference to these drawings, processing contents of a communication method used in the bidirectional communication system (multipoint conference system) of this example will be described.
This multipoint conference system, the conference terminal _{2 1,} 2 2, ..., ₂ from _{2 n} the conference terminal _1, 2 2, ..., a plurality of types of terminal state information multipoint conference representing the use state of the _{2 n} It is transmitted to the device 1 (transmission process). Further, the multipoint conference apparatus 1 holds each impose image identifier and each impose position information in a one-to-one correspondence with each terminal state information, and the conference terminals 2 ₁ , 2 ₂ ,. each image coding data terminal for any each meeting received from _{n 2 1,} 2 2, ..., while being transmitted to the _{2 n,} the respective conference terminals _{2 1,} 2 2, ..., received from the _{2 n} The impose image identifier and the impose position information corresponding to each terminal state information are inserted into each image encoded data and transmitted to any conference terminals 2 ₁ , 2 ₂ ,..., 2 _n .

By the conference terminals 2 ₁ , 2 ₂ ,..., 2 _n , one or a plurality of impose images for superimposing in the image displayed on the display unit 16 are in one-to-one correspondence with each impose image identifier. And an impose image corresponding to the impose image identifier given from the multipoint conference apparatus 1 is selected (impose image holding / selecting process), and the imposing image holding / selecting process is performed. The impose image selected in (1) is superimposed on the image displayed on the display unit 16 based on the impose position information given from the multipoint conference apparatus 1 (image composition processing).

That is, the conference terminal _{2 1,} 2 2, ..., In _{2 n,} the transmitting unit 17, an image stream _{vs 1,} vs 2, ..., vs _n and terminal status information _{si 1,} si 2, ..., si _n multi It is transmitted to the point conference device 1. In multipoint conference apparatus 1, the receiving unit _{21 1,} 21 2, ..., a 21 _n, an image stream _{vs 1,} vs 2, ..., vs _n and terminal status information _{si 1,} si 2, ..., si _n is received The Based on the terminal state information si ₁ , si ₂ ,..., Si _n received by the receiving units 21 ₁ , 21 ₂ ,..., 21 _n and the external state information ps generated by the external device 3 by the timing acquisition unit 26. Thus, each timing for inserting each impose image identifier and each impose position information into each image encoded data of the image stream vs ₁ , vs ₂ ,..., Vs _n is acquired.

Imposed information insertion section _{22 1,} 22 2, ..., the 22 _n, at each timing acquired by the timing acquisition unit 26, the terminal state information _{si 1,} si 2, ..., corresponding to si _n or external status information ps superimpose image identifier and superimpose position information, receiving unit _{21 1,} 21 2, ..., the image stream _vs 1 received at 21 _n, vs 2, ..., are inserted into the respective coded image data for vs _n Output as image streams vt ₁ , vt ₂ ,..., Vt _n . In this case, the inserted impose image identifier corresponds to information indicating that _{one of} the conference terminals 2 ₁ , 2 ₂ ,..., 2 _n is on hold, for example. In addition, the impose information insertion units 22 ₁ , 22 ₂ ,..., 22 _n indicate how many frames from the frame at which the impose image identifier is inserted and how many frames are to be imposed. Information such as impose start timing information and impose end timing information may be inserted.

  As a method for inserting the above information and the imposed image identifier into the encoded image data, for example, “User_data” existing immediately after “User_data_start_code” of MPEG-4, A variable “User_data_unregistered” or the like existing in one NAL (Network Abstraction Layer) unit type called H.264 SEI (Supplemental enhancement information) is used. “User_data” of MPEG-4 has an area of 8 bits (1 byte), but the information and the imposed image identifier can be stored by using a plurality of areas. H. The H.264 variable “User_data_unregistered” can store the above information and the imposed image identifier because 1-byte data has a payload size (payloadsize).

Information for imposing an image on a designated frame in the conference terminals 2 ₁ , 2 ₂ ,..., 2 _n by inserting the above information and the imposed image identifier into a region before the frame to be image-imposed. Can be added. However, in MPEG-4, “User_data does not exist in a frame on the MPEG-4 syntax. Therefore,“ User_data ”is inserted in a format that can be adapted to the MPEG-4 syntax. As shown in FIG. 6, in order to insert the above information and the imposed image identifier into the MPEG-4 variable “User_data”, DCI (Decoder Configuration Information) including the “User_data” is VOP (Video Object Plane, MPEG- “User_data” exists in DCI or immediately after DCI, but either of the above information and the position of an imposed image identifier is inserted. As a method of inserting the DCI, the start code on the image encoded data is searched, and the above information or Method of inserting a DCI containing Npozu image identifier can be considered. DCI used herein, the DCI of the image coded data input to the imposed information insertion section 22 _1, including the information and superimpose the image identifier " Use “User_data” added.

  Also, as shown in FIG. 7, “User_data” included in a GOV (Group of Vop) header may be used instead of the DCI in FIG. Since the GOV header needs to exist before the I-VOP (I; Intra, in a frame), the next VOP needs to be an I-VOP. In addition, when a GOV header is added, the time information included in the previous GOV header, the time information existing at the head of each VOP, or the time information owned by the apparatus is used. It is necessary to put time information in “time_code”. Further, the variable “closed_gov” and the variable “broken_link” exist in the GOV header. However, in the present embodiment, operations related to these variables are not performed, so that “0” may be fixed.

  As described above, in addition to the method of inserting DCI or GOV before the VOP start code, if it is compatible with the syntax, for example, DCI may be inserted before the GOV start code or already exists “User_data” including the above information and the imposed image identifier can be added to the GOV. Here, MPEG-4 “User_data” has been described as an example. The above insertion processing can also be applied to H.264 and image encoded data having other similar mechanisms. In addition, for example, it is conceivable to embed the above information or the imposed image identifier using a start code of “reserved” (reserved area).

Imposed information insertion section _{22 1,} 22 2, ..., 22 _n image outputted from the stream _{vt 1,} vt 2, ..., vt _n is conference terminal by the image selecting section 23 _{2 1, 2} 2, ..., 2 Any one of _n is selected as a transmission destination, and image streams vm ₁ , vm ₂ ,..., vm _n are output. Imposed information insertion section _{24 1,} 24 2, ..., the 24 _n, at each timing acquired by the timing acquisition unit 26, the terminal state information _{si 1,} si 2, ..., corresponding to si _n or external status information ps The impose image identifier and the impose position information are inserted into each encoded image data of the image streams vm ₁ , vm ₂ ,..., Vm _n output from the image selection unit 23, and the image streams vu ₁ , vu ₂ ,. It is output as vu _n. Image stream _{vu 1,} vu 2, ..., vu _n is transmitting unit _{25 1,} 25 2, ..., a 25 _n, conference terminals _{2 1,} 2 2, ..., are respectively transmitted to the _{2 n.}

Further, the image processing unit 30 shown in FIG. 5 may be used instead of the image selection unit 23 in FIG. In the image processing unit 30, an image stream _{vt 1,} vt 2, ..., vt _n, the decoder _{31 1,} 31 2, ..., is decoded in the decoded image by 31 _n, a mixer _{32 1,} 32 2, ..., to 32 _n It is input and mixed. This mixing is performed by an arbitrary method. Image data output from the mixers 32 ₁ , 32 ₂ ,..., 32 _n are converted into image encoded data by the encoders 33 ₁ , 33 ₂ ,..., 33 _n , and the image streams vm ₁ , vm ₂ ,. Output as _n . The image processing unit 30 has a configuration in which the mixers 32 ₁ , 32 ₂ ,..., 32 _n are deleted, and mixing may not be performed.

In conference terminal 2 _1, the image stream vs ₁ transmitted from the multipoint conference apparatus 1 is received by the receiver 11. The impose information analyzing unit 12 outputs the encoded image data va included in the image stream vs ₁ received by the receiving unit 11, and the encoded image data va is analyzed to determine the impose image identifier cm and the impose image identifier cm. Pause position information cn is acquired. The encoded image data va is decoded by the decoder 13, and an image signal vd is output. The impose data selection unit 15 selects an impose image vi corresponding to the impose image identifier cm from the stored data set. In the image synthesizing unit 14, the impose image vi is synthesized with the image signal vd based on the impose position information cn, so that the image signal ve is superimposed on the image displayed on the display unit 16. Is output. The display unit 16 displays an image based on the image signal ve. Further, the conference terminals ₂ 2, ..., even _{2 n,} the same processing as conference terminals _{2 1} is performed.

As described above, in the present embodiment, the terminal state information is transmitted from the conference terminals 2 ₁ , 2 ₂ ,..., 2 _n to the multipoint conference apparatus 1, and the multipoint conference apparatus 1 performs the conference terminal. 2 _{1, 2} 2, ..., ₂ each of the image encoded data terminal for any each meeting received from _{n 2 1,} 2 2, ..., while being transmitted to the _{2 n,} the respective conference terminals ₂ 1, 2 _2, ..., superimpose image identifier and superimpose position information is inserted into each coded image data the image selection unit 23 through to the conference terminal 2 ₁ corresponding to each terminal status information received from the 2 _n , 2 ₂ ,..., 2 _n and the conference terminal 2 ₁ , 2 ₂ ,..., 2 _n selects the impose image corresponding to the impose image identifier given from the multipoint conference device 1. The selected impose image is the same multipoint conference device. Based on imposing position information provided from and superimposed on the image displayed on the display unit 16. For this reason, in the multipoint conference apparatus 1, it is not necessary to perform decoding, imposing, and encoding, and the processing load is expected to be reduced.

In addition, when the image processing unit 30 is used instead of the image selection unit 23 in the multipoint conference apparatus 1, it is desired to display the same image addressed to a plurality of conference terminals and to impose different images. In some cases, the number of encodings can be reduced, and the processing load can be reduced. For example, conference terminals 2 ₁ The conference terminal 2 ₂ images sent _from, ..., when displayed on the 2 _n, and conference terminals 2 _2, ..., if the image to be imposed by 2 _n are different, in the conventional technique and the present embodiment, the decoding count is equal to one for decoding the image of the conference terminal 2 _1, the number of imposed and encoding, in the conventional technique to a (n-1) times On the other hand, in this embodiment, it is once. In addition, by using the impose data of each user at the conference terminals 2 ₁ , 2 ₂ ,..., 2 _n without image imposition in the multipoint conference apparatus 1, an image suitable for each user's preference can be obtained. Imposed.

[Second Embodiment]
In the multipoint conference system according to the second embodiment preferably implementing the present invention, instead of the multipoint conference device 1 and the conference terminals 2 ₁ , 2 ₂ ,..., 2 _n in FIG. A multipoint conference apparatus 1A and conference terminals 2 ₁ A, 2 ₂ A,..., 2 _n A are provided.
Multipoint conference device 1A, conference terminals _{2 1 A, 2 2 A,} ..., 2 image stream _vs 1 including the image coded data with _{n A,} vs 2, ..., other to respectively transmit and receive vs _n , The impose image identifier, the impose position information, and the impose adaptation section (that is, the section from the frame where the impose starts to the end frame) separately from the image encoded data, the conference terminals 2 ₁ A, 2 ₂ A, ..., sent to 2 _n A. Conference terminal _{2 1 A, 2 2 A,} ..., the _{2 n} A, the image stream _{vs 1,} vs 2 transmitted from the multipoint conference device 1A, ..., (not shown) for displaying an image corresponding to vs _n It has a display part and receives an impose image identifier and impose position information.

FIG. 8 is a block diagram showing the electrical configuration of the conference terminal used in the multipoint conference system according to the present embodiment, and is common to the elements in FIG. 3 showing the first embodiment. The code | symbol is attached | subjected.
In this conference terminal 2 ₁ A, a receiving unit 11A and an impose information acquiring unit 18 having different functions are provided in place of the receiving unit 11, the impose information analyzing unit 12 and the image synthesizing unit 14 in FIG. Yes. The receiving unit 11A receives the image stream vs ₁ including the encoded image data transmitted from the multipoint conference apparatus 1A, the imposed image identifier, the imposed position information, and the imposed adaptation section. The impose information acquisition unit 18 acquires the impose image identifier cm and the impose position information cn received by the reception unit 11A. The decoder 13 decodes an image corresponding to image encoded data included in the image stream vs ₁ received by the receiving unit 11A, and outputs an image signal vd. The other configuration is the same as in FIG.

FIG. 9 is a block diagram showing an electrical configuration of the multipoint conference apparatus used in the multipoint conference system according to the present embodiment, and is common to the elements common to the elements in FIG. 4 showing the first embodiment. The code | symbol is attached | subjected.
In the multipoint conference apparatus 1A of this example, as shown in FIG. 9, the impose information insertion units 22 ₁ , 22 ₂ ,..., 22 _n and the impose information insertion units 24 ₁ , 24 ₂ ,. , 24 _n are deleted, an impose information generation unit 27 is newly provided, and a timing acquisition unit 26 _</ b _{> A} having a different function is provided instead of the timing acquisition unit 26. The timing acquisition unit 26A is based on the terminal state information si ₁ , si ₂ ,..., Si _n received by the receiving units 21 ₁ , 21 ₂ ,..., 21 _n and the external state information ps generated by the external device 3. Thus, each impose image identifier, each impose position information, and each timing for generating an impose adaptation section are acquired. Imposed information generating unit 27, at each timing acquired by the timing acquisition unit 26A, the terminal status information si _1, si 2, _..., superimpose image identifier corresponding to si _n, imposed position information and imposing adaptive Generate an interval. The other configuration is the same as that of FIG.

The communication method used in the multipoint conference system of this example is different from the first embodiment in the following points.
That is, the multi-point conference device 1A, conference terminals _{2 1 A, 2 2 A,} ..., the image stream _vs 1 including the image coded data between the _{2 n A, vs 2, ...} , vs n is transmitted and received, respectively is also superimpose an image identifier, it is imposed position information and imposing adapter section, the image stream vs _1, vs 2, _..., the terminal for separately the conference from the image coded data included in vs _n 2 ₁ a , 2 ₂ A,..., 2 _n A. In this case, based on a predetermined protocol, for example, RTCP (Real Time Protocol) Control Protocol (APP) (A Posteriori Probability) packet, DTMF (Dual Tone Multi Frequency) RTP packet described in RFC2833, XML (extensible mark up language) ) Data etc. are used.

In this case, in the conference terminals 2 ₁ A, 2 ₂ A,..., 2 _n A, the transmission unit 17 causes the image stream vs ₁ , vs ₂ , ..., vs _n and the terminal state information si ₁ , si ₂ ,. si _n is transmitted to the multipoint conference unit 1A. In multipoint conference device 1A, the receiving unit _{21 1,} 21 2, ..., a 21 _n, an image stream _{vs 1,} vs 2, ..., vs _n and terminal status information _{si 1,} si 2, ..., si _n is received The In the timing acquisition unit 26A, based on the terminal state information si ₁ , si ₂ ,..., Si _n received by the receiving units 21 ₁ , 21 ₂ ,..., 21 _n and the external state information ps generated by the external device 3. Thus, each impose image identifier, each impose position information, and each timing for generating each impose adaptation section are acquired. In imposing information generating unit 27, at each timing acquired by the timing acquisition unit 26A, the terminal status information si _1, si 2, _..., superimpose image identifier corresponding to si _n, imposed position information and imposing adaptive An interval is generated. These superimpose image identifier, superimpose position information and superimpose adapter section, the transmission unit _{25 1,} 25 2, ..., a 25 _n, conference terminals ₂ 1 _A, 2 2 A, ..., respectively to _{2 n} A Sent.

In the conference terminal 2 ₁ A, the reception unit 11A receives the image stream vs ₁ , the imposed image identifier, the imposed position information, and the imposed adaptation section transmitted from the multipoint conference device 1A. The impose information acquisition unit 18 acquires the impose image identifier cm, the impose position information cn, and the impose adaptation section cq received by the reception unit 11A. Further, the decoder 13 decodes an image corresponding to the image encoded data included in the image stream vs ₁ received by the receiving unit 11A, and outputs an image signal vd. In the image composition unit 14, the impose image vi is synthesized with the image signal vd in the impose adaptation section cq based on the impose position information cn, thereby superimposing in the image displayed on the display unit 16. The image signal ve is output after being paused. Thereafter, processing similar to that of the first embodiment is performed by the impose data selection unit 15 and the display unit 16.

As described above, in this embodiment, the point conference apparatus 1A does not need to perform decoding, impose, and encode as in the first embodiment, and the processing load can be reduced.
Further, when the image processing unit 30 is used instead of the image selection unit 23 in the multipoint conference apparatus 1A, there are the same advantages as in the first embodiment.

As described above, the preferred embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above-described embodiments, and there are design changes and the like without departing from the gist of the present invention. However, it is included in this invention.
For example, multipoint conference apparatus 1 shown in FIG. 4, superimpose information insertion section _{22 1,} 22 2, ..., 22 _n or imposed information insertion section _{24 1,} 24 2, ..., the configuration in which the deleted 24 _n It may be. The image processing unit 30 shown in FIG. 5, a mixer _{32 1,} 32 2, ..., is configured not exist 32 _n, the decoder _{31 1,} 31 2, ..., each decoded image data output from the 31 _n is The encoders 33 ₁ , 33 ₂ ,..., 33 _n may be input.
Further, the multipoint conference apparatus 1, 1A, an image stream _{vs 1,} vs 2, ..., conference terminals ₂ 1 based on _{vs n, 2 2, ...,} 2 n, 2 1 A, 2 2 A, ..., if function of determining the use state of the 2 _n a is provided, the conference terminal _{2 1, 2 2, ...,} 2 n, 2 1 a, 2 2 a, ..., the _{2 n} a, terminal status information si _{1, si} 2, ..., function of transmitting si _n becomes unnecessary.

  The present invention can be applied to all multipoint conference systems in which each communication terminal is distributed at multiple points, such as a video conference system, and is particularly effective when an image is imposed on the display screen of each communication terminal. is there.

  This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2007-176663 for which it applied on July 4, 2007, and takes in those the indications of all here.

It is a figure which shows the structure of the bidirectional | two-way communication system which concerns on this invention. It is a block diagram which shows the electric constitution of the principal part of the bidirectional | two-way communication system which concerns on 1st Embodiment which implemented this invention suitably. Is a block diagram showing an electrical configuration of the conference terminal 2 ₁ in FIG. It is a block diagram which shows the electrical structure of the multipoint conference apparatus 1 in FIG. FIG. 4 is a block diagram illustrating an electrical configuration of an image processing unit 30 provided in place of the image selection unit 23 in FIG. 3. It is a figure which shows the example of a data structure at the time of inserting the impose image identifier and the impose position information in each image coding data based on the moving image compression standard; Mpeg-4. It is a figure which shows the example of a data structure at the time of inserting the impose image identifier and the impose position information in each image coding data based on the moving image compression standard; Mpeg-4. It is a block diagram which shows the electrical structure of the terminal for a conference used for the multipoint conference system which concerns on 2nd Embodiment which implemented this invention suitably. It is a block diagram which shows the electric constitution of the multipoint conference apparatus used for the multipoint conference system which concerns on 2nd Embodiment.

Explanation of symbols

1,1A multipoint conference device (communication server)
1a CPU (Central Processing Unit)
1b ROM (read-only memory)
2 ₁ , 2 ₂ , ..., 2 _n , 2 ₁ A, 2 ₂ A, ..., 2 _n A Conference terminal (communication terminal)
2a CPU (Central Processing Unit)
2b ROM (read-only memory)
3 External device (external condition detection device)
11, 11A receiving unit 12 impose information analyzing unit 13 decoder (decoding unit)
14 Image composition section (image composition means)
15 Impose data selection section (impose image holding selection means)
16 Display section (display means)
17 Transmitter (transmission means)
18 impose information acquisition unit 21 ₁ , 21 ₂ ,..., 21 _n reception unit 22 ₁ , 22 ₂ ,..., 22 _n impose information insertion unit 23 image selection unit (part of communication server)
24 ₁ , 24 ₂ ,..., 24 _n impose information insertion unit 25 ₁ , 25 ₂ ,..., 25 _n transmission unit 26, 26 A Timing acquisition unit 27 Impose information generation unit 30 Image processing unit (part of communication server)
31 ₁ , 31 ₂ ,..., 31 _n decoder (part of the image processing unit)
32 ₁ , 32 ₂ ,..., 32 _n mixer (part of the image processing unit)
33 ₁ , 33 ₂ ,..., 33 _n encoder (part of the image processing unit)

Claims (20)

A plurality of communication terminals, and a communication server that transmits and receives each encoded image data to and from each of the communication terminals, wherein each of the communication terminals corresponds to the image encoded data transmitted from the communication server. A two-way communication system having display means for displaying
Each communication terminal is
One or a plurality of impose images for superimposing in the image displayed on the display means are held in a one-to-one correspondence with each impose image identifier, and the given impose image An impose image holding / selecting means for selecting the impose image corresponding to the identifier;
Image combining means for superimposing the image for imposition selected by the imposition image holding / selecting means on the image displayed on the display means on the basis of given imposition position information;
Transmitting means for transmitting the encoded image data to the communication server;
The communication server is
Each of the imposed image identifiers and each of the imposed position information is held, and each of the encoded image data received from each of the communication terminals is transmitted to any of the respective communication terminals. A two-way communication system, characterized in that the impose position information is transmitted to any of the communication terminals. The communication server is
The bidirectional communication system according to claim 1, wherein the impose image identifier and the impose position information are inserted into the encoded image data and transmitted to the communication terminals. The communication server is
2. The bidirectional communication system according to claim 1, wherein the impose image identifier and the impose position information are transmitted to each of the communication terminals separately from the respective encoded image data. The communication server is
A plurality of receiving units for receiving the encoded image data from the communication terminals;
A timing acquisition unit for acquiring each timing for inserting each impose image identifier and each impose position information into each encoded image data;
And a plurality of impose information insertion units for inserting the impose image identifier and the impose position information into the respective encoded image data at the respective timings acquired by the timing acquisition unit. 2. The bidirectional communication system according to 2. Each communication terminal is
A receiving unit that receives the encoded image data transmitted from the communication server;
An impose information analyzing unit that analyzes the encoded image data received by the receiving unit and acquires the impose image identifier and the impose position information;
The bidirectional communication system according to claim 4, further comprising: a decoding unit that decodes an image corresponding to the encoded image data. The communication server is
A plurality of receiving units for receiving the encoded image data from the communication terminals;
A timing acquisition unit that acquires each timing for generating each impose image identifier and each impose position information;
4. The bidirectional communication system according to claim 3, further comprising an impose information generating unit that generates the impose image identifier and the impose position information at each timing acquired by the timing acquisition unit. Each communication terminal is
A receiving unit that receives the encoded image data, the imposed image identifier, and the imposed position information transmitted from the communication server;
An impose information acquisition unit for acquiring the impose image identifier and impose position information received by the receiving unit;
The bidirectional communication system according to claim 6, further comprising: a decoding unit that decodes an image corresponding to the encoded image data. An external state detection device that detects an external state of the bidirectional communication system and generates one or more types of external state information is provided,
The communication server is
Each impose image identifier and each impose position information are stored in association with each external state information, and each image encoded data received from each communication terminal is transmitted to any communication terminal. In addition, the apparatus is configured to transmit the imposed image identifier and the imposed position information corresponding to the external state information received from the communication terminals to any of the communication terminals. Item 2. The bidirectional communication system according to item 1. The communication server is
2. The structure according to claim 1, wherein the impose image identifier and the impose position information are inserted into the encoded image data based on a predetermined moving image encoding method and a predetermined moving image compression standard. The bidirectional communication system according to 2, 4, or 5. The communication server is
8. The bidirectional communication system according to claim 3, 6 or 7, wherein the two-way communication system is configured to transmit the imposed image identifier and the imposed position information to each of the communication terminals based on a predetermined protocol. A communication terminal that transmits and receives each encoded image data to and from a communication server,
Display means for displaying an image corresponding to the encoded image data transmitted from the communication server;
One or a plurality of impose images for superimposing in the image displayed on the display means are held in a one-to-one correspondence with each impose image identifier, and the given impose image An impose image holding / selecting means for selecting the impose image corresponding to the identifier;
Image combining means for superimposing the image for imposition selected by the imposition image holding / selecting means on the image displayed on the display means on the basis of given imposition position information;
A communication terminal provided with transmission means for transmitting the encoded image data to the communication server. A communication server that transmits and receives each encoded image data to and from a plurality of communication terminals having display means for displaying an image corresponding to the encoded image data,
Holding each impose image identifier and each impose position information in one-to-one correspondence with one or a plurality of impose images for superimposing in the image displayed on the display means; And while transmitting each said image coding data received from each said communication terminal to each said each communication terminal, the structure which transmits the said impose image identifier and the position information to any said each communication terminal, A communication server.   13. The communication server according to claim 12, wherein the impose image identifier and the impose position information are inserted into the encoded image data and transmitted to the communication terminals.   13. The communication server according to claim 12, wherein the impose image identifier and the impose position information are transmitted to each of the communication terminals separately from the respective encoded image data. A plurality of receiving units for receiving the encoded image data from the communication terminals;
A timing acquisition unit for acquiring each timing for inserting each impose image identifier and each impose position information into each encoded image data;
And a plurality of impose information insertion units for inserting the impose image identifier and the impose position information into the respective encoded image data at the respective timings acquired by the timing acquisition unit. 13. The communication server according to 13. A plurality of receiving units for receiving the encoded image data from the communication terminals;
A timing acquisition unit that acquires each timing for generating each impose image identifier and each impose position information;
The communication server according to claim 14, further comprising: an impose information generation unit that generates the impose image identifier and the impose position information at each timing acquired by the timing acquisition unit. A communication method for transmitting and receiving encoded image data to and from a communication terminal,
When transmitting the encoded image data to the communication terminal, also transmit the impose image identifier and the impose position information,
Based on the transmitted impose image identifier, an image for imposition is selected,
A communication method comprising superimposing the imposition image in an image based on the transmitted imposition position information.   18. The communication method according to claim 17, wherein the impose image identifier and the impose position information are inserted into the respective encoded image data and transmitted to the respective communication terminals.   19. The communication method according to claim 18, wherein the impose image identifier and the impose position information are transmitted to the communication terminals separately from the encoded image data.   A communication control program for causing a computer to control the bidirectional communication system according to any one of claims 1 to 10.

Images