EP2732622A1 - Multipoint connection apparatus and communication system - Google Patents

Multipoint connection apparatus and communication system

Info

Publication number
EP2732622A1
EP2732622A1 EP12810881.8A EP12810881A EP2732622A1 EP 2732622 A1 EP2732622 A1 EP 2732622A1 EP 12810881 A EP12810881 A EP 12810881A EP 2732622 A1 EP2732622 A1 EP 2732622A1
Authority
EP
European Patent Office
Prior art keywords
video
audio
information
volume
combined
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP12810881.8A
Other languages
German (de)
French (fr)
Other versions
EP2732622A4 (en
EP2732622B1 (en
EP2732622B8 (en
Inventor
Akihito AIBA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ricoh Co Ltd
Original Assignee
Ricoh Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ricoh Co Ltd filed Critical Ricoh Co Ltd
Publication of EP2732622A1 publication Critical patent/EP2732622A1/en
Publication of EP2732622A4 publication Critical patent/EP2732622A4/en
Publication of EP2732622B1 publication Critical patent/EP2732622B1/en
Application granted granted Critical
Publication of EP2732622B8 publication Critical patent/EP2732622B8/en
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems
    • H04N7/152—Multipoint control units therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00—Data switching networks
    • H04L12/02—Details
    • H04L12/16—Arrangements for providing special services to substations
    • H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
    • H04L12/1813—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast for computer conferences, e.g. chat rooms
    • H04L12/1827—Network arrangements for conference optimisation or adaptation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M3/00—Automatic or semi-automatic exchanges
    • H04M3/42—Systems providing special services or facilities to subscribers
    • H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
    • H04M3/567—Multimedia conference systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M2201/00—Electronic components, circuits, software, systems or apparatus used in telephone systems
    • H04M2201/38—Displays
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M2201/00—Electronic components, circuits, software, systems or apparatus used in telephone systems
    • H04M2201/50—Telephonic communication in combination with video communication
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04M—TELEPHONIC COMMUNICATION
    • H04M3/00—Automatic or semi-automatic exchanges
    • H04M3/42—Systems providing special services or facilities to subscribers
    • H04M3/56—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities
    • H04M3/568—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants
    • H04M3/569—Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants using the instant speaker's algorithm

Definitions

  • the present invention relates to a multipoint
  • connection apparatus and a communication system.
  • Video conference systems for exchanging video, audio, and the like with another party at a remote site have become proliferated in recent years along with development of networks and the like.
  • Video conference systems have a problem that a speaker has feelings of anxiety during speech because it is
  • Solutions to this problem include a method of displaying a . level meter or the like indicating volume of the speaker's voice on a screen.
  • a local terminal which is a terminal at a local site, receives information about speaker volume (a volume level of output audio) of a remote terminal transmitted from the remote terminal installed at a remote site, displays the information on a screen, and also displays microphone volume (a volume level of input audio) and speaker volume of the local terminal on the screen.
  • H7-226800 is a configuration in which speaker-volume
  • Patent Application Laid-open No. 2004-186870 and Japanese Patent Application Laid-open No. H7-226800 have a problem that when the remote terminal is different in configuration from the local terminal, the speaker-volume information about the remote terminal cannot be acquired from the remote side. Put another way, there is a problem that they are applicable only when the remote terminal and the local terminal are similar in configuration.
  • the multipoint connection apparatus connected to a plurality of video/audio terminals.
  • the multipoint connection apparatus includes a video/audio-signal receiving unit configured to receive video/audio signals output from the video/audio terminals, respectively; a volume-level calculating unit configured to calculate volume levels from the respective video/audio signals; a volume-display-image generating unit configured to generate volume display images indicating volume based on the respective volume levels calculated by the volume-level calculating unit; a layout-setting- information receiving unit configured to receive pieces of layout setting information, each indicating information about arrangement of videos to be displayed on the
  • a combined-video/audio- signal generating unit configured to generate a combined video/audio signal for each of the video/audio terminals by combining the video/audio signals and the volume display images based on the piece of layout setting information for the corresponding video/audio terminal; and a transmitting unit configured to transmit the combined video/audio signal to the corresponding video/audio terminal.
  • a multipoint connection apparatus connected to a plurality of video/audio terminals.
  • the multipoint connection apparatus includes a video/audio-signal receiving unit configured to receive video/audio signals output from the video/audio terminals, respectively; a volume-level calculating unit configured to calculate volume levels from the respective video/audio signals; a layout-setting-information receiving unit configured to receive pieces of layout setting
  • a combined-information-signal generating unit configured to generate a combined information signal for each of the video/audio terminals from the video audio signals, the volume levels, and the piece of layout setting information for the corresponding video/audio terminal; and a transmitting unit configured to transmit the combined information signal to the corresponding video/audio
  • a communication system that includes a multipoint connection apparatus; and a plurality of video/audio terminals, the multipoint connection apparatus and the video/audio terminals being connected to each other via a communication network.
  • the multipoint connection apparatus includes a video/audio-signal receiving unit configured to receive video/audio signals output from the video/audio terminals, respectively; a volume-level calculating unit configured to calculate volume levels from the respective video/audio signals; a layout-setting-information receiving unit configured to receive pieces of layout setting
  • a combined-information-signal generating unit configured to generate a combined information signal for at least one of the video/audio terminals from the video audio signals, the volume levels, and the piece of layout setting information for the corresponding video/audio terminal; and a transmitting unit configured to transmit the combined information signal to the corresponding video/audio
  • the at least one of the video/audio terminals includes a combined-information-signal receiving unit configured to receive the combined information signal; a volume-display-image generating unit configured to generate volume display images indicating volume from the volume levels and the piece of layout setting information
  • video/audio combining unit configured to combine the video/audio signals contained in the combined information signal with the volume display images.
  • Fig. 1 is an overall configuration diagram of a communication system to which the present invention is applied;
  • Fig. 2 is a block diagram illustrating the
  • Fig. 3 is a block diagram illustrating an internal configuration of a decoding unit according to the first embodiment
  • Fig. 4 is a diagram illustrating a method for
  • Figs. 5A and 5B are diagrams of volume display images generated by a volume-display-image generating unit
  • Figs. 6A and 6B are diagrams of video layouts
  • Figs. 7A and 7B are diagrams of videos combined by a combined-video/audio-signal generating unit according to the first embodiment
  • Figs. 8A and 8B are sequence chart diagrams
  • Fig. 9 is a block diagram illustrating the
  • Figs. 10A and 10B are diagrams of a data structure of additional information according to the second embodiment
  • Figs. 11A and 11B are diagrams of images combined by a combined-video/audio-signal generating unit according to the second embodiment
  • Fig. 12 is a sequence chart diagram illustrating operations of the communication system according to the second embodiment
  • Fig. 13 is a block diagram illustrating the configuration of a communication system according to a third embodiment.
  • Fig. 14 is a sequence chart diagram illustrating operations . of the communication system according to the third embodiment.
  • FIG. 1 is an overall configuration diagram of a communication system to which the present invention is applied.
  • Fig. 2 is a block diagram illustrating the configuration of a communication system according to the first embodiment.
  • the present embodiment is characterized in that a multipoint connection apparatus includes a volume-level calculating unit upstream of a combined-video/audio-signal generating unit.
  • a communication system 1 includes a plurality of video/audio terminals 100 ⁇ to 100 n and a multipoint
  • connection apparatus 200 that relays video/audio signals and various types of information to be transmitted and received by the video/audio terminals 100i to 100 n .
  • the video/audio terminals 100i to 100 n and the multipoint connection apparatus 200 are connected to each other via a communication network.
  • any one of the video/audio terminals 100i to 100 n and the multipoint connection apparatus 200 are connected to each other via a communication network.
  • a layout- setting-information transmitting unit 300 is connected to the multipoint connection apparatus 200 via the
  • the layout-setting-information transmitting unit 300 is a unit that transmits layout setting information which is information about video layouts and information to be displayed on display units of the video/audio terminals 100 to the multipoint connection apparatus 200.
  • n is any number equal to or
  • each of the video/audio terminals 100 includes a camera (capturing unit) that captures video of each point, a microphone (audio input unit) that
  • FIG. 2 illustrates a case where the video/audio terminals 100i to 100 n _i serve as sender terminals that transmit video and audio, and the video/audio terminal 100 n serves as a receiver terminal that receives the video and audio transmitted from the video/audio terminals 100i to 100 ⁇ - ⁇
  • the multipoint connection apparatus 200 includes a video/audio-signal receiving unit 201, a decoding unit 203, a volume-level calculating unit 205, a volume-display-image generating unit 207, a layout-setting-information receiving unit 209, a combined-video/audio-signal generating unit 211, an encoding unit 213, and a transmitting unit 215.
  • the video/audio-signal receiving unit 201 receives video/audio signals transmitted from the video/audio
  • the video/audio signals in a compressed state are generally transmitted from the video/audio terminals 100.
  • the decoding unit 203 decodes each of the compressed video/audio signals received by the video/audio-signal receiving unit 201.
  • An example of the decoding unit 203 is explained below.
  • Fig. 3 is a block diagram illustrating an internal configuration of a decoding unit according to the first embodiment.
  • the decoding unit 203 may have such a configuration as illustrated in Fig. 3 so as to be capable of supporting a plurality of encoding formats.
  • the decoding unit 203 includes a determining unit 221, a decoder selecting unit 223, and a single-format decoder group 227 that includes a plurality of single-format
  • the determining unit 221 determines the encoding format of the compressed
  • the decoder selecting unit 223 outputs the compressed
  • the selected single-format decoder decodes the compressed video/audio signal input thereto and outputs a video/audio signal.
  • the volume-level calculating unit 205 (Fig. 2)
  • volume levels each respectively from one of the video/audio signals decoded by the decoding unit 203 and outputs the volume levels as volume-level information.
  • Fig. 4 is a diagram illustrating a method for calculating a volume level from a video/audio signal according to the first embodiment.
  • a volume level vol max can be calculated using the maximum amplitude using the following equation.
  • a volume level vol rms can be calculated using the RMS using the followin equation.
  • the volume level is not necessarily actual maximum amplitude.
  • a method of outputting a volume level that is normalized and quantized to an integer in a predetermined range for example, from 0 to 10 in decibel value in such a manner that 0 corresponds to a silent state and 10 corresponds to a 0-dB state.
  • normalization means transforming data according to a certain rule to make the data more usable.
  • the volume-display-image generating unit 207 (Fig. 2) generates volume display images based on the volume-level information output from the volume-level calculating unit 205.
  • Figs. 5A and 5B are diagrams of volume display images generated by a volume-display-image generating unit
  • a volume display image 231 is formed from a volume display area 231a which is a
  • volume display area 231a is displayed in a predetermined base color.
  • the bar 231b contacts the volume display area 231a on at least three sides and is displayed in a color different from the base color of the volume display area 231a.
  • Volume is expressed as an area of the bar 231b in the volume display area 231a. More specifically, volume is expressed by assigning the different color only to a portion, of which area depends on the volume level, of the volume display area 231a. For example, as illustrated in (a-2) , the area of the bar 231b is zero at minimum volume, and only the volume display area 231a is displayed. As illustrated in (a-3) , the area of the bar 231b is maximized at maximum volume, and the entire volume display area 231a becomes the bar 231b.
  • a volume display image 233 is formed from a volume display area 235 (segment bar graph) , which is formed from a plurality of segment bars 235a arranged in one vertical line, and colored segment bars 237 that represent volume.
  • segment bar graph segment bar graph
  • the segment bars 237 are some of the segment bars 235a in the volume display area 235 filled with a color different from the base color.
  • volume is expressed as the number of the segment bars 237. More specifically, volume is expressed by assigning the different color only to the segment bars 235a, of which number depends on the volume level, of the volume display area 235. For example, as illustrated in (b-2), the number of the segment bars 237 is zero at minimum volume. As illustrated in (b-3) , all of the segment bars 235a in the volume display area 235 become the segment bars 237 at maximum volume.
  • any other image than those of the examples given in Figs. 5A and 5B can be employed so long as the image allow visual recognition of volume.
  • another shape such as an elliptical shape, a circular shape, or a triangular shape, than the rectangular shape can be employed.
  • the layout-setting-information receiving unit 209 (Fig.
  • the layout setting information is information indicating information about arrangement of videos. More specifically, the layout setting information is information indicating in what layout, or arrangement, the videos of the video/audio terminals lOOi to 100 n are to be displayed. Meanwhile, the layout-setting-information transmitting unit 300 is generally incorporated in the video/audio terminal 100 n and configured such that layout setting can be
  • Figs. 6A and 6B are diagrams of video layouts
  • Examples where videos at two points are displayed on one screen are illustrated. More specifically, the examples are of a case where a video at a local point and a video at a remote point in a
  • a display screen 241 illustrated in Fig. 6A is an example of a layout of an area 241A dominating the entire screen and an area 241B that is displayed as being overlaid on the area 241A and smaller than the area 241A.
  • a video at a certain point is displayed on the area 241A, while a video at another point is displayed on the area 241B.
  • the area 241B is displayed as being overlaid on a bottom right portion of the area 241A; however, a position where the area 241B is to be displayed is not limited thereto.
  • a similar layout in which, for example, the area 241B is arranged at bottom left, top left, or top right can be employed.
  • a display screen 243 illustrated in Fig. 6B is an example layout in which a screen is vertically divided into two parts with an area 243C on the left and an area 243D on the right. A video at a certain point is displayed on the area 243C, while a video at another point is displayed on the area 243D. Examples of a similar layout include that in which a top portion and a bottom portion of each of the areas 243C and 243D are blanked to maintain an aspect ratio of videos to be displayed.
  • the combined-video/audio-signal generating unit 211 (Fig. 2) combines the video/audio signals decoded by the decoding unit 203 with the volume display images generated by the volume-display-image generating unit 207 based on the layout setting information received by the layout- setting-information receiving unit 209.
  • the combined video/audio signal is a signal into which at least
  • the combined video/audio signal according to the present embodiment is obtained by
  • FIGs. 7A and 7B are diagrams of videos combined by a combined-video/audio-signal generating unit according to the first embodiment.
  • the combined video illustrated in Fig. 7A is an example in which layout setting information corresponding to the layout illustrated in Fig. 6B is received and such a volume display image as illustrated in Fig. 5A is used, for example.
  • the volume display images 231 are displayed in lower portions of the area 243C and the area 243D on the display screen 243.
  • Fig. 7B is an example in which layout setting information
  • volume display images 233 are displayed in a left end portion of the area 241A and a right end portion of the area 241B on the display screen 243.
  • the encoding unit 213 (Fig. 2) encodes the combined video/audio signal output from the combined-video/audio- signal generating unit 211 to compress data size.
  • the transmitting unit 215 transmits the combined video/audio signal compressed by the encoding unit 213 to the video/audio terminal 100 n .
  • the video/audio terminal 100 n outputs video that is combined based on the received combined video/audio signal as illustrated in Fig. 7A or 7B to a display unit such as a display or a projector.
  • a display unit such as a display or a projector.
  • n signals are input/output to/from the layout- setting-information receiving unit 209, the combined- video/audio-signal generating unit 211, the encoding unit 213, and the transmitting unit 215.
  • FIGs. 8A and 8B are sequence chart diagrams illustrating operations of the communication system according to the first embodiment.
  • the layout-setting-information transmitting unit 300 transmits layout setting information to the multipoint connection apparatus 200 (Step SI) .
  • the layout-setting- information receiving unit 209 of the multipoint connection apparatus 200 receives the layout setting information (Step S2) and stores it in a storage unit (not shown) (Step S3) .
  • the video/audio terminals lOOi to 100 n transmit compressed video/audio signals to the multipoint connection apparatus 200 (Steps Sll and S12).
  • the video/audio-signal receiving unit 201 of the multipoint connection apparatus 200 receives the n
  • the decoding unit 203 decodes the compressed video/audio signals into the n compressed video/audio signals (Step S14).
  • the volume- level calculating unit 205 calculates n volume levels from the n video/audio signals decoded by the decoding unit 203 (Step S15) .
  • the volume-display-image generating unit 207 generates n volume display images from information about the n volume levels output from the volume-level
  • the combined-video/audio- signal generating unit 211 retrieves the layout setting information stored at Step S3 of Fig. 8A and combines the n video/audio signals with the n volume display images (Step S17).
  • the encoding unit 213 encodes the combined
  • the transmitting unit 215 transmits the compressed combined video/audio signal generated by the encoding unit 213 to the video/audio terminal 100 n (Step S19) .
  • the video/audio terminal 100 n receives the compressed combined video/audio signal (Step S20), decodes it (Step S21) , and displays the obtained video/audio signals on the display unit (Step S22).
  • the speaker volume of the remote terminal and the microphone volume of the local terminal are information about audio transmitted from the local site to the remote site. Not only such information as described above but also information, such as speaker volume of the local terminal and microphone volume of the remote terminal, about audio transmitted from the remote site to the local site are preferably displayed simultaneously. This permits performing adjustment of speech volume level, which is performed while viewing a volume display related to audio on the local side, based on relationship between how audio provided by a speaker is audible on the local side and volume display related to audio on the remote side, thereby facilitating the adjustment. Furthermore, it is also advantageous that at occurrence of a trouble, the
  • the multipoint connection apparatus includes the volume-level calculating unit and the volume-display- image generating unit, and transmits a combined video/audio signal into which the volume display images are combined by the combined-video/audio-signal generating unit to the video/audio terminal.
  • the video/audio terminal can receive the combined video/audio signal in which the volume display images are already contained. Therefore, the video/audio terminal which is a receiving side can display microphone volume of the other video/audio terminals individually independent of the configuration of the video/audio
  • volume levels appropriate values as the volume levels. Furthermore, utilizing the RMS in calculation of the volume levels permits employing easy-to-handle values as the volume levels. Furthermore, utilizing the volume levels that are normalized and quantized permits displaying volume in an intuitive and understandable manner.
  • the video/audio terminals 100i to 100 n -i also receive
  • FIG. 9 is a block diagram illustrating the configuration of a communication system according to the second embodiment.
  • the present embodiment is characterized in that video/audio terminals perform generating volume display images and combining the generated volume display images with video/audio signals.
  • n is any number equal to or greater than two
  • Fig. 9 illustrates a case where the video/audio terminal 110 n receives video/audio signals from the video/audio terminals 110i to llOn-i- Note that terminals having the same
  • the multipoint connection apparatus 250 includes the video/audio-signal receiving unit 201, the decoding unit 203, the volume-level calculating unit 205, a combined- information-signal generating unit 251, an encoding unit 253, the layout-setting-information receiving unit 209, and a transmitting unit 255.
  • calculating unit 205 and the layout-setting-information receiving unit 209 are similar to corresponding units of the multipoint connection apparatus 200 of the first embodiment and designated by same reference numerals as those of the first embodiment; accordingly, description thereof is omitted.
  • the combined-information-signal generating unit 251 combines the video/audio signals decoded by the decoding unit 203 based on the layout setting information received by the layout-setting-information receiving unit 209.
  • the combined-information-signal generating unit 251 generates a combined information signal by
  • volume-level information which is additional information
  • layout setting information additional information
  • a video/audio terminal e.g., the video/audio terminal 110
  • a volume- display-image generating unit 115 e.g., the volume- display-image generating unit 115
  • combining unit 117 which will be described later, to acquire and utilize video/audio signals, and also prevents an increase in data size.
  • Figs. 10A and 10B are diagrams of a data structure of additional information according to the second embodiment.
  • the volume-level information and the layout setting information have a data structure in which the volume-level information about the video/audio terminals is associated with the corresponding layout information.
  • Layout positional information about each of the video/audio terminals can be expressed in, for example, XY coordinate values of a video display position.
  • Fig. 10B illustrates an example where actual values are assigned. Values corresponding to Fig. 7A are assigned in this example.
  • Coordinates (160, 560) and (800, 560) of a top left point and coordinates (320, 620) and (960, 620) of a bottom right point of the volume display images 231 are assigned for terminal numbers 1 and 2, respectively, as the layout setting information.
  • Negative values in layout setting for a terminal number 3 indicate that video of the video/audio terminal to which this number is assigned is not to be displayed.
  • the encoding unit 253 (Fig. 9) encodes the combined information signal output from the combined-information- signal generating unit 251 to compress data size.
  • the transmitting unit 255 transmits the combined information signal compressed by the encoding unit 253 to the video/audio terminal 110 n -
  • the video/audio terminal 110 n includes a combined- information-signal receiving unit 111, a decoding unit 113, the volume-display-image generating unit 115, and the video/audio combining unit 117.
  • the combined-information-signal receiving unit 111 receives the compressed combined information signal
  • the decoding unit 113 decodes the compressed combined information signal.
  • the volume-display-image generating unit 115 generates a display screen including volume display images indicating volume based on the volume-level information and the layout setting information embedded in the combined information signal.
  • Figs. 11A and 11B are diagrams of images combined by a combined-video/audio-signal generating unit according to the second embodiment.
  • the combined video illustrated in Fig. 11A is an example of an image generated with such a volume display method as illustrated in Fig. 5A and from the layout setting information corresponding to Fig. 6B, for example.
  • the volume display images 231 that indicate volume of the video/audio terminals, to which the terminal numbers 1 and 2 are assigned, are individually displayed in a bottom. portion of a display screen 121.
  • the combined video illustrated in Fig. 11B is an example of an image generated with such a volume display method as illustrated in Fig. 5B and from the layout setting information corresponding to Fig. 6A, for example.
  • the volume display images 233 that indicate volume of the video/audio terminals, to which the terminal numbers 1 and 2 are assigned, are displayed in a lower portion of a left end portion and a lower portion of a right end portion, respectively, of a display screen 123.
  • any video other than the examples illustrated in Figs. 11A and 11B may be obtained by combining so long as the volume levels and the display images, and the layout setting information and the display position correspond to each other.
  • the video/audio combining unit 117 (Fig. 9) combines the combined information signal decoded by the decoding unit 113 with the volume display images generated by the volume-display-image generating unit 115. More
  • the video/audio combining unit 117 assumes the combined information signal that contains the volume- level information as the video/audio signals and utilizes the combined information signal as it is without performing an operation of separating only the video/audio signals from the combined information signal. For example, when the combined information signal that is assumed as the video/audio signals is combined with such volume display images as illustrated in Fig. 11A, such a video/audio signal as illustrated in Fig. 7A is obtained.
  • the video/audio terminal 110-n outputs such video as illustrated in Figs. 7A and 7B based on the combined video/audio signal to a display unit such as a display or a projector.
  • Fig. 12 is a sequence chart diagram illustrating the operations of the communication system according to the second embodiment. This diagram illustrates only
  • Operations related to video/audio processing are similar to those of the first embodiment (see Fig. 8A) and description thereof is omitted. Processes similar to those of the first embodiment are designated by like step numbers and description thereof is omitted.
  • the video/audio terminals 110i to 110 n transmit compressed video/audio signals to the multipoint connection apparatus 250 (Steps Sll and S12) .
  • the multipoint connection apparatus 250 performs processing from Step S13 to Step S15.
  • the combined-information-signal generating unit 251 of the multipoint connection apparatus 250 retrieves the layout setting information stored at Step S3 of Fig. 8A and combines the n video/audio signals based on the layout setting information (Step S31) . Furthermore, the combined- information-signal generating unit 251 generates a combined information signal by embedding additional information containing the layout setting information and the volume- level information into the video/audio signal obtained by combining (Step S32).
  • the combined-information-signal receiving unit 111 of the video/audio terminal 110 n receives the compressed combined information signal (Step S20')-
  • the decoding unit 113 decodes the compressed combined information signal (Step S21').
  • the volume-display-image generating unit 115 generates volume display images based on the volume-level information and the layout setting information embedded in the decoded combined information signal (Step S33) .
  • the video/audio combining unit 117 combines the combined information signal with the volume display images (Step S34). The obtained video/audio is displayed on the display unit (Step S22 ' ) .
  • generating unit of the multipoint connection apparatus generates a combined information signal by embedding additional information containing layout setting
  • each of the video/audio terminals obtain a similar video irrespective of the configuration of the video/audio terminal; however, volume display can disadvantageously become redundant in such a case where the video/audio terminal includes a volume display unit of its own or the like.
  • the video/audio terminal includes the volume-display-image generating unit and the video/audio combining unit.
  • FIG. 13 is a block diagram illustrating the configuration of a
  • the combined-information-signal generating unit 251 of a multipoint connection apparatus 260 of a communication system 3 includes two units which are an additional- information generating unit 261 and a combined-video/audio- signal generating unit 263.
  • the additional-information generating unit 261 generates additional information having such a data structure as illustrated in Figs. 10A and 10B from volume levels and layout setting information.
  • the combined-video/audio-signal generating unit 263 outputs a combined video/audio signal into which video/audio signals of video/audio terminals 130 are combined based on the layout setting information. Meanwhile, the additional information and the combined video/audio signal output from the combined-information-signal generating unit 251 according to the present invention are collectively
  • a combined information signal referred to as a combined information signal.
  • the encoding unit 253 includes an additional- information encoding unit 265 and a combined-video/audio- signal encoding unit 267.
  • the additional-information encoding unit 265 encodes the additional information
  • the combined-video/audio-signal encoding unit 267 encodes the combined video/audio signal to compress data size.
  • the transmitting unit 255 includes an additional- information transmitting unit 269 and a combined- video/audio-signal transmitting unit 271.
  • the additional- information transmitting unit 269 transmits the compressed additional information.
  • the combined-video/audio-signal transmitting unit 271 transmits the compressed combined video/audio signal.
  • the combined-information-signal receiving unit 111 of the video/audio terminal 130 n includes an additional- information receiving unit 131 and a combined-video/audio- signal receiving unit 133.
  • the additional-information receiving unit 131 receives the compressed additional information.
  • the combined-vided/audio-signal receiving unit 133 receives the compressed combined video/audio signal .
  • the decoding unit 113 includes an additional- information decoding unit 135 and a combined-video/audio- signal decoding unit 137.
  • the additional-information decoding unit 135 decodes the compressed additional
  • the combined-video/audio-signal decoding unit 137 decodes the compressed combined video/audio signal.
  • Fig. 14 is a sequence chart diagram illustrating the operations of the communication system according to the third embodiment. This diagram illustrates only operations related to video/audio processing. Operations related to layout setting information processing are similar to those of the first embodiment (see Fig. 8A) and description thereof is omitted. Processing similar to that of the first and second embodiments is designated by a like step number and description thereof is omitted.
  • the video/audio terminals 130i to 130 n transmit compressed video/audio signals to the multipoint connection apparatus 260 (Steps Sll and S12) .
  • the multipoint connection apparatus 260 performs processing from Step S13 to Step S15.
  • Video audio signals are processed through a first channel of the multipoint connection apparatus 260.
  • the combined-video/audio-signal generating unit 263 retrieves the layout setting information stored at Step S3 of Fig. 8A and combines the n video/audio signals based on the layout setting information (Step S41).
  • the combined-video/audio- signal encoding unit 267 encodes the combined video/audio signal combined by the combined-video/audio-signal
  • the combined-video/audio- signal transmitting unit 271 transmits the compressed combined video/audio signal encoded by the combined- video/audio-signal encoding unit 267 to the video/audio terminal 130 n (Step S43) .
  • additional information is processed through a second channel of the multipoint connection apparatus 260.
  • the additional-information generating unit 261 retrieves the layout setting information stored at Step S3 of Fig. 8A and generates additional information
  • the additional-information encoding unit 265 encodes the additional information generated by the additional-information generating unit 261 (Step S45) .
  • the additional-information transmitting unit 269 transmits the compressed additional information encoded by the additional-information encoding unit 265 to the video/audio terminal 130 n (Step S46) .
  • the video/audio signal is processed through a first channel of the video/audio terminal 130 n .
  • the combined- video/audio-signal receiving unit 133 receives the
  • Step S47 The combined-video/audio-signal decoding unit 137 decodes the compressed combined video/audio signal (Step S48).
  • the additional information is processed through a second channel of the video/audio terminal 130 n .
  • the additional-information receiving unit 131 receives the compressed additional information (Step S49) .
  • the additional-information decoding unit 135 decodes the compressed additional information (Step S50).
  • the volume-display-image generating unit 115 generates volume display images based on the volume-level information and the layout setting information contained in the decoded additional information (Step S33').
  • the video/audio combining unit 117 combines the combined video/audio signal decoded by the combined- video/audio-sighal decoding unit 137 with the volume display images generated by the volume-display-image generating unit 115 (Step S34 ' ) .
  • the obtained video/audio signals are displayed on the display unit (Step S22 ' ' ) .
  • the additional information and the video/audio signals are transmitted through the different channels.
  • volume display can disadvantageously become redundant in such a case where the video/audio terminal includes a volume display unit of its own or the like.
  • the video/audio terminal includes the volume-display-image generating unit and the
  • volume levels are 0
  • volume display images are generated from the volume levels, and a signal into which the volume display images and the video/audio signals transmitted from the video/audio terminals are combined is generated. Accordingly, even when a multipoint conference is held by connecting three or more remote sites, each of the video/audio terminals can display microphone volume of the points separately.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • Telephonic Communication Services (AREA)

Abstract

A multipoint connection apparatus (200) includes a video/audio-signal receiving unit (201) that receives video/audio signals from video/audio terminals (100); a volume-level calculating unit (205) that calculates volume levels from the video/audio signals; a volume-display-image generating unit (207) that generates volume display images indicating volume from the volume levels; a layout-setting-information receiving unit (209) that receives layout setting information indicating information about arrangement of videos to be displayed on the video/audio terminal (100); a combined-video/audio-signal generating unit (211) that generates a combined video/audio signal by combining the video/audio signals and the volume display images based on the layout setting information; and a transmitting unit (215) that transmits the video/audio signal to the video/audio terminal (100).

Description

DESCRIPTION
MULTIPOINT CONNECTION APPARATUS AND COMMUNICATION SYSTEM TECHNICAL FIELD
The present invention relates to a multipoint
connection apparatus and a communication system.
BACKGROUND ART
Video conference systems for exchanging video, audio, and the like with another party at a remote site have become proliferated in recent years along with development of networks and the like.
Video conference systems have a problem that a speaker has feelings of anxiety during speech because it is
difficult to determine whether the speaker's voice is carried to the other party at appropriate volume.
Solutions to this problem include a method of displaying a . level meter or the like indicating volume of the speaker's voice on a screen.
For example, disclosed in Japanese Patent Application Laid-open No. 2004-186870 is a configuration in which a local terminal, which is a terminal at a local site, receives information about speaker volume (a volume level of output audio) of a remote terminal transmitted from the remote terminal installed at a remote site, displays the information on a screen, and also displays microphone volume (a volume level of input audio) and speaker volume of the local terminal on the screen.
Disclosed in Japanese Patent Application Laid-open No.
H7-226800 is a configuration in which speaker-volume
setting information about a remote terminal is transmitted to a local terminal. However, the configurations disclosed in Japanese
Patent Application Laid-open No. 2004-186870 and Japanese Patent Application Laid-open No. H7-226800 have a problem that when the remote terminal is different in configuration from the local terminal, the speaker-volume information about the remote terminal cannot be acquired from the remote side. Put another way, there is a problem that they are applicable only when the remote terminal and the local terminal are similar in configuration.
Therefore, there is a need for a multipoint connection apparatus that allows each of video/audio terminals to display microphone volume of the points separately, and a communication system.
DISCLOSURE OF INVENTION
It is an object of the present invention to at least partially solve the problems in the conventional technology.
According to an embodiment, there is provided a
multipoint connection apparatus connected to a plurality of video/audio terminals. The multipoint connection apparatus includes a video/audio-signal receiving unit configured to receive video/audio signals output from the video/audio terminals, respectively; a volume-level calculating unit configured to calculate volume levels from the respective video/audio signals; a volume-display-image generating unit configured to generate volume display images indicating volume based on the respective volume levels calculated by the volume-level calculating unit; a layout-setting- information receiving unit configured to receive pieces of layout setting information, each indicating information about arrangement of videos to be displayed on the
corresponding video/audio terminal; a combined-video/audio- signal generating unit configured to generate a combined video/audio signal for each of the video/audio terminals by combining the video/audio signals and the volume display images based on the piece of layout setting information for the corresponding video/audio terminal; and a transmitting unit configured to transmit the combined video/audio signal to the corresponding video/audio terminal.
According to another embodiment, there is provided a multipoint connection apparatus connected to a plurality of video/audio terminals. The multipoint connection apparatus includes a video/audio-signal receiving unit configured to receive video/audio signals output from the video/audio terminals, respectively; a volume-level calculating unit configured to calculate volume levels from the respective video/audio signals; a layout-setting-information receiving unit configured to receive pieces of layout setting
information, each indicating information about arrangement of videos to be displayed on the corresponding video/audio terminal; a combined-information-signal generating unit configured to generate a combined information signal for each of the video/audio terminals from the video audio signals, the volume levels, and the piece of layout setting information for the corresponding video/audio terminal; and a transmitting unit configured to transmit the combined information signal to the corresponding video/audio
terminal .
According to still another embodiment, there is provided a communication system that includes a multipoint connection apparatus; and a plurality of video/audio terminals, the multipoint connection apparatus and the video/audio terminals being connected to each other via a communication network. The multipoint connection apparatus includes a video/audio-signal receiving unit configured to receive video/audio signals output from the video/audio terminals, respectively; a volume-level calculating unit configured to calculate volume levels from the respective video/audio signals; a layout-setting-information receiving unit configured to receive pieces of layout setting
information, each indicating information about arrangement of videos to be displayed on the corresponding video/audio terminal; a combined-information-signal generating unit configured to generate a combined information signal for at least one of the video/audio terminals from the video audio signals, the volume levels, and the piece of layout setting information for the corresponding video/audio terminal; and a transmitting unit configured to transmit the combined information signal to the corresponding video/audio
terminal. The at least one of the video/audio terminals includes a combined-information-signal receiving unit configured to receive the combined information signal; a volume-display-image generating unit configured to generate volume display images indicating volume from the volume levels and the piece of layout setting information
contained in the combined information signal; and a
video/audio combining unit configured to combine the video/audio signals contained in the combined information signal with the volume display images.
The above and other objects, features, advantages and technical and industrial significance of this invention will be better understood by reading the following detailed description of presently preferred embodiments of the invention, when considered in connection with the
accompanying drawings .
BRIEF DESCRIPTION OF DRAWINGS
Fig. 1 is an overall configuration diagram of a communication system to which the present invention is applied;
Fig. 2 is a block diagram illustrating the
configuration of a communication system according to a first embodiment;
Fig. 3 is a block diagram illustrating an internal configuration of a decoding unit according to the first embodiment;
Fig. 4 is a diagram illustrating a method for
calculating a volume level from a video/audio signal according to the first embodiment ;
Figs. 5A and 5B are diagrams of volume display images generated by a volume-display-image generating unit
according to the first embodiment;
Figs. 6A and 6B are diagrams of video layouts
according to the first embodiment;
Figs. 7A and 7B are diagrams of videos combined by a combined-video/audio-signal generating unit according to the first embodiment;
Figs. 8A and 8B are sequence chart diagrams
illustrating operations of the communication system
according to the first embodiment;
Fig. 9 is a block diagram illustrating the
configuration of a communication system according to a second embodiment;
Figs. 10A and 10B are diagrams of a data structure of additional information according to the second embodiment;
Figs. 11A and 11B are diagrams of images combined by a combined-video/audio-signal generating unit according to the second embodiment;
Fig. 12 is a sequence chart diagram illustrating operations of the communication system according to the second embodiment;
Fig. 13 is a block diagram illustrating the configuration of a communication system according to a third embodiment; and
Fig. 14 is a sequence chart diagram illustrating operations . of the communication system according to the third embodiment.
BEST MODE(S) FOR CARRYING OUT THE INVENTION
First Embodiment
A first embodiment according to the present invention is explained with reference to Figs. 1 and 2. Fig. 1 is an overall configuration diagram of a communication system to which the present invention is applied. Fig. 2 is a block diagram illustrating the configuration of a communication system according to the first embodiment. The present embodiment is characterized in that a multipoint connection apparatus includes a volume-level calculating unit upstream of a combined-video/audio-signal generating unit.
A communication system 1 includes a plurality of video/audio terminals 100χ to 100n and a multipoint
connection apparatus 200 that relays video/audio signals and various types of information to be transmitted and received by the video/audio terminals 100i to 100n. The video/audio terminals 100i to 100n and the multipoint connection apparatus 200 are connected to each other via a communication network. Hereinafter, any one of the
video/audio terminals 100i to 100n is simply referred to as the "video/audio terminal 100". In addition, a layout- setting-information transmitting unit 300 is connected to the multipoint connection apparatus 200 via the
communication network. The layout-setting-information transmitting unit 300 is a unit that transmits layout setting information which is information about video layouts and information to be displayed on display units of the video/audio terminals 100 to the multipoint connection apparatus 200.
Connected to the illustrated multipoint connection apparatus 200 are the n (n is any number equal to or
greater than two) video/audio terminals 100χ to 100n. As illustrated in Fig. 1, each of the video/audio terminals 100 includes a camera (capturing unit) that captures video of each point, a microphone (audio input unit) that
receives input of audio during a conference and converts the audio into electrical signals, and a display, a
projector, or the like (display unit) that displays video from the points and the like. Fig. 2 illustrates a case where the video/audio terminals 100i to 100n_i serve as sender terminals that transmit video and audio, and the video/audio terminal 100n serves as a receiver terminal that receives the video and audio transmitted from the video/audio terminals 100i to 100η-ι·
The multipoint connection apparatus 200 includes a video/audio-signal receiving unit 201, a decoding unit 203, a volume-level calculating unit 205, a volume-display-image generating unit 207, a layout-setting-information receiving unit 209, a combined-video/audio-signal generating unit 211, an encoding unit 213, and a transmitting unit 215.
The video/audio-signal receiving unit 201 receives video/audio signals transmitted from the video/audio
terminals 100i to 100n. Meanwhile, the video/audio signals in a compressed state are generally transmitted from the video/audio terminals 100.
The decoding unit 203 decodes each of the compressed video/audio signals received by the video/audio-signal receiving unit 201. An example of the decoding unit 203 is explained below. Fig. 3 is a block diagram illustrating an internal configuration of a decoding unit according to the first embodiment. The decoding unit 203 may have such a configuration as illustrated in Fig. 3 so as to be capable of supporting a plurality of encoding formats.
The decoding unit 203 includes a determining unit 221, a decoder selecting unit 223, and a single-format decoder group 227 that includes a plurality of single-format
decoders 225a, 225b, 225c, ... The determining unit 221 determines the encoding format of the compressed
video/audio signal and outputs a result of the
determination to the decoder selecting unit 223. The decoder selecting unit 223 outputs the compressed
video/audio signal to one single-format decoder selected from the single-format decoders 225a, 225b, 225c, ... based on the result of the determination output from the
determining unit 221. The selected single-format decoder decodes the compressed video/audio signal input thereto and outputs a video/audio signal.
The volume-level calculating unit 205 (Fig. 2)
calculates volume levels each respectively from one of the video/audio signals decoded by the decoding unit 203 and outputs the volume levels as volume-level information.
There are a variety of methods for calculating the volume levels. Examples of the method include calculation using maximum amplitude or RMS of an audio signal in a unit time. The "maximum amplitude" denotes an absolute maximum value of signal values in a certain period. The "RMS" denotes root mean square of signal values in a certain period. Fig. 4 is a diagram illustrating a method for calculating a volume level from a video/audio signal according to the first embodiment.
An example case where an audio signal changes with time as illustrated in Fig. 4, in which s(t) represents an audio signal and At represents a unit time, is discussed below. A volume level volmax can be calculated using the maximum amplitude using the following equation.
vol max max s|
e [t!, t2 f
A volume level volrms can be calculated using the RMS using the followin equation.
Meanwhile, the volume level is not necessarily actual maximum amplitude. For example, a method of outputting a volume level that is normalized and quantized to an integer in a predetermined range, for example, from 0 to 10 in decibel value in such a manner that 0 corresponds to a silent state and 10 corresponds to a 0-dB state. The
"normalization" means transforming data according to a certain rule to make the data more usable.
The volume-display-image generating unit 207 (Fig. 2) generates volume display images based on the volume-level information output from the volume-level calculating unit 205. Figs. 5A and 5B are diagrams of volume display images generated by a volume-display-image generating unit
according to the first embodiment.
For example, in Fig. 5A, a volume display image 231 is formed from a volume display area 231a which is a
laterally-elongated rectangle and a bar 231b displayed in the volume display area 231a. The volume display area 231a is displayed in a predetermined base color. The bar 231b contacts the volume display area 231a on at least three sides and is displayed in a color different from the base color of the volume display area 231a. Volume is expressed as an area of the bar 231b in the volume display area 231a. More specifically, volume is expressed by assigning the different color only to a portion, of which area depends on the volume level, of the volume display area 231a. For example, as illustrated in (a-2) , the area of the bar 231b is zero at minimum volume, and only the volume display area 231a is displayed. As illustrated in (a-3) , the area of the bar 231b is maximized at maximum volume, and the entire volume display area 231a becomes the bar 231b.
In Fig. 5B, a volume display image 233 is formed from a volume display area 235 (segment bar graph) , which is formed from a plurality of segment bars 235a arranged in one vertical line, and colored segment bars 237 that represent volume. Each of the segment bars 235a is
displayed in a predetermined base color. The segment bars 237 are some of the segment bars 235a in the volume display area 235 filled with a color different from the base color. In Fig. 5B, volume is expressed as the number of the segment bars 237. More specifically, volume is expressed by assigning the different color only to the segment bars 235a, of which number depends on the volume level, of the volume display area 235. For example, as illustrated in (b-2), the number of the segment bars 237 is zero at minimum volume. As illustrated in (b-3) , all of the segment bars 235a in the volume display area 235 become the segment bars 237 at maximum volume.
Meanwhile, any other image than those of the examples given in Figs. 5A and 5B can be employed so long as the image allow visual recognition of volume. For example, another shape, such as an elliptical shape, a circular shape, or a triangular shape, than the rectangular shape can be employed.
The layout-setting-information receiving unit 209 (Fig.
2) receives the layout setting information transmitted from the layout-setting-information transmitting unit 300.
Meanwhile, the layout setting information is information indicating information about arrangement of videos. More specifically, the layout setting information is information indicating in what layout, or arrangement, the videos of the video/audio terminals lOOi to 100n are to be displayed. Meanwhile, the layout-setting-information transmitting unit 300 is generally incorporated in the video/audio terminal 100n and configured such that layout setting can be
selected and specified by a user of the video/audio
terminal 100n as desired using an operating button or the like of the video/audio terminal 100n.
Figs. 6A and 6B are diagrams of video layouts
according to the first embodiment. Examples where videos at two points are displayed on one screen are illustrated. More specifically, the examples are of a case where a video at a local point and a video at a remote point in a
conference between two points are displayed and a case where videos at two remote points in a conference among three points are displayed.
A display screen 241 illustrated in Fig. 6A is an example of a layout of an area 241A dominating the entire screen and an area 241B that is displayed as being overlaid on the area 241A and smaller than the area 241A. A video at a certain point is displayed on the area 241A, while a video at another point is displayed on the area 241B. In Fig. 6A, the area 241B is displayed as being overlaid on a bottom right portion of the area 241A; however, a position where the area 241B is to be displayed is not limited thereto. A similar layout in which, for example, the area 241B is arranged at bottom left, top left, or top right can be employed.
A display screen 243 illustrated in Fig. 6B is an example layout in which a screen is vertically divided into two parts with an area 243C on the left and an area 243D on the right. A video at a certain point is displayed on the area 243C, while a video at another point is displayed on the area 243D. Examples of a similar layout include that in which a top portion and a bottom portion of each of the areas 243C and 243D are blanked to maintain an aspect ratio of videos to be displayed.
As a matter of course, a video layout other than the examples illustrated in Figs. 6A and 6B may be employed.
The combined-video/audio-signal generating unit 211 (Fig. 2) combines the video/audio signals decoded by the decoding unit 203 with the volume display images generated by the volume-display-image generating unit 207 based on the layout setting information received by the layout- setting-information receiving unit 209. The combined video/audio signal is a signal into which at least
video/audio signals from a plurality of video/audio
terminals are combined. The combined video/audio signal according to the present embodiment is obtained by
combining not only the video/audio signals but also the volume display images. Figs. 7A and 7B are diagrams of videos combined by a combined-video/audio-signal generating unit according to the first embodiment.
The combined video illustrated in Fig. 7A is an example in which layout setting information corresponding to the layout illustrated in Fig. 6B is received and such a volume display image as illustrated in Fig. 5A is used, for example. The volume display images 231 are displayed in lower portions of the area 243C and the area 243D on the display screen 243.
Similarly, the combined video illustrated in Fig. 7B is an example in which layout setting information
corresponding to the layout illustrated in Fig. 6A is received and such a volume display image as illustrated in Fig. 5B is used, for example. The volume display images 233 are displayed in a left end portion of the area 241A and a right end portion of the area 241B on the display screen 243.
The encoding unit 213 (Fig. 2) encodes the combined video/audio signal output from the combined-video/audio- signal generating unit 211 to compress data size.
The transmitting unit 215 transmits the combined video/audio signal compressed by the encoding unit 213 to the video/audio terminal 100n.
The video/audio terminal 100n outputs video that is combined based on the received combined video/audio signal as illustrated in Fig. 7A or 7B to a display unit such as a display or a projector. In practice, each of the
video/audio terminals 100i to 100n-i also receives the video/audio signals from the video/audio terminals 100. Accordingly, n signals are input/output to/from the layout- setting-information receiving unit 209, the combined- video/audio-signal generating unit 211, the encoding unit 213, and the transmitting unit 215.
Operations of the communication system 1 are explained with reference to sequence charts. Figs. 8A and 8B are sequence chart diagrams illustrating operations of the communication system according to the first embodiment.
First, operations related to layout setting
information processing are explained with reference to Fig. 8A. The layout-setting-information transmitting unit 300 transmits layout setting information to the multipoint connection apparatus 200 (Step SI) . The layout-setting- information receiving unit 209 of the multipoint connection apparatus 200 receives the layout setting information (Step S2) and stores it in a storage unit (not shown) (Step S3) .
Next, operations related to video/audio processing are explained with reference to Fig. 8B.
The video/audio terminals lOOi to 100n transmit compressed video/audio signals to the multipoint connection apparatus 200 (Steps Sll and S12).
The video/audio-signal receiving unit 201 of the multipoint connection apparatus 200 receives the n
compressed video/audio signals from the video/audio
terminals 100i to 100n (Step S13) . The decoding unit 203 decodes the compressed video/audio signals into the n compressed video/audio signals (Step S14). The volume- level calculating unit 205 calculates n volume levels from the n video/audio signals decoded by the decoding unit 203 (Step S15) . The volume-display-image generating unit 207 generates n volume display images from information about the n volume levels output from the volume-level
calculating unit 205 (Step S16) . The combined-video/audio- signal generating unit 211 retrieves the layout setting information stored at Step S3 of Fig. 8A and combines the n video/audio signals with the n volume display images (Step S17). The encoding unit 213 encodes the combined
video/audio signal obtained by combining by the combined- video/audio-signal generating unit 211 (Step S18). The transmitting unit 215 transmits the compressed combined video/audio signal generated by the encoding unit 213 to the video/audio terminal 100n (Step S19) .
The video/audio terminal 100n receives the compressed combined video/audio signal (Step S20), decodes it (Step S21) , and displays the obtained video/audio signals on the display unit (Step S22).
The speaker volume of the remote terminal and the microphone volume of the local terminal are information about audio transmitted from the local site to the remote site. Not only such information as described above but also information, such as speaker volume of the local terminal and microphone volume of the remote terminal, about audio transmitted from the remote site to the local site are preferably displayed simultaneously. This permits performing adjustment of speech volume level, which is performed while viewing a volume display related to audio on the local side, based on relationship between how audio provided by a speaker is audible on the local side and volume display related to audio on the remote side, thereby facilitating the adjustment. Furthermore, it is also advantageous that at occurrence of a trouble, the
information can be used as reference information for determining a cause of the trouble.
As described above, according to the present
embodiment, the multipoint connection apparatus includes the volume-level calculating unit and the volume-display- image generating unit, and transmits a combined video/audio signal into which the volume display images are combined by the combined-video/audio-signal generating unit to the video/audio terminal. The video/audio terminal can receive the combined video/audio signal in which the volume display images are already contained. Therefore, the video/audio terminal which is a receiving side can display microphone volume of the other video/audio terminals individually independent of the configuration of the video/audio
terminal .
Furthermore, utilizing the maximum amplitude in calculation of the volume levels permits employing
appropriate values as the volume levels. Furthermore, utilizing the RMS in calculation of the volume levels permits employing easy-to-handle values as the volume levels. Furthermore, utilizing the volume levels that are normalized and quantized permits displaying volume in an intuitive and understandable manner.
Meanwhile, in the present embodiment, only receiving operations by only the video/audio terminal 100n are described for convenience of explanation. However, the video/audio terminals 100i to 100n-i also receive
video/audio signals in practice, and processing similar to that described above is performed.
Second Embodiment
A second embodiment according to the present invention is explained with reference to Fig. 9. Fig. 9 is a block diagram illustrating the configuration of a communication system according to the second embodiment. The present embodiment is characterized in that video/audio terminals perform generating volume display images and combining the generated volume display images with video/audio signals.
Connected to a multipoint connection apparatus 250 of a communication system 2 are n (n is any number equal to or greater than two) video/audio terminals 110i to 110n. Fig. 9 illustrates a case where the video/audio terminal 110n receives video/audio signals from the video/audio terminals 110i to llOn-i- Note that terminals having the same
configuration as those of the first embodiment can be used as the video/audio terminals 110i to 110n-i.
The multipoint connection apparatus 250 includes the video/audio-signal receiving unit 201, the decoding unit 203, the volume-level calculating unit 205, a combined- information-signal generating unit 251, an encoding unit 253, the layout-setting-information receiving unit 209, and a transmitting unit 255. The video/audio-signal receiving unit 201, the decoding unit 203, the volume-level
calculating unit 205, and the layout-setting-information receiving unit 209 are similar to corresponding units of the multipoint connection apparatus 200 of the first embodiment and designated by same reference numerals as those of the first embodiment; accordingly, description thereof is omitted.
The combined-information-signal generating unit 251 combines the video/audio signals decoded by the decoding unit 203 based on the layout setting information received by the layout-setting-information receiving unit 209.
Furthermore, the combined-information-signal generating unit 251 generates a combined information signal by
embedding information (volume-level information, which is additional information) about the volume levels calculated by the volume-level calculating unit 205 and the layout setting information (additional information) into the obtained combined signal. This embedding of the additional information is performed so as to reduce difference between video/audio signals obtained by decoding the combined information signal and the original video/audio signals. For example, the additional information is embedded into high-frequency components of the video/audio signals.
Embedding in this way allows even an video/audio terminal that does not have a special configuration or, more
specifically, even a video/audio terminal (e.g., the video/audio terminal 110) that does not include a volume- display-image generating unit 115 and a video/audio
combining unit 117, which will be described later, to acquire and utilize video/audio signals, and also prevents an increase in data size.
Figs. 10A and 10B are diagrams of a data structure of additional information according to the second embodiment. As illustrated in Fig. 10A for example, the volume-level information and the layout setting information have a data structure in which the volume-level information about the video/audio terminals is associated with the corresponding layout information. Layout positional information about each of the video/audio terminals can be expressed in, for example, XY coordinate values of a video display position. Fig. 10B illustrates an example where actual values are assigned. Values corresponding to Fig. 7A are assigned in this example. Coordinates (160, 560) and (800, 560) of a top left point and coordinates (320, 620) and (960, 620) of a bottom right point of the volume display images 231 are assigned for terminal numbers 1 and 2, respectively, as the layout setting information. Negative values in layout setting for a terminal number 3 indicate that video of the video/audio terminal to which this number is assigned is not to be displayed.
The encoding unit 253 (Fig. 9) encodes the combined information signal output from the combined-information- signal generating unit 251 to compress data size.
The transmitting unit 255 transmits the combined information signal compressed by the encoding unit 253 to the video/audio terminal 110n- The video/audio terminal 110n includes a combined- information-signal receiving unit 111, a decoding unit 113, the volume-display-image generating unit 115, and the video/audio combining unit 117.
The combined-information-signal receiving unit 111 receives the compressed combined information signal
transmitted from the multipoint connection apparatus 250.
The decoding unit 113 decodes the compressed combined information signal.
The volume-display-image generating unit 115 generates a display screen including volume display images indicating volume based on the volume-level information and the layout setting information embedded in the combined information signal. Figs. 11A and 11B are diagrams of images combined by a combined-video/audio-signal generating unit according to the second embodiment.
The combined video illustrated in Fig. 11A is an example of an image generated with such a volume display method as illustrated in Fig. 5A and from the layout setting information corresponding to Fig. 6B, for example. The volume display images 231 that indicate volume of the video/audio terminals, to which the terminal numbers 1 and 2 are assigned, are individually displayed in a bottom. portion of a display screen 121.
Similarly, the combined video illustrated in Fig. 11B is an example of an image generated with such a volume display method as illustrated in Fig. 5B and from the layout setting information corresponding to Fig. 6A, for example. The volume display images 233 that indicate volume of the video/audio terminals, to which the terminal numbers 1 and 2 are assigned, are displayed in a lower portion of a left end portion and a lower portion of a right end portion, respectively, of a display screen 123.
As a matter of course, any video other than the examples illustrated in Figs. 11A and 11B may be obtained by combining so long as the volume levels and the display images, and the layout setting information and the display position correspond to each other.
The video/audio combining unit 117 (Fig. 9) combines the combined information signal decoded by the decoding unit 113 with the volume display images generated by the volume-display-image generating unit 115. More
specifically, the video/audio combining unit 117 assumes the combined information signal that contains the volume- level information as the video/audio signals and utilizes the combined information signal as it is without performing an operation of separating only the video/audio signals from the combined information signal. For example, when the combined information signal that is assumed as the video/audio signals is combined with such volume display images as illustrated in Fig. 11A, such a video/audio signal as illustrated in Fig. 7A is obtained.
The video/audio terminal 110-n outputs such video as illustrated in Figs. 7A and 7B based on the combined video/audio signal to a display unit such as a display or a projector.
Operations of the communication system 2 explained above are explained with reference to a seguence chart. Fig. 12 is a sequence chart diagram illustrating the operations of the communication system according to the second embodiment. This diagram illustrates only
operations related to video/audio processing. Operations related to layout setting information processing are similar to those of the first embodiment (see Fig. 8A) and description thereof is omitted. Processes similar to those of the first embodiment are designated by like step numbers and description thereof is omitted.
The video/audio terminals 110i to 110n transmit compressed video/audio signals to the multipoint connection apparatus 250 (Steps Sll and S12) .
The multipoint connection apparatus 250 performs processing from Step S13 to Step S15.
The combined-information-signal generating unit 251 of the multipoint connection apparatus 250 retrieves the layout setting information stored at Step S3 of Fig. 8A and combines the n video/audio signals based on the layout setting information (Step S31) . Furthermore, the combined- information-signal generating unit 251 generates a combined information signal by embedding additional information containing the layout setting information and the volume- level information into the video/audio signal obtained by combining (Step S32).
Encoding and transmission of the obtained combined video/audio signal are performed (Steps S18' and S19').
The combined-information-signal receiving unit 111 of the video/audio terminal 110n receives the compressed combined information signal (Step S20')- The decoding unit 113 decodes the compressed combined information signal (Step S21'). The volume-display-image generating unit 115 generates volume display images based on the volume-level information and the layout setting information embedded in the decoded combined information signal (Step S33) . The video/audio combining unit 117 combines the combined information signal with the volume display images (Step S34). The obtained video/audio is displayed on the display unit (Step S22 ' ) .
As described above, it is possible to display
microphone volume at a plurality of points individually according to the present embodiment. Furthermore, it is possible to provide the display described above
irrespective of the configuration of a video/audio terminal at another point serving as a sender terminal.
Furthermore, the combined-information-signal
generating unit of the multipoint connection apparatus generates a combined information signal by embedding additional information containing layout setting
information and volume levels in video/audio signals, and transmits the combined information signal to the
video/audio terminals. Accordingly, it is possible to transmit and receive the volume levels and the layout setting information without increasing a data size of the original video/audio signals.
In the first embodiment, each of the video/audio terminals obtain a similar video irrespective of the configuration of the video/audio terminal; however, volume display can disadvantageously become redundant in such a case where the video/audio terminal includes a volume display unit of its own or the like. However, such a situation will not occur with the second embodiment because the video/audio terminal includes the volume-display-image generating unit and the video/audio combining unit.
Third Embodiment
A third embodiment which is a variation of the second embodiment is explained. In the communication system 2 (Fig. 9) according to the second embodiment, the additional information made up of the volume-level information and the layout setting information is embedded in the combined information signal to transmit the volume-level information and the layout setting information to the video/audio terminal. Contrasted therewith, a configuration in which the combined video/audio signal is transmitted through a first channel and the additional information is transmitted through a second channel to the video/audio terminal rather than performing the embedding may be employed. Fig. 13 is a block diagram illustrating the configuration of a
communication system according to the third embodiment. Components similar to those of the first and second
embodiments are designated by like reference numerals and description thereof is omitted.
The combined-information-signal generating unit 251 of a multipoint connection apparatus 260 of a communication system 3 includes two units which are an additional- information generating unit 261 and a combined-video/audio- signal generating unit 263. The additional-information generating unit 261 generates additional information having such a data structure as illustrated in Figs. 10A and 10B from volume levels and layout setting information. The combined-video/audio-signal generating unit 263 outputs a combined video/audio signal into which video/audio signals of video/audio terminals 130 are combined based on the layout setting information. Meanwhile, the additional information and the combined video/audio signal output from the combined-information-signal generating unit 251 according to the present invention are collectively
referred to as a combined information signal.
The encoding unit 253 includes an additional- information encoding unit 265 and a combined-video/audio- signal encoding unit 267. The additional-information encoding unit 265 encodes the additional information, and the combined-video/audio-signal encoding unit 267 encodes the combined video/audio signal to compress data size.
The transmitting unit 255 includes an additional- information transmitting unit 269 and a combined- video/audio-signal transmitting unit 271. The additional- information transmitting unit 269 transmits the compressed additional information. The combined-video/audio-signal transmitting unit 271 transmits the compressed combined video/audio signal.
The combined-information-signal receiving unit 111 of the video/audio terminal 130n includes an additional- information receiving unit 131 and a combined-video/audio- signal receiving unit 133. The additional-information receiving unit 131 receives the compressed additional information. The combined-vided/audio-signal receiving unit 133 receives the compressed combined video/audio signal .
The decoding unit 113 includes an additional- information decoding unit 135 and a combined-video/audio- signal decoding unit 137. The additional-information decoding unit 135 decodes the compressed additional
information. The combined-video/audio-signal decoding unit 137 decodes the compressed combined video/audio signal.
Operations of the communication system 3 explained above are explained with reference to a sequence chart. Fig. 14 is a sequence chart diagram illustrating the operations of the communication system according to the third embodiment. This diagram illustrates only operations related to video/audio processing. Operations related to layout setting information processing are similar to those of the first embodiment (see Fig. 8A) and description thereof is omitted. Processing similar to that of the first and second embodiments is designated by a like step number and description thereof is omitted.
The video/audio terminals 130i to 130n transmit compressed video/audio signals to the multipoint connection apparatus 260 (Steps Sll and S12) .
The multipoint connection apparatus 260 performs processing from Step S13 to Step S15.
Video audio signals are processed through a first channel of the multipoint connection apparatus 260. The combined-video/audio-signal generating unit 263 retrieves the layout setting information stored at Step S3 of Fig. 8A and combines the n video/audio signals based on the layout setting information (Step S41). The combined-video/audio- signal encoding unit 267 encodes the combined video/audio signal combined by the combined-video/audio-signal
generating unit 263 (Step S42). The combined-video/audio- signal transmitting unit 271 transmits the compressed combined video/audio signal encoded by the combined- video/audio-signal encoding unit 267 to the video/audio terminal 130n (Step S43) .
On the other hand, additional information is processed through a second channel of the multipoint connection apparatus 260. The additional-information generating unit 261 retrieves the layout setting information stored at Step S3 of Fig. 8A and generates additional information
containing the layout setting information and the volume- level information (Step S44). The additional-information encoding unit 265 encodes the additional information generated by the additional-information generating unit 261 (Step S45) . The additional-information transmitting unit 269 transmits the compressed additional information encoded by the additional-information encoding unit 265 to the video/audio terminal 130n (Step S46) .
The video/audio signal is processed through a first channel of the video/audio terminal 130n. The combined- video/audio-signal receiving unit 133 receives the
compressed combined video/audio signal (Step S47). The combined-video/audio-signal decoding unit 137 decodes the compressed combined video/audio signal (Step S48).
On the other hand, the additional information is processed through a second channel of the video/audio terminal 130n. The additional-information receiving unit 131 receives the compressed additional information (Step S49) . The additional-information decoding unit 135 decodes the compressed additional information (Step S50). The volume-display-image generating unit 115 generates volume display images based on the volume-level information and the layout setting information contained in the decoded additional information (Step S33').
The video/audio combining unit 117 combines the combined video/audio signal decoded by the combined- video/audio-sighal decoding unit 137 with the volume display images generated by the volume-display-image generating unit 115 (Step S34 ' ) . The obtained video/audio signals are displayed on the display unit (Step S22 ' ' ) .
As described above, according to the present
embodiment, the additional information and the video/audio signals are transmitted through the different channels.
Accordingly, it is possible to transmit and receive the volume-level information without modifying the original video/audio signals.
In the first embodiment, a similar video is obtained irrespective of the configuration of the video/audio
terminal; however, volume display can disadvantageously become redundant in such a case where the video/audio terminal includes a volume display unit of its own or the like. However, such a situation will not occur with the third embodiment because the video/audio terminal includes the volume-display-image generating unit and the
video/audio combining unit.
According to the embodiments, volume levels are
calculated from video/audio signals each transmitted from video/audio terminals respectively, volume display images are generated from the volume levels, and a signal into which the volume display images and the video/audio signals transmitted from the video/audio terminals are combined is generated. Accordingly, even when a multipoint conference is held by connecting three or more remote sites, each of the video/audio terminals can display microphone volume of the points separately.
Although the invention has been described with respect to specific embodiments for a complete and clear disclosure, the appended claims are not to be thus limited but are to be construed as embodying all modifications and alternative constructions that may occur to one skilled in the art that fairly fall within the basic teaching herein set forth.

Claims

1. A multipoint connection apparatus connected to a plurality of video/audio terminals, the multipoint
connection apparatus comprising:
a video/audio-signal receiving unit configured to receive video/audio signals output from the video/audio terminals, respectively;
a volume-level calculating unit configured to
calculate volume levels from the respective video/audio signals ;
a volume-display-image generating unit configured to generate volume display images indicating volume based on the respective volume levels calculated by the volume-level calculating unit;
a layout-setting-information receiving unit configured to receive pieces of layout setting information, each indicating information about arrangement of videos to be displayed on the corresponding video/audio terminal;
a combined-video/audio-signal generating unit
configured to generate a combined video/audio signal for each of the video/audio terminals by combining the
video/audio signals and the volume display images based on the piece of layout setting information for the
corresponding video/audio terminal; and
a transmitting unit configured to transmit the
combined video/audio signal to the corresponding
video/audio terminal.
2. The multipoint connection apparatus according to claim 1, wherein each of the volume levels is a maximum amplitude in a unit time of an audio signal.
3. The multipoint connection apparatus according to claim 1, wherein each of the volume levels is a root mean square in a unit time of an audio signal.
4. A multipoint connection apparatus connected to a plurality of video/audio terminals, the multipoint
connection apparatus comprising:
a video/audio-signal receiving unit configured to receive video/audio signals output from the video/audio terminals, respectively;
a volume-level calculating unit configured to
calculate volume levels from the respective video/audio signals ;
a layout-setting-information receiving unit configured to receive pieces of layout setting information, each indicating information about arrangement of videos to be displayed on the corresponding video/audio terminal;
a combined-information-signal generating unit
configured to generate a combined information signal for each of the video/audio terminals from the video audio signals, the volume levels, and the piece of layout setting information for the corresponding video/audio terminal; and a transmitting unit configured to transmit the
combined information signal to the corresponding
video/audio terminal.
5. The multipoint connection apparatus according to claim 4, wherein each of the volume levels is a maximum amplitude in a unit time of an audio signal.
6. The multipoint connection apparatus according to claim 4, wherein each of the volume levels is a root mean square in a unit time of an audio signal.
7. The multipoint connection apparatus according to claim 4, wherein the combined information signal is a signal obtained by embedding the piece of layout setting
information and the volume levels into the video/audio signals.
8. The multipoint connection apparatus according to claim 4, wherein
the combined information signal includes a combined video/audio signal into which the corresponding video/audio signals are combined based on the piece of layout setting information, and additional information made up of the piece of layout setting information and the volume levels, the combined-information-signal generating unit outputs the combined video/audio signal through a first channel and outputs the additional information through a second channel, and
the transmitting unit transmits to the corresponding video/audio terminal the combined video/audio signal through a first channel, and the additional information through a second channel.
9. A communication system comprising:
a multipoint connection apparatus; and
a plurality of video/audio terminals, the multipoint connection apparatus and the video/audio terminals being connected to each other via a communication network, wherein
the multipoint connection apparatus includes
a video/audio-signal receiving unit configured to receive video/audio signals output from the video/audio terminals, respectively; a volume-level calculating unit configured to calculate volume levels from the respective video/audio signals;
a layout-setting-information receiving unit configured to receive pieces of layout setting information, each indicating information about arrangement of videos to be displayed on the corresponding video/audio terminal;
a combined-information-signal generating unit configured to generate a combined information signal for at least one of the video/audio terminals from the video audio signals, the volume levels, and the piece of layout setting information for the corresponding video/audio terminal; and a transmitting unit configured to transmit the combined information signal to the corresponding
video/audio terminal, and
the at least one of the video/audio terminals includes a combined-information-signal receiving unit configured to receive the combined information signal;
a volume-display-image generating unit configured to generate volume display images indicating volume from the volume levels and the piece of layout setting
information contained in the combined information signal; and
a video/audio combining unit configured to combine the video/audio signals contained in the combined information signal with the volume display images.
10. The communication system according to claim 9, wherein each of the volume levels is a maximum amplitude in a unit time of an audio signal.
11. The communication system according to claim 9, wherein each of the volume levels is a root mean square in a unit time of an audio signal.
12. The communication system according to claim 9, wherein the combined information signal is a signal obtained by embedding the piece of layout setting information and the volume levels into the video/audio signals.
13. The communication system according to claim 9, wherein the combined information signal includes a combined
video/audio signal into which the corresponding video/audio signals are combined based on the piece of layout setting information, and additional information made up of the piece of layout setting information and the volume levels, the combined-information-signal generating unit outputs the combined video/audio signal through a first channel and outputs the additional information through a second channel, and
the transmitting unit transmits to the corresponding video/audio terminal the combined video/audio signal through a first channel, and the additional information through a second channel,
the combined-information-signal receiving unit
receives the combined video/audio signal through the first channel, and receives the additional information through the second channel,
the volume-display-image generating unit generates volume display images indicating volume from the additional information, and ·
the video/audio combining unit combines the volume display images with the combined video/audio signal.
EP12810881.8A 2011-07-14 2012-07-10 Multipoint connection apparatus and communication system Not-in-force EP2732622B8 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2011155690A JP5817276B2 (en) 2011-07-14 2011-07-14 MULTIPOINT CONNECTION DEVICE, VIDEO / AUDIO TERMINAL, COMMUNICATION SYSTEM, AND SIGNAL PROCESSING METHOD
PCT/JP2012/068039 WO2013008941A1 (en) 2011-07-14 2012-07-10 Multipoint connection apparatus and communication system

Publications (4)

Publication Number Publication Date
EP2732622A1 true EP2732622A1 (en) 2014-05-21
EP2732622A4 EP2732622A4 (en) 2014-12-24
EP2732622B1 EP2732622B1 (en) 2021-01-06
EP2732622B8 EP2732622B8 (en) 2021-03-24

Family

ID=47506216

Family Applications (1)

Application Number Title Priority Date Filing Date
EP12810881.8A Not-in-force EP2732622B8 (en) 2011-07-14 2012-07-10 Multipoint connection apparatus and communication system

Country Status (8)

Country Link
US (1) US9392224B2 (en)
EP (1) EP2732622B8 (en)
JP (1) JP5817276B2 (en)
KR (1) KR101561371B1 (en)
CN (1) CN103703766A (en)
AU (1) AU2012281463B2 (en)
CA (1) CA2841096C (en)
WO (1) WO2013008941A1 (en)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014195229A (en) * 2013-03-29 2014-10-09 Brother Ind Ltd Server device program, server device and communication apparatus program
CN105592286B (en) * 2014-10-22 2019-03-01 阿里巴巴集团控股有限公司 Instant messaging interface information processing method and device
US9794685B2 (en) 2015-01-23 2017-10-17 Ricoh Company, Ltd. Video audio recording system, video audio recording device, and video audio recording method
CN108182097B (en) * 2016-12-08 2021-07-30 武汉斗鱼网络科技有限公司 A kind of realization method and device of volume bar
CN108182047B (en) * 2016-12-08 2020-10-16 武汉斗鱼网络科技有限公司 Volume information display method and device
US10516852B2 (en) * 2018-05-16 2019-12-24 Cisco Technology, Inc. Multiple simultaneous framing alternatives using speaker tracking
CN110636329A (en) * 2018-06-22 2019-12-31 视联动力信息技术股份有限公司 Audio data processing method and device
CN119232871A (en) * 2023-06-29 2024-12-31 中兴通讯股份有限公司 Terminal, terminal interaction method, device and computer readable storage medium

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3085317B2 (en) 1991-05-07 2000-09-04 株式会社リコー Telephone equipment
GB2284968A (en) * 1993-12-18 1995-06-21 Ibm Audio conferencing system
JP3449772B2 (en) 1994-02-15 2003-09-22 沖電気工業株式会社 Multipoint conference equipment
JPH099229A (en) * 1995-06-15 1997-01-10 Oki Electric Ind Co Ltd Multi-spot conference system
JP2004186870A (en) 2002-12-02 2004-07-02 Nippon Telegr & Teleph Corp <Ntt> Video conference method, processing method of user terminal device, user terminal device, program, and recording medium
JP4308555B2 (en) * 2003-03-10 2009-08-05 パナソニック株式会社 Receiving device and information browsing method
US7499969B1 (en) * 2004-06-25 2009-03-03 Apple Inc. User interface for multiway audio conferencing
US7899170B2 (en) * 2005-04-28 2011-03-01 Apple Inc. Multi-participant conference setup
US7864209B2 (en) * 2005-04-28 2011-01-04 Apple Inc. Audio processing in a multi-participant conference
US7949117B2 (en) * 2005-04-28 2011-05-24 Apple Inc. Heterogeneous video conferencing
US20060248210A1 (en) 2005-05-02 2006-11-02 Lifesize Communications, Inc. Controlling video display mode in a video conferencing system
JP4522332B2 (en) * 2005-07-01 2010-08-11 日本電信電話株式会社 Audiovisual distribution system, method and program
JP4567543B2 (en) * 2005-07-22 2010-10-20 パイオニア株式会社 Electronic conference system and its conference terminal
WO2007063922A1 (en) 2005-11-29 2007-06-07 Kyocera Corporation Communication terminal and communication system, and display method of communication terminal
US7768543B2 (en) * 2006-03-09 2010-08-03 Citrix Online, Llc System and method for dynamically altering videoconference bit rates and layout based on participant activity
JP2008085677A (en) * 2006-09-27 2008-04-10 Toshiba Corp Information control apparatus, information synthesis apparatus, and program
JP2008096746A (en) * 2006-10-12 2008-04-24 Canon Inc Display control device, display device, and multi-display system
KR100847147B1 (en) 2007-04-18 2008-07-18 한국전자통신연구원 Video conference control method and apparatus
KR101414603B1 (en) 2007-05-07 2014-07-03 엘지전자 주식회사 A multiparty call method and a communication terminal capable of implementing the same
JP2009065336A (en) * 2007-09-05 2009-03-26 Hitachi Communication Technologies Ltd Video conference system
US8421840B2 (en) * 2008-06-09 2013-04-16 Vidyo, Inc. System and method for improved view layout management in scalable video and audio communication systems
JP2010103853A (en) 2008-10-24 2010-05-06 Panasonic Corp Sound volume monitoring apparatus and sound volume monitoring method
JP5857674B2 (en) 2010-12-22 2016-02-10 株式会社リコー Image processing apparatus and image processing system

Also Published As

Publication number Publication date
EP2732622A4 (en) 2014-12-24
US9392224B2 (en) 2016-07-12
AU2012281463A1 (en) 2014-01-30
CA2841096C (en) 2017-01-17
CN103703766A (en) 2014-04-02
CA2841096A1 (en) 2013-01-17
EP2732622B1 (en) 2021-01-06
US20140139611A1 (en) 2014-05-22
WO2013008941A1 (en) 2013-01-17
AU2012281463B2 (en) 2015-11-05
KR101561371B1 (en) 2015-10-26
EP2732622B8 (en) 2021-03-24
JP5817276B2 (en) 2015-11-18
KR20140012755A (en) 2014-02-03
JP2013021633A (en) 2013-01-31

Similar Documents

Publication Publication Date Title
CA2841096C (en) Multipoint connection apparatus and communication system
JP6179834B1 (en) Video conferencing equipment
CN102638672B (en) Automatic video layout for multi-stream multi-site telepresence conferencing systems
US8259625B2 (en) Method, system, and device of video communication
EP3319344A1 (en) Method and apparatus for generating and playing audio signals, and system for processing audio signals
US9497390B2 (en) Video processing method, apparatus, and system
EP2457625A1 (en) Audio processing in a multi-participant conference
CN104113721B (en) The display methods and device of conference materials in a kind of video conference
CN102387338B (en) Distributed type video processing method and video session system
JP4331049B2 (en) Data transmission apparatus and data transmission method
CN101742221A (en) Method and device for synthesizing multiple pictures in video conference system
JP2018056766A (en) Television conference device
US10244207B2 (en) Videoconference communication device
CN102256099A (en) Parameter control method and device
JP2013046319A (en) Image processing apparatus and image processing method
CN107277425A (en) A kind of server, meeting-place terminal and cloud meeting processing method
JP5540492B2 (en) Communication device, sound effect output control program, and sound effect output control method
KR101427278B1 (en) Video conference server and video conference terminal and video conference method using the same
CN113473144B (en) YUV4:4:4 image transmission method
CN101742217B (en) Video communication method and system
JP2016189505A (en) Video conference system
JP2000203193A (en) Displaying device of electronic white board and conferential terminal using it and television conference system
JP2016009904A (en) Image processing system and image processing apparatus
JP2005142639A (en) Signal processing device
KR20120042197A (en) Method for broadcasting using device able to call with image information and apparatus therefor

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20140103

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
A4 Supplementary search report drawn up and despatched

Effective date: 20141120

RIC1 Information provided on ipc code assigned before grant

Ipc: H04N 7/15 20060101AFI20141114BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190725

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20200804

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1353626

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210115

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602012074047

Country of ref document: DE

GRAT Correction requested after decision to grant or after decision to maintain patent in amended form

Free format text: ORIGINAL CODE: EPIDOSNCDEC

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: PK

Free format text: BERICHTIGUNG B8

RAP2 Party data changed (patent owner data changed or rights of a patent transferred)

Owner name: RICOH COMPANY, LTD.

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20210106

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1353626

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210106

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210506

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210407

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210406

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210406

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210506

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602012074047

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

26N No opposition filed

Effective date: 20211007

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20210731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210731

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210506

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210710

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210710

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20120710

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230522

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20230720

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230725

Year of fee payment: 12

Ref country code: DE

Payment date: 20230719

Year of fee payment: 12

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210106

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602012074047

Country of ref document: DE

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20240710

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20250201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20240710